Situation

A team used a large model to triage support tickets and draft replies, but every member classified differently, dozens of prompt versions had no comparison record, and newcomers started from scratch. The goal: fix the taxonomy, templates, and acceptance criteria in documents.

Approach

• Pull the last 200 real tickets; define categories and what must not be automated
• Trial-label a small batch; merge or split ambiguous categories before scaling
• Split prompt templates by scenario; variables and refusal conditions live in one doc
• Build a small evaluation set; every prompt change runs against it

Deliverables

• Task definition and category descriptions
• Annotation spec, boundary samples, and QA ratio
• Prompt templates split by scenario
• Evaluation set and failure case records