Build a representative evaluation set
Sample normal cases, important edge cases, and known failures. Keep a held-out set separate from examples used to tune instructions. Remove sensitive data or obtain appropriate permission. Track the origin and version of each case.
Use the right denominator
Precision asks how many selected positives were correct; recall asks how many actual positives were found. Accuracy can hide poor performance on a rare but important class. Report task-specific slices and error costs, not just a single headline average.
Compare under controlled conditions
Evaluate candidates on the same cases and conditions. Record model, prompt, settings, and scoring version. Blind human reviewers where practical, calibrate their rubric, and inspect disagreements. A model judge is another measurement system that needs validation.
A worked example
Precision is 18 / 24 = 75%. Recall is 18 / 30 = 60%. If missing an urgent ticket is costly, improving recall may matter more than maximizing overall accuracy.
The point: Metrics describe different errors; choose them from the business consequence.
Before you take the exam
Explain these concepts in your own words, work through a different example, and describe why the tempting alternatives fail. Use the references to go deeper.