Treat estimates as uncertain
A small observed improvement may be sampling noise. Use enough representative cases, paired comparisons where suitable, and uncertainty estimates appropriate to the metric. Choose acceptable failure thresholds before observing results. High-risk rare events need targeted testing beyond random sampling.
Challenge the measurement itself
Check for benchmark leakage, judge preference for verbosity, and rubric shortcuts. Use ablations to isolate the contribution of a component. Review whether optimizing the metric has reduced the quality people actually need. An evaluation is evidence with limitations, not a universal guarantee.
Connect offline evidence to operations
Stage deployment, monitor outcome slices, and define rollback triggers. Changes in user behavior, data, tools, or policy can invalidate earlier results. Keep evaluation cases versioned and add real failures without losing a stable comparison set.
A worked example
Inspect the segment, confirm sample size and uncertainty, and apply the pre-agreed harm threshold. Delay or constrain rollout while investigating. A higher overall average cannot by itself justify a critical subgroup regression.
The point: Deployment decisions need explicit constraints as well as average performance.
Before you take the exam
Explain these concepts in your own words, work through a different example, and describe why the tempting alternatives fail. Use the references to go deeper.