Write the evaluation first

The fastest way to waste a week on an AI feature is to start tuning the prompt. Prompt changes feel productive. Each edit produces different output, the output often looks better, and there is no obvious moment when you are finished.

The habit worth building is boring: write down how you would know the change helped, before you make it.

What that looks like in practice

Collect twenty or thirty real inputs — not invented ones. Real inputs are messier than the examples you would write yourself, and the mess is the point. For each one, record what a good answer looks like. Not the exact wording; the properties that matter. Did it find the right record? Did it decline when there was no answer? Did it stay inside the format the next step expects?

Then run the current version against that set and count. You now have a number, and the number is almost always worse than you assumed.

Why the small set is enough

Twenty examples will not give you a statistically defensible measurement, and it is worth being honest that they don’t. What they give you is a tripwire. When a prompt change quietly breaks the cases that used to work, twenty examples will usually notice. Vibes will not.

The set grows on its own. Every failure someone reports becomes another example, and after a few months the file is the most valuable artifact in the project — more valuable than the prompt, which will have been rewritten several times.

The part that stays uncomfortable

Some qualities resist counting. Tone, tact, whether a summary emphasised the right thing. For those, a small graded review by a person is still the honest option, and it is worth doing on a schedule rather than never.

Being able to measure four of the five things that matter is a much better position than measuring none of them, as long as you remember which one you left out.