2 min read
Write the evaluation first
Before changing a prompt, decide how you would know the change helped. Twenty labelled examples beat an afternoon of intuition.
- evaluation
- prompts
Learning to build with AI,and writing down whatactually holds up.
kept since 2026, still being revised
A working notebook about the distance between an AI idea that demonstrates nicely and one that holds up once people depend on it. One decision per note, trade-offs included.
the most recent note
Write the evaluation first
Before changing a prompt, decide how you would know the change helped. Twenty labelled examples beat an afternoon of intuition.
2 min read
Premise
why any of this is written down
AI Learn Lab is where we keep notes while learning and deploying AI-assisted software. It is written mostly for ourselves — a record of what worked, what turned out to be more complicated than expected, and what we would do differently next time.
The notes are deliberately narrow. They are about specific decisions: how to tell whether a change actually improved anything, what to log, where to draw the boundary around a first version. We try to write down the trade-offs rather than only the conclusions, because the trade-offs are usually the part that transfers.
What we keep coming back to
Evaluation before optimisation
Decide how you would recognise an improvement before spending a week trying to produce one. Twenty labelled examples beat an afternoon of intuition.
Small, checkable scope
One task with a bounded output beats a broad capability nobody can verify. Narrow first, then widen once the narrow version is measurably working.
The gap at deployment
Timeouts, malformed responses, cost per call, and what the system does when it has no good answer. None of it shows up in a demo.
Recent notes
All notes1 min read
What deployment actually changes
A prototype answers the question "can this work?". Production answers "what happens when it doesn't?" — a different question, with different work behind it.
- deployment
- observability
1 min read
Keeping the scope small on purpose
The projects that get finished tend to describe one narrow task with a checkable answer. Breadth is easier to promise and much harder to verify.
- scoping
- evaluation
Getting in touch
one address, read by a person
hello@ailearnlab.org
If something here is wrong, or matches or contradicts your own experience, we would genuinely like to hear about it. Replies are written by hand, so they are not immediate.