Context: evaluation decides what matters
It is easy to treat evaluation as a quality gate bolted on at the end: build the feature, then check it. Working on AI products has pushed me to the opposite view. Evaluation is part of product design because it decides what behavior matters. The moment I write down what the system must and must not do, I am making product decisions, not just engineering ones.
What I observed
A model can pass a narrow prompt test and still fail the product. It fails when the workflow changes around it, when the input is noisy instead of clean, or when the user expects a different kind of answer than the one the test rewarded. None of those failures show up if the evaluation only replays the happy path.
The finding
Good evaluation starts with the user promise. Before writing test cases, I ask three questions. What should the system never do? What should it always show? And what should it admit when it does not know? The answers define the behavior worth testing — and they are product questions before they are technical ones.
How I apply it
I design tests around failure modes, acceptance criteria, user language, and repeatable behavior. That is why the layers matter together rather than separately: UAT for how real people actually phrase things, prompt evals for model behavior, Playwright flows for the surface the user touches, and unit tests for the logic underneath.
On ReviewLens AI, that meant shipping with the evaluation evidence attached: 168 of 173 Vitest tests passing with the 5 failures documented, alongside 6/6 promptfoo LLM evals. On the HR Intelligence Platform, it meant 1,909 tests across Pytest and Playwright E2E. The numbers are not the point — the point is that each layer checks a different part of the promise.