The Failures You Can't See Are the Expensive Ones
Backlot chats about TV shows and must never spoil one. Prompting wasn't enough, so I built an eval suite.
Backlot answers questions about TV shows. Cast, recaps, what happened in episode four. The whole product falls apart if it spoils something you haven’t watched yet.
That is a harder constraint than it sounds.
Prompting is not a guarantee
The obvious move is to tell the model not to spoil. I did that. It works most of the time, which is exactly the problem.
“Most of the time” is fine for a writing assistant. It is not fine when a single leak means someone had a season finale ruined by your product. There is no undo. You don’t get to apologise your way out of it.
The gap between “usually behaves” and “reliably behaves” is where a lot of AI products quietly live. You cannot see it from the happy path, because the happy path is the one you tested.
So I built evals
Not accuracy evals. Behavioural ones. A set of cases whose entire job is to try to make the model leak, and a runner that fails the build when any of them do.
The useful cases were not the obvious attacks. They were the sideways routes:
- Asking about a character’s arc in a way that implies they survive
- Follow-up suggestion chips that summarise ahead of where the viewer is
- A recap that is technically about episode two but references episode nine
- Innocent questions where the honest answer is itself the spoiler
That last category is the one that got me. Retrieval was pulling the right document and the model was answering it correctly, and the correct answer was the thing it must not say. Nothing in the system was broken. The behaviour was still wrong.
What I actually learned
Guardrails belong in the test suite, not the prompt. A prompt is a request. A test is a fact.
The dangerous failures are the invisible ones. A wrong answer gets reported. A leaked spoiler quietly costs you a user who never tells you why.
Adversarial cases beat volume. Twenty cases written specifically to break the thing found more than a hundred generic ones would have.
Evals are never finished. Every time I find a new leak, it becomes a case. The suite is a record of every way this has gone wrong so far, which is the only honest definition of coverage.
Where else this applies
Any system where the model must not assert something it cannot support. Legal research that must not invent a citation. Medical summaries that must not state a dose. Financial answers that must not imply advice.
The domain changes. The discipline does not: write down what must never happen, turn it into tests, and run them every single time.