Which Models Actually Hold a Schema
Scrollem needs every agent to return valid structured data. Cheap tokens turned out not to mean cheap output.
Scrollem runs a team of agents that pressure-test a business idea. Each one does a job and hands structured data to the next stage. Free text is useless to it. The output has to fit a schema or the pipeline stops.
That single requirement turned out to be the whole engineering problem.
The cheap model trap
I went looking for the lowest cost per token, because the pipeline makes a lot of calls. DeepSeek is dramatically cheaper than the frontier models and perfectly capable at the reasoning itself.
It could not reliably return valid JSON against a fixed schema.
Not “occasionally malformed”. It would return something plausible, well argued, and structurally wrong in a way you only discover when you try to parse it. A missing required field. An array where an object was specified. A number rendered as a string. The content was often fine. The shape was not.
Cost per token is not cost per correct output
Once you add retries, the arithmetic changes completely.
A model at a fifth of the price that needs three attempts and still fails sometimes is not cheaper. It is cheaper per call and more expensive per completed job, and it costs you a validation layer, a retry policy, and a fallback path that you now maintain forever.
I ended up routing by task rather than picking one model. Reasoning where the output is prose goes to the cheap model. Anything that has to hold a schema goes to a model with native structured output support.
What I would tell anyone building this
Validate at the boundary, always. Never assume the output matches, whatever the model. Parse against the schema and treat a failure as a normal event rather than an exception.
Make retries bounded and visible. Silent retries hide the fact that a model is failing most of the time, right up until the bill arrives.
Test schema adherence separately from quality. They are different failures with different fixes. A model can be excellent at one and useless at the other, and an eval that mixes them tells you nothing actionable.
Where this shows up next
Anything that turns unstructured input into structured records. Reading a receipt into line items. Turning a photographed recipe into ingredients and quantities. Pulling parties and dates out of a contract.
Same problem every time: the model has to produce something a program can consume, and hoping is not a strategy.