A cheaper model looked fine in a demo of Kai, the AI coach inside Livio. Our LLM-as-judge evals in production said it gave worse coaching, so it did not ship. If you run an LLM feature for real users and want to know how to evaluate an AI agent before switching models, we used a golden dataset of real cases and a judge that scores two models' answers blind, and we agreed in advance that the score would overrule the demo.
The situation
Kai is the whole product experience of Livio, a subscription app for people losing weight in the United States. It is an autonomous Claude agent that streams its replies and calls more than 30 tools to read and change a member's food, weight, sleep and workout data. The member confirms every change it proposes with one tap. Every number, from calories to macros to trends, is computed in code, and the model only explains them.
Because the arithmetic lives in functions, a model switch raises a narrower question for evals: is the coaching as good, and does the agent still pick the right tools? Both can be measured.
A cheaper model was proposed to run Kai's coaching. In a demo it looked fine. The replies read well and nothing obviously broke. A demo is a handful of conversations someone chose. It tells you a model can be good, not that it is as good across the conversations your members actually have.
What the golden dataset and the judge measure
The eval harness has two parts. The first is a golden dataset: a fixed set of cases with known good behaviour. The cases are built from the kinds of conversations members actually have, with anything personal stripped out. Each one checks both halves of the job: whether the agent called the right tools with the right fields, and whether the coaching around them was good. The set does not change between runs, so a score on Monday and a score on Friday mean the same thing.
The second part is a blind LLM-as-judge test of coaching quality. For each case, both models answer. A judge model reads the two answers side by side without being told which model wrote which, and says which is better coaching. Coaching quality cannot be scored by string matching, so a judge with clear criteria is a practical way to score it at volume.
Blinding matters because if the judge, or a person, knows which answer came from the expensive model, the result leans toward what they already believe.
Keeping the LLM judge honest
A judge is another model, and it has its own biases. These are the ways the Kai harness keeps it honest.
- Blind pairs. Neither answer is labelled.
- Position swapping. Judges tend to prefer whichever answer comes first, so each pair is scored twice with the order flipped, and a split verdict counts as a tie.
- A written rubric. The judge scores against stated criteria for good coaching.
- Human spot checks. A person reads a sample of the judge's verdicts and checks they agree with it.
- A pass bar set in advance. The candidate has to match or beat the current model before anyone looks at the results.
The judge can be imperfect as long as it is consistent and pointed at what your members care about.
The result: the switch was blocked
The cheaper model scored worse in the blind test. It had looked fine in the demo and it was worse across the golden set, so it did not ship.
Without the harness, the switch would have been an argument between "it looked fine" and a feeling. With the harness, the score settled it.
The harness exists because of this switch. We built it after the cheaper model was proposed, when it should have existed before the first prompt. That is the first thing I would change.
What the cost work did instead
The reason to try a cheaper model was cost, and cost was a real problem. Blocking the switch did not make it go away. We worked on cost per message without swapping the model doing the coaching.
Simple requests are routed to a small model, and the main model handles the conversations where coaching quality is the point. A simple request is one like logging a meal or asking for today's totals, where the answer comes from a tool and there is little coaching to do.
The parts of the prompt that repeat across turns are cached, so we stop paying full price for them on every message.
We also fail over between AWS Bedrock and the Anthropic API, which is about reliability more than cost. With failover, an outage at one provider means a slow afternoon.
The routing decision itself should go through the same harness: a request that goes to the small model is a model switch for that request, and it deserves the same evidence.
What I would change
Build the eval harness before the first model prompt. Built early, it gets used for every prompt change and every model release.
Instrument cost per conversation from the first week. The routing and caching work would have started months earlier with that number in front of us.
And treat a demo as a hypothesis for the eval to test.
If you are making the same call
If you need to know whether a cheaper model is good enough for your product, this is the work I do in AI operations.