Your LLM judge might agree with your experts less than a coin flip

Most people think the hard part of an eval is over once the LLM judge prints a score. It is not. A judge that agrees with your human experts less often than a coin flip is not measuring quality, it is manufacturing confidence you have not earned. We watched a talk by Hamel Husain with Lucas Rocha, who runs AI at Nova Escola, a Brazilian education product where an LLM writes lesson plans teachers actually trust, and it lined up almost exactly with how we think about evals on client builds. This is the method, the traps, and where we apply it.

The credit for the method here is theirs. Rocha took Husain's evals course and rolled the practice out across a 29-person company with six people in technology. What follows is our reading of what worked, with the parts we already do on GattyWorks builds called out as we go.

A score is not the same as a judge you can trust

The core move in a modern eval is LLM-as-a-judge: you have one model grade another model's output against a rubric. The trap is treating the judge's score as ground truth the moment it appears. It is not ground truth until you have checked that the judge agrees with a human who knows the domain.

The way you check is boring and non-negotiable: hand-label a set of outputs yourself (pass or fail), run the judge on the same set, and measure two numbers. True positive rate: of the outputs a human called good, how many did the judge also call good. True negative rate: of the ones a human called bad, how many did the judge catch. In Rocha's case, on some criteria two human annotators disagreed with each other more often than random chance. If your humans cannot agree, no judge trained on their labels can either.

# The only thing that turns a score into a judge you can trust.
# labels: what a domain expert said. preds: what the LLM judge said.
tp = sum(1 for h, j in zip(labels, preds) if h == 1 and j == 1)
fn = sum(1 for h, j in zip(labels, preds) if h == 1 and j == 0)
tn = sum(1 for h, j in zip(labels, preds) if h == 0 and j == 0)
fp = sum(1 for h, j in zip(labels, preds) if h == 0 and j == 1)

tpr = tp / (tp + fn)   # judge catches the good ones
tnr = tn / (tn + fp)   # judge catches the bad ones
# If either is near 0.5, the judge is guessing. Fix the rubric, not the score.

Lesson 1: instrument before you evaluate

You cannot evaluate what you cannot see. The first thing Rocha's team did was set up observability with Langfuse: full traces of every run, which tool got called, what the RAG step retrieved, what context went into the prompt, what came back. Before that, getting raw conversations out of the database meant an engineer exporting a CSV from Postgres by hand.

This is the same order we work in. On a build with any LLM feature, tracing goes in before we write a single eval, because a failing eval with no trace behind it tells you a number and nothing about why. If you are wiring an AI feature and have not decided where the traces land, that is the first task, not the last.

Lesson 2: do error analysis before you write a single eval

The most expensive mistake Rocha's team made early was labeling data for problems that were not problems. They had a pedagogical rubric, so they graded everything on it, including things like whether a lesson plan's title was too generic. That turned out to be a trivial fix that needed no eval at all: they spotted it, corrected the prompt, and it stopped happening.

Evals are not free. Every one you write is data to label and a judge to calibrate and maintain. So look at real failing outputs first, cluster them into actual failure modes, and only build evals for the failures that are both real and hard to fix by hand. Husain's rule of thumb, and ours: a persistent error you do not know how to fix is the clearest sign you need an eval. A one-line prompt fix is not.

Lesson 3: one expert who decides, not two who argue

Rocha's team put two annotators on the same data and watched them disagree, badly, on nuanced education criteria. Two people labeling the same traces sounds more rigorous. In practice it is tremendous friction: now you need a third process just to resolve the disagreements, and the disagreement itself was a symptom of a rubric that was not specific enough.

The fix is a single domain expert who owns the definition of good, what Husain calls the benevolent dictator, plus a rubric sharp enough that two people would land on the same pass or fail. When your annotators disagree, do not average them. Ask why the rubric let them read the same output two different ways, and tighten it.

Lesson 4: split your data, and never calibrate on the training set

This one bit Rocha's team on camera. They were measuring how good a judge was using data that overlapped with the data used to build it. A model checked its own homework and reported a flattering score. The moment they cleaned the training data out of the evaluation split, the real number dropped and the actual state of the judge became visible.

Split your labeled data into train, dev, and test, and keep the calibration honest by measuring the judge only on data it never saw. Watch balance too: a lopsided set (mostly passes, few fails) makes a judge look good while it quietly misses every failure. The exact ratio matters less than the discipline of not grading on the training set.

Lesson 5: judges belong in CI and in production

Once a judge is calibrated, it earns its keep in two places. Rocha runs his calibrated judges as a CI gate: every PR that changes the agent runs the judges, and the question is whether this version regressed on any criterion. And he samples production daily, in batch, at whatever hour is cheapest, to watch for drift, because a model that passed last week can quietly degrade on inputs it was never tested against.

One detail worth stealing: pin the judge model. He calibrated with a strong model (a GPT-5 tier) for judging and a cheaper, smaller model for generating the lesson plans, and he uses the exact same judge model on every later run, so the TPR and TNR he measured still apply. Change the judge model and your calibration is stale.

The production sampling has one gotcha he hit: 2 percent of daily traffic sounds fine until you realise it gives you almost no coverage of the long tail, third grade, or one specific subject, so the scores swing wildly day to day. He had to push the sample rate up (toward 80 percent in his case) before the numbers were stable enough to read. Sample enough to be stable, not just enough to feel principled.

How this shows up in our builds

Our third pillar is AI workflows and custom agents, and we ship LLM features into products under a hard turnaround. That guarantee is exactly why this method matters to us: a 48-hour MVP that ships a confidently wrong model is worse than one that ships slower and knows its own failure modes. So on any build with a judge in the loop, we keep the domain expert (often the client, who knows their users better than we do) at the center, trace before we evaluate, and refuse to trust a score we have not calibrated against a human. The tools change, the discipline does not.

Next step: if you have an LLM feature scoring itself and you are not sure the judge agrees with a person, run the twenty-line TPR/TNR check above on fifty hand-labeled examples before you trust another dashboard. If the judge is guessing, you want to know now. If you want a second pair of eyes on an AI feature you are shipping, write to us at hello@gattyworks.com.