Every AI system that improves over time has a way to tell a good answer from a bad one. Code has tests. Self-driving has miles per disengagement. An SRE agent has nothing like it in the moment. Confirmation comes later, and not on every incident. Did the alert stop firing, did the system recover, did the fix hold. When the agent finishes, that verdict is not in yet, so there is no ground truth to check the investigation against.
Our agent investigates production incidents: a pod is crash-looping, a latency alert fires, and it works out the root cause. When it reports that a pod restart came from a Kafka consumer rebalance, the same symptoms could equally point to a node running out of memory or a liveness probe set too tight, and two on-call engineers will often disagree about which.
What we ultimately want is a confidence number on every investigation, how likely it is to be correct. The agent needs it to know when to stop investigating, and the engineer needs it to know whether to trust the answer. Without it, the agent cannot tell when it is done, and the engineer cannot tell an accurate answer from a merely confident one.
That confidence number has to be learned from verified outcomes, the cases where we later saw the fix hold or the alert come back. Outcomes arrive days late and only for a fraction of investigations, so learning it can take weeks to months in every new environment. Because outcomes are slow, we still need to know whether the agent is improving and whether our changes help, and that needs a signal we can read on every investigation now, without knowing which one was right.
This post is how we built that signal without any labels at all. We score each investigation by comparing it against others, an idea from reinforcement-learning methods like GRPO. It works from the first week in a new environment, and it has caught production regressions that our other evals missed.
We compare two investigations of the same incident. One stalls and the other reaches a confirmed root cause.
Why the evals we already had did not fill the gap
Before the comparison scorer, we were already running three other evals, each covering a different kind of failure. None of them could tell us, week to week, whether the agent overall was getting better.
| Unsupervised | Every investigation | Env drift | Weekly trend | |
|---|---|---|---|---|
| Offline eval suite | ||||
| Invariant checklist | First 60s only | |||
| On-call ratings | ~2% rated | |||
| Comparison scorer |
Each existing eval leaves the weekly-trend column empty. Only the comparison scorer fills it.
The first is an offline simulation environment that runs the agent against a collection of chaos scenarios with known root causes. It catches regressions in tools, prompts, and reasoning we have seen before, but it cannot capture production. Each customer’s environment drifts in ways our offline simulations cannot anticipate, so an agent that passes every offline eval can still degrade the week a customer’s infrastructure shifts underneath it while the suite reports green.
Customer environments drift from the eval suite over time.
The second is an invariant checklist that fires at the start of an investigation: did the agent pick the right service, pull the right dashboard, narrow to the right time window. It is cheap and catches a real failure class at the front of the trace, but by minute three the investigation has branched into too many paths for a fixed checklist, and the failures that matter happen after the branching.
A fixed checklist only helps at the start of the investigation.
The third is the ratings and corrections the on-call engineers send back, the most direct feedback we have and the sparsest. Engineers rate only about two percent of investigations, and we cannot ask for more. The agent exists to take investigation work off the on-call rotation, so asking those same engineers to rate every one puts the work back on the people it was meant to relieve. Two percent is too little to teach an agent at any useful speed, so we use what we get to check that an automated scorer agrees with the people using the agent.
Each of these three either needs a human in the loop or misses what changes inside a customer’s environment, so none of them runs unsupervised on every investigation to catch a customer’s quality degrading.
Why we compare instead of grade
There are three ways to score an agent without a human in the loop:
- Keep an absolute LLM score and try to calibrate it.
- Run the agent several times on the same input and measure self-consistency, whether the runs agree with each other.
- Compare two investigations and ask which is better.
We had already shipped the absolute LLM score, so we know how it fails. An LLM judge rewards answers that look confident and complete. It is trained on human preference data, so it scores how an answer reads. With no ground truth for reference, almost every investigation landed in the 80-100% range, scored as near-perfect even when the conclusion was wrong, and the scores gave us no way to rank them.
With no ground truth for reference, the absolute scorer rates almost everything near the top and cannot rank.
Of the three, only comparison needs no ground truth while still separating better investigations from worse. Self-consistency rewards an agent that gives the same answer every time, even when that answer is wrong. Asking which of two investigations is better needs no absolute scale to calibrate, only a judgment of which did the better job: which one pursued the right evidence, which one needed the on-call engineer to step in and correct it. That judgment can be made without ever knowing the root cause.
Each comparison is one match. The matches fit a Bradley-Terry model that becomes an Elo rating.
We compare each investigation against a handful of recent investigations of similar incidents and ask an LLM judge which one is better. Each comparison is one match, and the matches accumulate into a tournament that gives a rating system the win-loss record it needs. We fit them with a Bradley-Terry model, the standard way to turn win-loss records into a strength estimate for each player, and convert that to an Elo rating, the same scale chess uses to rank players even when not every pair has played. An investigation never gets an absolute score, only enough comparisons to place it.
Each arrow points to the winner of one match, and a ranking emerges even though not every pair played.
We reached for comparison because relative scoring has worked well in other domains. RLHF trains reward models from human comparisons using a pairwise logistic preference loss, similar to Bradley-Terry. Arena, formerly Chatbot Arena, ranks chat models from pairwise votes the same way. OpenPipe’s RULER has an LLM judge rank groups of agent trajectories with no labels at all. What we had to build was running this on production SRE traces: multi-turn investigations, dozens of tool calls each, and no held-out ground truth.
What we built
What the judge reads
A trace is the full multi-turn investigation, far more than the final answer. It holds the agent’s messages, dozens of tool calls and their results, the subagents the main agent spawned for sub-tasks (a datadog-logs-agent or a grafana-loki-logs-agent, for example), and the on-call engineer’s follow-ups. The judge reads the whole trace because the conclusion alone does not separate two agents that took different paths to it. Both can reach “OOM in checkout-service from a leak in v3.4.2,” but one confirms it against a heap dump while the other guesses and gets corrected twice, and only the trace shows the difference.
The judge reads the whole trace, down to the tool calls.
We pick which traces to compare by embedding similarity, so the judge only ever sees two incidents that are alike, never a Kafka rebalance against an expired TLS certificate.
The rubric
The judge reads the two traces and picks the stronger one, by the criteria in the rubric below. There is no correct answer to score against. A few simplified examples of what we tell the judge:
- Completing the task is the most important factor. An investigation that produced an actionable explanation of the incident outranks one that produced none.
- Findings that are specific and grounded in evidence the agent retrieved beat generic or surface-level ones, and the more findings backed by real citations, the better.
- Penalize unproductive detours, speculation that runs past the evidence the agent gathered, and a conclusion that contradicts evidence the agent itself collected earlier in the same trace.
- Weigh the methodological choices the two agents made (in our testing the judge caught one agent using the wrong Prometheus rate window).
- Read the on-call engineer’s follow-up messages as signal. A correction from the engineer means the agent missed something a human caught.
The judge returns one of three calls: the first investigation is better, the second is better, or it calls a draw. That third option matters. The judge works from incomplete information, so a draw is often the right call, and forcing a winner onto two investigations that look equally good would only add noise to the ratings. We also check, on the synthetic scenarios where we know the right answer, how often it draws when it should.
Two levels of comparison
Comparison runs at two levels. At the investigation level it compares the main agent’s messages across two whole investigations. At the subagent level it compares two runs of the same subagent type. Splitting it this way keeps each match scoped, holding one level fixed instead of judging a whole branching investigation at once.
Subagent comparisons produce specific findings, because the task is narrow and the two runs are doing close to the same job. In our testing they caught:
- agents stopping at service boundaries instead of tracing errors across services,
- agents reporting error counts without computing rates or comparing against a baseline,
- agents not falling back to a secondary data source when the primary one failed.
Main-agent comparison is less precise and more prone to process-compliance noise, where the judge rewards following a procedure and overlooks the outcome itself.
Grounding the judge
The judge is itself an LLM, so before trusting its calls we had to ground them against something with a known answer. We do that two ways.
- On synthetic data we manufacture ground truth: we run the same eval test many times, and each run gets a recall score (how much of the known root cause it surfaced). The recall tells us which of two runs was the better investigation, which gives us a known ordering to hold the judge against.
- On production data there is no root cause at the time, so we pair similar real investigations and check the judge’s win, draw, and lose calls against the on-call engineers’ star ratings and against verified outcomes.
On synthetic data, recall against the known root cause tells us which run was the better investigation.
Every match, across weeks and customers, goes into one Bradley-Terry fit, so all of them land on a single Elo scale. That shared scale lets us read each customer’s trend over time, with an error bar wide enough that we do not over-read a single week.
Does it track quality?
Engineers rate about two percent of investigations, usually because something caught their attention, often a mistake. A rating is one engineer’s read of the investigation, and when they flag a mistake it is the most direct signal we get of what the agent got wrong. That the ratings cluster on bad investigations helps us, because those are the cases the scorer has to catch. If the judge agrees with the engineers on that rated two percent, it has shown it can recognize failure the way they do, which is the property we rely on when it scores the unrated ninety-eight percent. To run the check, we bucket the rated investigations by star rating and, for each bucket, compute how often the judge scored them a win, a draw, or a loss against the ones they paired with. If the judge tracks quality, lower-rated investigations should lose more of their matches.
For each star bucket, how often the judge scored those investigations a win, draw, or loss against their pairs.
Five-star investigations correlate strongly with the judge. They won half their matches and lost 11%. Four stars is by far the most common rating, and those matches drew most often, which is what pairing two adequate traces should produce. Ratings of one to three stars are thankfully sparse, so their split is noisy, but they lost 37% of their matches, more than three times the five-star rate. The worse an investigation looked to a human, the more often the judge ranked it below its peers.
That covers the production side of the check. The other side runs on our offline simulations, which do have ground truth. There, agreement means the judge ranked the pair the same way their recall scores do, with the higher-recall run against the known root cause counted as the better investigation. Since each call is three-way, win, draw, or loss, random guessing sits at 33%. The absolute scorer reproduced the recall ordering 27% of the time, which makes it indistinguishable from guessing: it scores almost everything near the top, so most pairs come back as ties, and ties count as misses. The pairwise judge, after tuning the rubric and what we feed it, reproduced it 67% of the time. That closes about half the gap between chance and a full-information oracle, which is about as much as we could have hoped for from traces with no ground truth. The judge ranks investigations against each other without ever verifying that the agent reached the correct root cause.
Share of synthetic pairs ordered like their recall scores: absolute scorer 27%, pairwise judge 67%, against a 33% chance baseline.
What it flagged over six weeks
We ran the scorer on a sample of production traces for six weeks, around a thousand comparisons drawn from roughly ten investigations per customer per week, out of the several hundred investigations a week the agent ran across those customers. It produced five weekly Elo snapshots, since the pipeline scores each week against the prior, so the first week has nothing to anchor it. The trends worth acting on showed up per customer, but the combined line washed them out.
The four customers shifted by very different amounts:
- Customer A came online during the window and climbed 246 points, mostly its environment stabilizing instead of the agent improving.
- Customer B gained 139.
- Customer C and Customer D each drifted about 40 points down over the window, inside the per-customer confidence interval, so we read them as flat.
Combined into one global line, the five snapshots ran 1475, 1426, 1491, 1530, 1555, an aggregate dominated by Customer A coming online.
Each customer’s Elo trend on the shared scale, with the combined global line dashed. Shaded bands are the per-customer 95% interval.
In week 2, global Elo fell from 1475 to 1426. We traced the drop to a handful of investigations in one customer where the on-call engineer kept correcting the agent, and instead of trusting the number on its own, we read those correction-heavy traces and confirmed a real regression, with the judge scoring them as losses against the stronger investigations they were paired with. None of the other three evals surfaced it. The offline evals had no test for the new kind of incident that customer was hitting, and the checklist passed because the agent’s first thirty seconds were clean before it went sideways three minutes in.
That is what the scorer is for. Running unsupervised on every investigation, it catches regressions like this one and points us straight at the traces to read.
We also point the scorer at our own changes. Running a new prompt or tool against the old one on the same incidents, the judge compares them head to head, and the biases the two runs share cancel out, so only the effect of the change shows.
What to know if you adopt this
Log exactly what the judge reads, then read it yourself. We had a bug where each follow-up from the on-call engineer stored its real text in a user_content field, while the content field the judge read held a chat-history XML blob. For weeks the judge scored against markup it could not parse, and the scores looked plausible enough that nobody questioned them. We caught it only by dumping the judge’s literal input and reading it.
Don’t hand the judge the agent’s system prompt. It is mostly process instructions, so the judge starts rewarding compliance, using the scratchpad, querying in the prescribed order, over the quality of the investigation. Both traces in a pair run under the same prompt anyway, so it says nothing about which one is better. Removing it raised the judge’s agreement with our ground-truth simulations from 60% to 67%.
Removing the system prompt raised agreement with the ground-truth simulations from 60% to 67%.
The judge penalizes the agent for things it never sees. Some events that affect an investigation never land in the trace. A runtime hook tells the agent to wrap up at around ten minutes, but that hook is not recorded in the agent’s messages. The judge then sees an investigation that stopped early and marks the agent down for ignoring its own process, when the system told it to stop. Until those events are logged as visible lines, the scorer blames the agent for the runtime’s choices, so we filter those pairs out.
Your confidence intervals are wider than your weekly deltas. At roughly ten scored investigations per customer per week, a single outlier swings that customer’s Elo by 50 points or more, so the 95% interval on a customer-week is wider than most weekly changes. When a customer sits near 1490 for three weeks, a single week that dips 30 points is almost certainly noise. Scoring more investigations per week is what narrows the interval.
About ten investigations per customer-week. A single outlier moves the Elo 50+ points. Production volume narrows the band.
The cost is modest. One comparison is a Gemini 3 Flash call over both traces, about $0.06. Scoring a full week of investigations at three comparisons each lands in the range of the absolute scorer we already run.
What it does not do
It is only as trustworthy as the pairing is comparable. The judge is close to apples-to-apples in synthetic scenarios and shadow runs, where we control which incidents get compared. In live production it is messier. The same alert in a different environment, or at a different time, is not really the same task, and the call gets less reliable the less alike the pair is. Repeated alerts and the narrower subagent comparisons hold up best for exactly this reason.
It catches mistakes and misses omissions. The judge is good at spotting an active error one agent made, a wrong rate window, a conclusion that contradicts the evidence. It mostly misses the steps an agent should have taken but skipped, unless the other investigation in the pair happened to take them. The road not taken does not show up.
The rationale needs aggregating. The judge explains every call in natural language, but turning hundreds of those into the handful of patterns worth acting on is separate work. It can be automated, but it does not come for free, and the system does not produce that report for you.
Why comparison is enough
None of this tells us whether the agent was actually right. That answer is the confidence number we want on every investigation, learned from verified outcomes. Those outcomes are the real ground truth, the fix that held or the alert that came back, and collecting them is its own problem, covered in How we verify Cleric’s production fixes. They arrive late and only for some investigations, so learning from them is slow, weeks to months of data in every new environment.
The comparison scorer runs on every investigation, with or without that number in place. It never claims an investigation was right, only that it beat or lost to the ones it was paired with, which is enough to score investigations from week one. The scorer caught a production regression that none of the other evals saw. We measure our own changes against it. And it is the independent check on the confidence model, so a model that has learned to game its own target still has to beat its peers.
If you run a multi-turn agent on tool-heavy traces with no ground truth, and user feedback too sparse to count on, a pairwise relative scorer will tell you where to look. It works because comparison asks for less than scoring does. A judge can reliably pick the better of two runs well before it could put a trustworthy number on either one alone.