Production verification is necessary but hard
An agent investigating a production issue does not get a clear feedback signal. It examines a crash-looping pod or a latency spike, settles on a root cause, recommends a fix, and finishes. Whether it was right is not yet knowable.
The agent’s self-assessment cannot be trusted. Even at its least confident, our agent gave itself a 60% probability of being correct. When we checked, it was only right in a third of these cases. Even worse, investigations assigned a confidence of 80% were no more likely to be correct than those with 60% confidence. “Is the diagnosis right?” is a hard question even for humans. Two engineers investigating the same issue often focus on different parts of the explanation. “Is it fixed?” is ultimately what matters more. If the fix Cleric prescribed was applied and the problem stopped, the diagnosis was very likely correct, or more importantly, useful. As a measure of the value delivered, this is hard to argue with, but tricky to build in practice.
Two things need to be verified: that the underlying problem (not just the symptoms) went away, and that Cleric’s fix is what made it go away. The evidence that settles these questions may take a week to arrive. Waiting this long may be necessary to account for the value delivered, but it is not sufficient to create the tight feedback loop analogous to tests for coding agents. Polling production with a frontier model for days is feasible but would cost much more than the investigation itself and would be slow to improve for the same reason; the metrics we want to track are delayed by days. So the design had to enable a faster way to iterate.
The verifier is a separate agent
When building out our investigation agent we tried many approaches to address the self-assessment problem. Prompt engineering helped to an extent, but quickly reached a plateau. Ultimately, we found an agent working toward one objective (reaching a confident diagnosis) performs poorly against a second, conflicting objective (flagging evidence gaps and uncertainty). So we adopted a multi-agent design: a separate auditor agent critiques the investigation’s process as it runs, checking whether the evidence cited actually supports the claims. Our investigation agent became much better at acknowledging uncertainty and user satisfaction grew. It was clear that whatever assesses a diagnosis has to sit outside the agent that produced it.
Verification follows the same principle: it is handled by a separate agent, one per issue. The verifier reads the investigation agent’s findings, but forms its own judgment from production evidence. Whenever the issue updates, it has two options. It can record a verdict (resolved, open, or undetermined), or it can decide more evidence is needed and write a check.
Checks are generated for each issue but run outside the agent
The monitoring period can run for days or weeks, but almost all of that time is uneventful. Only two steps in the workflow need intelligence: deciding what conditions to monitor, and interpreting a change when it appears. Everything in between is repetitive data collection. The verifier generates code dynamically for each condition it wants to check, and then a scheduler re-runs it every few minutes. The agent itself is re-invoked only when a check’s status changes, when the user engages with the issue, or when the monitoring window expires. Because the check re-runs so often, Cleric notices a resolution within minutes of it happening. And because the model is only involved when needed, the whole thing, even if it lasts a week, costs around 20% of the investigation it follows.
Dynamic code generation gives the agent flexibility to compose multiple data sources (e.g. kubectl and cloud CLIs, source code, network probes, data warehouses, third-party MCPs, as well as traditional observability), apply smart heuristics (e.g. baseline comparisons, seasonality adjustments) and express more complex conditional logic. The agent used this flexibility in surprisingly creative ways, often it worked and sometimes it didn’t. Over time we added guidance and guardrails to reinforce the successful patterns and avoid the observed failures.
This iteration cycle was faster because code generation has another advantage: it’s easier to test. Every check is validated before it is scheduled. It runs once, in the same sandbox it will run in later, and we can assert a number of invariants we know must hold. If the validation fails, the agent can correct its mistake immediately. Past issues, whose outcomes we now know, can be replayed through the verifier and we can evaluate whether the new or old checks would have caught the real-world state changes more accurately. The feedback loop that takes days in production can run in minutes.
How we decide “is it fixed?”
Another area that needed heavy iteration was the verifier’s rubric, the prompt that guides the agent on how to interpret new evidence and external state changes detected by its checks. We started with a loose definition of “resolved” and hoped the verifier would fill the gap sensibly. In practice it was fooled often. The rubric we use today was reached by iterating on each of these failure cases.
Focusing on deeper signals
An alert recovering proves little on its own: alerts go quiet all the time on traffic dips, time-of-day effects, and transient upstream load, while the defect underneath is untouched. This prompted us to stop treating “resolved” as a single layer. A production incident has depth: the alert on top, symptoms beneath it, and the defect the investigation identified at the bottom. The verifier considers the deeper layers first. This is where we add most value over traditional incident tooling that only sees whether the alert re-fired; the verifier monitors the specific anomalies and defects uncovered during the investigation.
Learning from experience
The failure that taught us the most involved the verifier’s own memory. Verdicts feed back into the system: when a new issue resembles an old one, the agents are shown how the earlier one turned out, and this feedback is a large part of why later investigations outperform early ones. It is also a potential channel for errors to compound. One recurring issue made this clear: a set of records stuck in an invalid state in a batch pipeline, with a recommended fix to clear them by hand. The symptom kept going quiet on its own, and on each recurrence the verifier interpreted the same quiet period a little more generously than the time before.
Comparing the four rationales side by side, each reads slightly more assertive than the last. The first notes that the symptom has gone quiet but gives the fix no credit; by the fourth, the verifier states with confidence that the proposed remediation had been applied and resolved the issue. No new evidence entered over that period. The production signal never changed, but the verifier’s context did: on each recurrence it could see its own earlier conclusion, and the nuance and uncertainty were gradually lost. The evidence against the fix was available the whole time: the same set of stuck records kept reappearing as an open problem, difficult to reconcile with a fix that worked.
Underneath it we found a standard-of-proof error. The verifier was accepting a quiet symptom, “balance of probabilities” evidence at best, in support of a fix-applied verdict that needs something closer to “beyond reasonable doubt.” We tightened the rubric in several ways:
- “Was the prescribed remediation applied?” and “did the issue resolve?” are now assessed on their own evidence before the verifier reasons about whether one caused the other.
- A remediation counts as applied only when the change itself is observed: the code or config that shipped, an audit or log signal, or a human confirming it in the thread.
- The verifier now infers patterns from a run of recurring issues rather than indexing on any single past verdict. The same fix marked applied while the same problem keeps recurring is evidence that the fix did not work.
- The verdict vocabulary now deliberately includes outcomes that give Cleric no credit: the problem was transient or the fix is unidentified. Positive verdicts are reliable because weak evidence can be recorded truthfully.
By the end of this process, “resolved via Cleric’s fix” had a precise meaning. An issue counts as resolved only when the underlying defect, not just the symptoms, is gone and stays gone. A fix is only considered applied when we directly observe the change taking effect. Whether Cleric’s remediation caused the resolution is a separate judgment, made only once both predicates are established on their own evidence. Pinning down these distinctions did not stay inside the verifier; it reshaped how Cleric models issue states and their transitions across the product.
What the outcome data revealed
We built our production verifier to get an accurate and automated measure of how often Cleric actually fixes things. This has provided a rich set of internal metrics that help us evaluate our changes, as well as a performance page within the product that helps account for the value we deliver. Much of what we learned from this data came as a surprise.
Our SRE agent is used in two main ways: interactive sessions, where Cleric and an engineer collaborate on a problem together, and alert-driven issues, where Cleric picks up an alert, posts a diagnosis and a fix, and often hears nothing back. Before we built the verifier, we had no indication whether those silent threads were even being read, and we were a little afraid of the answer. The outcome data settled it: among alert-driven issues, about four in five of the fixes that get applied land on threads with no human reply, and a thread that does get a reply is barely more likely to end in an applied fix than one that doesn’t. Engineers were reading the recommendations and applying them without ever replying.
Clusters of recurring alerts showed how to cut through alert noise to the real signal. Cleric already collapsed related alerts into a single issue, but only within a short time window. When the same underlying problem throws alerts that fire and recover over days or weeks, each burst lands as its own separate issue. The fix that would clear them ties these issues together. In one case, a single recurring fix connected over thirty separate issues. Instead of asking an engineer to review every issue separately, we can now surface a handful of distinct fixes for approval.
Groups of recurring alerts also provide data to validate our approach. When our fixes are applied, do these recurring issues actually stop? Of course the same alert can fire again for a different reason so we wouldn’t expect the alerts to disappear completely, but what we saw was reassuring. 64% of recurring alert clusters disappeared completely from our data after a fix was applied, compared to 22% when the fix was not applied. Before a fix, the average alert had an 18% chance of recurring on any given day. After a fix that fell to 7%. The effect was particularly evident for alert clusters with a higher baseline frequency. The counterfactual data, where the fix was not applied, showed no meaningful distribution shift.
These results were not perfect; although not common, some “fixed” issues did continue to occur. When we dug into the alert clusters that recurred despite an applied fix, we saw problems with both our suggestions, sometimes mitigations or workarounds rather than real fixes, and our verifier, for example equating “on main” with rolled out to production. Validating our outcome dataset against the actual stream of new alerts helped us tighten both areas.
A second dataset to correct our mistakes came for almost free. When a user doesn’t apply Cleric’s fix, the verifier keeps watching production, and it often sees what they did instead. Cleric proposed removing a hard-coded version pin; the engineer bumped it. Cleric proposed repairing a broken resource; the team deleted it. Each of these is a labeled example we never had to ask for, what Cleric recommended and what an expert actually did. We use this data to improve the quality of what Cleric suggests.
Finally, we learned that Cleric expresses uncertainty much more accurately than we expected. We knew that self-assessment did not work, but we were unsure how much our independent auditor agent was helping. Scored against the same set of verified outcomes, the auditor’s critique, grounded in the investigation’s evidence rather than its conclusions, predicted the true outcome far better than the self-scoring. The auditor’s grounded critique tracks the actual success rate closely, while self-assessment stays almost flat, largely unable to discriminate correct and incorrect diagnoses.
We still have room for improvement here, and it’s an area we’re continuing to invest in. Accurate confidence scores for each investigation are what build user trust, and as the predictions get more precise we will be able to catch and correct mistakes earlier, even before they happen. A dataset of issues and suggested fixes labeled with ground-truth outcomes opens up new ways to approach the problem. Improving these predictions is no longer a prompt-engineering exercise; it has become a traditional supervised learning problem and this has made our progress much faster.
Diagnosis is only the beginning
We set out to answer a narrow question: is this diagnosis correct? For an agent that diagnoses, it is the natural thing to measure. What we and our users actually cared about was: did the problem get fixed? Measuring that instead brought many benefits, a renewed confidence in our investigation agent and, where we missed something, a stream of labeled examples showing what we should have done instead.
After the success of this project, we started to ask bigger questions. A fix is only one kind of change that lands on production. Every PR is shipped with an expectation: needles to move and invariants that must not be violated. Everything we described here generalizes beyond fixes. An independent verification agent, cheap deterministic checks, a strict definition of “it worked”, and the eval iteration loop to refine it can serve as the foundation to verify any code or configuration change. For every PR we roll out, we can ask “did it do what it was supposed to do?”. We’ll dive into this in part two.