One of the hardest technical problems we have to solve at Cleric is how to find gaps in our knowledge about the customer’s environment.
Increasingly we’ve found that it’s these knowledge gaps that hold our product back the most. They are sins of omission, not commission—things like not knowing that the logs we have access to have already been sampled, and where to find the unsampled logs.
Our first strategy to find gaps was to analyze traces (manually and with coding agents), but this wasn’t enough. The main issue is that there’s no ground truth about investigation outcomes or what Cleric should have known during an investigation. It’s very difficult to tell from a trace whether an investigation succeeded, much less whether a specific knowledge gap contributed to its failure.
So we built a gap finder agent that runs inside the customer environment and periodically samples recent traces to find knowledge gaps. It’s aided by the resolution monitor, an agent that measures the outcome of a Cleric investigation, using signals like whether the suggested fix was applied. The gap finder then looks for suspected gaps that are more prevalent in low-confidence investigations than high-confidence ones. This helps us prioritize the gaps most likely to be affecting Cleric’s performance, so learning agents can work to resolve them, with targeted questions for customers when necessary.
-
Alert or question
-
Team of agents
Investigates
-
Resolution monitor
Follows the issue to its outcome
-
Confidence score
Was the diagnosis right, in hindsight?
-
Gap finder
What separates low- from high-confidence work?
-
Learning agents
Close the gap
- If they cannot close it
Ask the customer
In a chat
-
Environment knowledge
- …and back to the team of agents for the next investigation
How the gap finder works
The gap finder agent delegates to trace-analyst subagents, which read individual runs in parallel and report back. Typically we sample a number of low-confidence investigations at a time to generate candidate gaps.
The trace analysts inspect the reasoning during the run: What did the agent intend to check but never execute? What hypotheses did it leave open or mark as unverifiable? The same lead left open across many investigations is strong evidence of a gap. Runs that keep exploring (searching for the right repository, tag, index, or cluster) also indicate a gap.
The gap finder describes each candidate as a plain-language trace pattern that can be checked against other investigations. The gap finder evaluates each candidate using three sources of evidence:
- Live environment: Does the gap actually exist in the customer’s live environment? The agent tries to search for the missing information, and if it can find it, that indicates there’s no gap.
- Test run: Can the agent reproduce the failure mode in a fresh investigation?
- Prevalence: Is the suspected gap associated with worse investigation outcomes?
Scheduled review
Enough new runs since the last one
Read open gaps
From earlier reviews
List the agent’s recent runs
With confidence scores
Trace-analyst subagents
Read selected runs closely, in parallel
Candidate gap
Check it
Prevalence
How common, and does it separate low- from high-confidence runs?
Live environment
Does the fact exist, is the source reachable?
Test run
Does a fresh run without the knowledge repeat the struggle?
Survives?
Record a gap
Add a note
With this window’s evidence
Record nothing
And say why
These aren’t strict pass/fail criteria. The gap finder weighs the available evidence to decide which gaps are worth addressing.
Measuring the prevalence of a gap
A suspected gap might appear in hundreds of investigations without preventing Cleric from reaching the right conclusion. To prioritize the gaps worth resolving, we use a prevalence tool that compares how often the gap shows up in high- and low-confidence investigations. Doing that with a frontier LLM would be prohibitively expensive, so we use Jev, a smaller, calibrated model that can evaluate traces at a fraction of the cost.
As described earlier, the resolution monitor and Jev produce a confidence score for every investigation in the gap finder’s window. The prevalence tool draws a sample of up to 400 investigations per call, aiming for roughly equal numbers of high- and low-confidence runs (we exclude runs where the confidence score is uncertain). Then, for each sampled trace, we use Jev to estimate the probability that the candidate gap’s trace pattern is present.
Because Jev’s probabilities are calibrated, we can average them to estimate how prevalent the pattern is in each group. We’ve validated this calibration against traces annotated by hand.
The tool then runs a statistical test to determine whether the pattern’s prevalence differs between high- and low-confidence investigations. A pattern that’s significantly more common in low-confidence investigations is evidence that the suspected gap may be contributing to poor outcomes. If the test is inconclusive, we can’t confidently say whether the pattern is associated with better or worse outcomes.
The agent’s runs
In the window
Split by confidence
High-confidence quartile
Low-confidence quartile
Failed runs
Private runs
Tests, probes
Stratified sample
Hundreds of runs
Jev, per run
Probability the pattern is present
Prevalence
Mean probability per group
Verdict
Low minus high, 95% interval
The gap finder can use this tool iteratively to investigate different patterns and refine its hypotheses. It considers the statistical results alongside evidence from individual traces and probes of the live environment, using that combined evidence to decide which gaps to prioritize.
Closing the loop
When the gap finder identifies a gap worth addressing, it records and tracks it like an issue.
Each gap gets an ID (GAP-n), description, evidence, impact, priority, trace pattern. The next time the gap finder agent runs, it goes through open gaps and adds a short note with that window’s prevalence verdict and anything new.
Every gap gets a first pass from a learning agent, which can often close it from the environment itself: data, docs, code, or knowledge Cleric holds that never reached the runs. Probes then test whether the knowledge now works. Because the gap is stored, it can be measured again at any time. If the gap is no longer prevalent after it’s been addressed we can automatically mark it closed.
If (and only if) the learning agent can’t close a gap, Cleric reaches out to the customer via Slack, providing the evidence for why the gap matters. We’ve found customers respond well to these questions, because Cleric has already done the legwork.
We hold the gap finder to the same standard as any other agent. It has its own eval suite, with scenarios built from real (anonymized) production gaps. If a learning agent or customer can’t fix a gap, that tells us it was misscoped, misidentified, or not really a gap. These cases become lessons and eval examples for the gap finder.
What the gap finder is finding
The gap finder is already identifying a range of problems across customer environments and our own internal investigations. A few examples:
- Missing errors because of a logging convention. In one environment, Cleric queried logs using an uppercase severity value, but the stored values were lowercase. The query succeeded and returned zero results, even though relevant 504 errors existed. The gap finder identified the pattern as a potential reason Cleric was missing evidence during investigations.
- Missing a diagnosis that engineers had already made. In our own environment, Cleric repeatedly struggled to diagnose an issue involving event-loop starvation, even though our engineers had already identified the cause in previous pull requests. The gap finder found that Cleric was consistently overlooking this information across 22 investigations, suggesting that relevant knowledge wasn’t reaching the investigation agents.
- Conflicting information in Slack. In another customer environment, Cleric encountered contradictory information about how a system worked in different Slack messages. The gap finder identified the inconsistency as something that couldn’t be reliably resolved from the available evidence and needed clarification from the customer.
- Missing Kafka consumer group mappings. In one customer environment, Cleric struggled to find Kafka consumer lag metrics because consumer group names didn’t consistently match deployment names. The gap finder identified the missing knowledge and prompted Cleric to ask a platform engineer how to find the correct mappings. Within two minutes of receiving the answer, Cleric had verified it, successfully rerun the query, and saved the procedure for future investigations.
These examples illustrate how much of an investigation’s success can depend on seemingly small details about a customer’s environment. Cleric might have the tools and reasoning capabilities to diagnose a problem, but still struggle because it doesn’t know where to find the right information, or because the information it has is incomplete or incorrect.
That’s why we increasingly see knowledge as one of the biggest opportunities to improve our investigation agents. But knowledge is a hard, multifaceted problem. We need to establish that what Cleric knows is accurate, make the right information available to agents at the right time, and continually identify what’s missing.
The gap finder addresses that last challenge. By analyzing real investigations and their outcomes, we can identify what Cleric needs to learn and direct our learning agents toward the gaps most likely to affect future investigations.