How we made evals 100× cheaper with multi-agent architecture

We’ve been building Cleric for about three years now, starting with a single agent, initially based on LangGraph, and later switching to Claude Agent SDK.

Cleric’s investigation agent takes a system alert (service down, latency up, memory/disk pressure, and so on), plans an investigation, and queries observability systems and Kubernetes for logs, metrics, traces, pod statuses, etc. It then generates hypotheses, collects findings, and repeats this in a loop, each time going deeper, until it finds a root cause with evidence pointing back to the source systems.

In the last month, our agent has evolved in some interesting ways—the core difference being that it’s not one agent anymore. This post gets into why, what it unlocked, and how we’re thinking about the future.

Where it started: evals

Evaluations are the engine of agent improvement loop, and so from the early start of Cleric, we invested heavily in this area. Our evals were end-to-end investigations. We’d simulate a system failure with a known root cause, have our agent run a full investigation on it, and check whether the agent found that expected root cause.

Each run cost around $1.5 and took about four minutes to complete. The full suite cost $200 and took half an hour to run.

{
  "alert": "checkout-api p99 latency > 2s",
  "truth": "payments-db connection pool exhausted"
}

def test_payment_latency():
    env.setup("payment_latency")
    result = cleric.investigate(env.alert)
    assert result.root_cause.matches(env.truth)

This system worked pretty well for us for a while. It was great at catching regressions, and we’d run it on every major agent change to find any new issues before they reached our users. With this suite, we made steady improvements: tuning prompts, improving agent tools, and adding various capabilities like agent memories, knowledge, and skills. Little by little, quality was climbing up.

But we had some issues with this approach as well. First, it was slow and costly to test, because every agent change required full investigation runs to validate. This created a reverse incentive to work on the agent.

Second, whenever an eval failed, it manifested as a root cause miss. Understanding what actually caused that miss required a deeper analysis. For 150 eval runs, that’s an impossible task. The same was true for tracking what precisely improved with any given change.

Third, we were locked into one agent. Our tracing, cost tracking, and eval tooling were all built around the Claude harness and its native subagents, so trying anything else was a nonstarter. We’re big fans of the Claude agent, and still are, but an investigation requires many different kinds of work: planning and reasoning about hypotheses, but also fast, narrow tasks like reading logs or querying metrics. We wanted to experiment with other models and harnesses for some of these tasks, and our single-agent architecture didn’t allow for it.

We finally hit an inflection point when, a few months ago, we spent over $12,000 on evals in a single month. We knew this cost was only going to grow, and quickly. Something had to change. The more we thought about it, the more the lessons from pre-AI software architecture started knocking on the door. What do you do when your monolithic application becomes hard to reason about, slow to test, and risky to change? You break it up.

Breaking up our agent

A few months ago, we picked one, low risk area—the task of analyzing Prometheus metrics—and split it off into a separate agent. We moved it out of the Claude harness into ADK, a lightweight agent harness from Google, designed around very different principles than the Claude harness. We also picked a very different model, Gemini 3.5 Flash Light, which is built specifically for speed and trades off reasoning power, which we had a feeling the Prometheus analyst didn’t really require.

We then deployed it in shadow mode, running both versions of the architecture in parallel in production without affecting users, so we could compare the results side by side.

The new agent showed solid results on day one. In production, over the first ~50 paired investigations, the shadow ran at comparable quality with the main agent, but significantly cheaper and faster.

So we went all in, and over the next two weeks rearchitected our single Claude agent to a team of ~40 loosely coupled agents, which we refer to by “role” (analyst, judge, auditor, and so on), all orchestrated by a central investigator agent. We ran this architecture in shadow mode alongside our previous agent for a few days, and then gradually switched all user traffic to it. Our evals followed the same pattern: the previously chunky end-to-end evals were mostly replaced with smaller, more focused per-agent evals.

The unlock

The impact of this architecture change was dramatic. Testing a change became 100x cheaper and 5x faster. We can now eval a single change to an analyst in 30 seconds at a cost of ~1 cent. The whole eval suite now costs $1.50 and finishes in under 5 minutes.

Because evals became so much cheaper and faster, we started running a lot more of them. In mid-August we ran about 180 evals a week. When the cost per run fell off a cliff with the new architecture, we stopped rationing. We now run 5,000 evals a week (30x more), at about 8 cents each on average.

Cheap evals, in turn, unlocked fast iteration on the agent, which then translated into improvements in our KPIs. Both cost and time per investigation dropped significantly over the next few weeks, fueled by cheap experimentation. These two KPIs have been our main target, and so quality stayed roughly on par during this period.

Experimentation in practice

To provide a sense for how rapidly we can iterate under this new approach, here are walkthroughs of a few recent experiments.

Haiku 5.5 vs. Luna

Haiku 5.5 had just come out, and I was eager to take it for a spin. I picked one of the logs analyst agents and ran its evals using the current model, OpenAI’s Luna, and then the new Haiku.

Haiku was a bit cheaper and faster to run, but recall, which is key in this use case, was inconclusive. I thought this didn’t warrant a change, so Luna stays. Finding this out cost about a dollar in LLM spend and 5 minutes of my time.

Jev vs. Haiku

Here’s the same kind of experiment, for a different role: the intent classifier that’s responsible for routing users’ requests to the right agent. In this case, we tested Jev against the incumbent, Haiku.

Recall difference was inconclusive again, indicating a possible slight regression, but latency and cost are not: Jev is ten times faster and fourteen times cheaper. And for this role, we care about latency a lot. So this time the new model wins, and 15 minutes later, Jev is serving our production traffic.

We run similar experiments very frequently these days. Every role now runs on its own stack, each selected based on real evaluation results, measured with statistical rigor.

That’s our answer to “locked in” and “one tool for all the jobs.” Trying a new model or a harness for one role is a config change and a focused eval run for just that role. Initial results usually come in within 5–10 minutes and at a cost of a few dollars at most.

What’s still hard

There are tradeoffs to any change like this, and for us, three things have gotten harder.

• Complexity: We went from one agent to about forty, across two harnesses and several model families, soon adding more. All our tooling, from orchestration to tracing, cost tracking, and profiling, had to grow to match it. One investigation is now distributed across multiple traces that we need to correlate to understand what happened.
• Agent contracts: Agents talk to each other in prose, and prose can be ambiguous. Either the instructions one agent sends aren’t clear enough, or the response it gets back is unclear, and nothing in between enforces a contract. Every time we swap a model or a harness, we change one side of this conversation, so new miscommunications become more likely and these cases are hard to catch.
• Parts don’t equal the whole: Each agent working correctly doesn’t guarantee a successful investigation. Every analyst can pass its own evals and the investigation can still miss the root cause. So we still run end-to-end evals, just not for every change.

What we believe

We based this change on three beliefs or, seen another way, bets about what’s coming next.

First, designing for evolution: Off-the-shelf agents will keep getting better, and we don’t want to rebuild our system every time one improves. We want to plug it in, run evals within 5–10 minutes, and keep whatever wins. There’s a term for this: evolutionary architecture, meaning specifically designing the system for change as a first-class property. In the future, perhaps we’ll automate this process fully, so as new models and harnesses come out, the system picks the best one without our involvement.

Second, nothing works without evals. Cheap, focused evals are what enable you to respond to change. Evals play the role of fitness functions; but they’re only as good as how closely they represent the real world. We will continue investing here, to make sure our evals are representative.

Lastly, if models and harnesses are swappable, what is truly durable? We believe it’s the context: what our agents know about the environments, including the inevitable tribal knowledge and idiosyncrasies. That’s where we’re putting the most effort at the moment: building an autonomous system for agents to acquire knowledge about the environment they operate in.