The Verifiability Litmus Test For Agent Design

The verifiability litmus test for agent design

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

Back to Blog

-->

In Brief

Before spending more compute on your agent, ask this first: Is your agent able to verify its own outcome (i.e., can your agent recognize when it has the right answer)? For some tasks, verifiability is straightforward; the task hands you conditions to check candidates against and make a stopping decision. For others, it’s less clear. In this post, we discuss why that single property – verifiability – should dictate where you spend your budget next: on verification when completion is checkable, and on diversity and aggregation when it isn’t.

The case against scaling first

Poll most engineers about how to improve an agent and most will say: scale it. More attempts, longer and enhanced context, a stronger model. It assumes the current architecture is already right – just underpowered. Most of the time, scaling works. But it’s the expensive way to go.

Over a year of agent optimization research, we have seen over and over again why this assumption (right architecture, not enough power) is wrong. The giveaway is discovering inefficiencies in the architecture: applying a uniform budget to variable tasks, generating signals and rollouts only to discard them, running a frontier model when an open model would do just as well. Each of these moments indicates there’s room to further optimize the agent’s architecture before scaling.

There’s a simple litmus test: can a candidate be checked against anything? That’s a property of the task, and it forks your architecture. If yes, invest in verification and selection. If no, invest in diversity and aggregation.

Then a second question, about your pipeline rather than the task: is it already exploiting the check available to it? That one we answer with an oracle experiment.

Figure 1. Decision framework for agent architecture. The governing question is whether a candidate answer can be verified, a property of the task rather than of the model. Verifiable tasks favor investment in verification and selection; unverifiable tasks favor diversity and aggregation.

A verification mechanism buys you one thing: permission to commit to a single candidate and throw the rest away. When you can’t build one, diversity and aggregation are the proxy: run varied attempts, then combine them, because without a check you can’t safely commit to any single one.

In this post, we use this litmus test to examine agent pipelines across four domains – agentic search, deep research, RAG indexing, and agentic coding – using our answers to optimize existing architecture and lift performance on each one.

Task #1: Agentic Search

Can the agent tell when it has the right answer? Yes. The answer is a single entity, and the conditions are explicit.

A multi-hop agentic search question specifies its answer through conditions, and the answer is a single short entity, such as one person. That shape is what makes checking efficient. Verifying one entity against stated conditions covers a far narrower space than the original search did.

Change the shape and the answer changes with it. Ask instead for every Nobel laureate whose mother didn’t finish high school, and each candidate you find is still checkable, but knowing you’ve found them all isn’t, because confirming completeness means re-running the search. Precision stays verifiable, coverage doesn’t. That same split returns in Task #4.

Before building anything, we ran an oracle experiment: we made the selection based on the ground-truth answers, creating a ceiling no runtime system could beat, since no real system actually gets to see them. It’s easy to do – and it disqualifies strategies fast. For instance, we saw that pass@k sits far above pass@1, suggesting that a correct answer routinely shows up across many attempts, yet any one single attempt usually won’t be the one to find it.

Knowing that the answers were already in the pool – just not chosen – helped us direct our focus to verification. We demonstrate two ways to improve verification signals, paying only when quality warrants it:

The free signal waiting to be utilized. On BrowseComp-Plus, models’ own confidence scores track accuracy well enough, once calibrated, to make a selection. Paired with an ensemble of models that fail on different queries, this reached 95.18% accuracy and state of the art .

The signal worth paying for. Confidence is self-assessment, so it adds no new evidence about the world, and models are perfectly capable of being wrong and confident at the same time. That’s tolerable when generators are strong and well calibrated.

We saw that when testing ensembles of small models on FACTS-Search. Turns out, small models frequently produce wrong answers – a problem majority voting can’t overturn as long as the wrong answer is a popular one. So instead, we swapped confidence scores with an independent verifier that re-researches each candidate from scratch and vetoes the wrong ones.

Training our own 8B verifier, we matched the quality of a frontier verifier on an open + closed model ensemble – at a 3.2x lower cost. Applying the same verifier to an all-open-source ensemble, it lifted accuracy 28% over plain majority voting on identical candidates – more than 80% of what a frontier verifier got on that pool, for roughly 1/100th the price.

Figure 2. Majority voting versus verification prior to voting, on a single query. Six sampled attempts yield three distinct answers (A ×3, B ×2, C ×1), of which only C is correct. Majority voting selects A. An independent verifier rejects A and B before aggregation, and voting over the surviving candidates selects C.

Task #2: Deep Research

Can the agent tell when it has the right answer? No. There is no way to know what a complete report should contain.

A deep research task asks an agent to investigate an open question and produce a report. What makes a report good is coverage: the set of relevant facts, findings and caveats it surfaces. Nobody can write that set down in advance. Compare Task #1, where the question’s own conditions were the enumeration, here the target is a set no one can enumerate before the work is done.

That splits the task in two. Any individual claim in the report is checkable, the same...