First Scale Then Enrich How The Right Execution Strategy Helped Us Reach State Of The Art On Swe Rebench

First scale, then enrich: How the right execution strategy helped us reach state-of-the-art on SWE-rebench | AI21

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

In brief

We present a new state-of-the-art result on the SWE-rebench benchmark: a 60.9% issue resolve rate for 123 issues between December and March, which surpasses the previous best published result on this same slice.

This achievement came from rethinking the agent’s context extraction phase – a counterintuitive design choice we pursued and refined. Instead of applying an “enrich then solve” approach, where context extraction happens first and solution generation follows, we reverse the order: solution generation at large scale comes first, and then the generated candidates are exploited for better context extraction. Reaching state-of-the-art took more than the reversal itself – it required carefully optimizing how the extractor exploits those candidates to surface the most relevant code.

Our findings suggest that simple, initial parallel N attempts at solving a task can significantly aid codebase exploration, producing a highly targeted contextual map that can be utilized during the final evaluation step of the agent to generate more accurate output. This execution strategy noticeably improves the agent’s accuracy without driving up cost, and surpasses naive scaling of the generation step in its traditional order.

This work joins AI21’s broader line of agent-optimization research, alongside earlier case studies ( SWE-bench Verified , BrowseComp-Plus, Deep Research Bench ), each benefitting from a different strategy of the AI21 Maestro optimization approach. We hope our contributions around coding agents’ pipeline order can serve as a blueprint for architecting increasingly accurate and cost efficient AI software engineers.

In this post, we walk through each step from a baseline ReAct loop to this result – and show how we pushed accuracy higher while keeping total cost below that of a leading coding agent.

The architecture of agentic coding

​Agentic coding has rapidly become one of the most exciting and heavily researched frontiers in artificial intelligence. Moving beyond simple code-completion autocomplete tools, modern coding agents – like Claude Code, Codex, and Cursor – are designed to act as autonomous software engineers – reasoning through complex problems, navigating large codebases, and executing multi-step tasks to resolve real-world issues.

​The anatomy of a coding benchmark

​To evaluate these agents accurately, the industry has gravitated toward a standardized benchmark architecture. The setup typically looks like this:

​ Input: The agent is given a natural language issue description (such as a GitHub issue) describing a bug or a feature request.

​ Environment: The agent is dropped into a Docker container that houses the target code repository alongside the necessary runtime environment.

​ Execution: The agent interacts with this environment by executing actions – most notably, running bash commands through the terminal to search files, run tests, and edit code.

​ Output: The goal is for the agent to produce a solution, typically in the form of a git patch . The benchmark then evaluates this patch against a set of hidden unit tests to verify if the issue was genuinely resolved.

Establishing a baseline: A classic ReAct loop

​When building a coding agent from scratch, the most straightforward starting point is a classic ReAct (Reasoning and Acting) loop.

​To run this basic implementation on the coding benchmark environment, the agent is equipped with just a single, powerful tool: access to the Docker terminal . The agent reads the issue, reasons about what to do next, executes a terminal command (e.g., grep, cat, or pytest), observes the output, and repeats the cycle until it believes it has solved the issue.

To establish our baseline, we implemented this classic ReAct loop and evaluated it on the 12/15/2025–3/1/2026 evaluation window of SWE-rebench using GPT-5.2 as our generation model, set at medium reasoning. Under these conditions, the vanilla agent achieved a 53.8% resolve rate. We use SWE-rebench precisely because it’s built to resist the contamination and overfitting that affect older datasets – and the gains that follow come from general design choices that apply beyond this benchmark. While this is a decent baseline score, we’d need more sophisticated orchestration strategies to push our success rate further.

Reaching a higher resolve rate: Utilizing common optimization strategies

Researchers have proposed different ways to improve agent accuracy, such as context enrichment (gathering relevant code repository information), horizontal scaling (generating and evaluating multiple candidate solutions in parallel) and the combined benefits of an “enrich then scale” approach to agent architecture. We will consider each one separately below.

Strategy #1: Context enrichment

In a basic ReAct loop, there are no explicit steps; the agent operates with the single, broad goal of solving the issue. While this setup naturally leads the agent to perform some context-gathering actions on its own, starting completely from scratch makes navigating large codebases a massive hurdle.

Our experiments show that introducing a dedicated step for context enrichment at the very beginning improves results. It doesn’t mean the agent won’t need to gather more information on the fly, yet it begins from a more informed starting point than the basic ReAct loop implementation. When we introduced this dedicated step – architected as a short, focused ReAct loop powered by the same model with the sole goal of collecting relevant source code – we saw our baseline resolve rate rise from 53.8% to 55.6% .

Strategy #2: Horizontal scaling

Since LLM-powered coding agents are inherently non-deterministic, running the exact same agent on the exact same issue description multiple times can yield entirely different trajectories and results. An agent might get stuck in a loop on its first attempt, but hit the right solution on its second or third run simply by chance, a possibility we demonstrated in earlier work on SWE-bench Verified .

Horizontal scaling leverages this behavior by treating compute budget as a lever for accuracy. Instead of relying on a single, linear run, we scale...