Gitlab Case Study

How we cut GitLab Duo Code Review's cost by 98%

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

Back to Blog

-->

In Brief

We cut GitLab Duo Code Review’s cost by 98% on Martian’s Code Review Bench without losing review quality, and all of it came from optimizing the harness: which model each stage runs on and how much context each stage reads. Neither change gets there alone. The open-weight model stalled on a handful of oversized reviews, and compacting the context is what made them fit.

Figure 1. Cost against F1 for every setup in the results table. F1 stays within two points of the baseline throughout. The same compaction step saves 19% from the baseline and 84% after the model switch; on a log axis, arrow length reflects the percentage saved.

An agent’s cost is eventually set by its harness: which stages it runs, what each stage reads, and which model does the work. We took GitLab Duo Code Review, whose harness is source available, measured where its money sits, and, with our own agent helping run the experiments, changed one thing at a time until the bill stopped dropping.

But the real story is not the final changes in the harness; it’s the way we got there. Switching to a smaller model in every stage hurt the quality. Moving to an open-weight model on every stage cut 85% of the cost, but we didn’t stop there. While compaction looks marginal at first glance, the 98% only shows up with both, because the open-weight model’s deployment window decides which requests it can take at all. The model sets the price. The harness decides whether you get to use it.

We ran this on public pull requests, with no GitLab production traffic or customer data. Quality is F1 on one public benchmark, scored by two judges. Cost is token spend at published list prices, including for the model we serve ourselves, so it measures what the harness consumes. What our own infrastructure costs is a separate question. Configurations were chosen and tuned on the same 50 pull requests they’re scored on, so read these as measured results on this benchmark; how they hold on a new codebase is untested. The full methodology is at the end.

A pipeline with four stages

Duo Code Review , the Code Review Flow of GitLab Duo Agent Platform, reviews merge requests and posts comments. At the time of testing, it ran Claude Sonnet 5 by default, and GitLab published the flow, built on LangGraph, in its source-available ai-assist repository. That, plus a public benchmark, is why we chose it.

It runs four model stages, each with its own prompt and at most two tools:

Exploration lists the repository tree around the changed files.

File fetching reads related files: tests, callers, custom instructions.

Analysis turns what was fetched into notes.

Review reads the diff and most changed files in full, then posts comments.

Figure 2. Duo Code Review’s four stages. Each starts from a fresh prompt with at most two tools.

Every generic optimization we had built targets coding agents like Claude Code, which resend one growing conversation full of tool output and tool definitions. Duo Code Review’s stages are independent: each starts with a fresh prompt, carries at most two tools, and has no history to trim. Our defaults had nothing to act on and changed zero requests.

It turns out there’s no one-size-fits-all approach to harness optimization ; each harness requires its own careful analysis.

Where the money goes

Figure 3. Where the baseline’s $0.589 per PR goes. The review is 65% of the bill, split almost evenly between what it reads and how much it reasons.

The review is 65% of the bill, and its output alone is a third of the total: free to think before it posts, Sonnet 5 writes about 18,600 tokens per review, nearly all reasoning. Exploration and fetching, which only decide which files to read, are 27%. Analysis is 9%.

The money mainly goes in three places: the context every stage reads, the mechanical stages, and the review’s reasoning. No single change reaches all three but each step below reaches one of them.

Finding 1: less context, more reasoning

Can we cut what the review reads without hurting it? Yes, and it saves less than arithmetic suggests.

The review stage builds its context including changed files in full, plus a diff where every line sits in a tag repeating its type and both line numbers. A unified diff marks the same line with one character.

We compacted both in front of the flow, with rules rather than a model: nothing is summarized. File windowing keeps each changed hunk plus 25 lines either side of both the old and new line ranges, and elides the rest. Diff compaction rewrites the per-line diff as a standard unified diff with the same hunks and line numbers.

The median review prompt fell from 59K to 32K tokens. Spending on review context fell further, by 62%, because the largest prompts shrank hardest.

Yet the review’s cost fell by only a fifth. With compaction on, Sonnet 5 also wrote more : about 23,800 output tokens per review instead of 18,600, nearly all reasoning. We haven’t isolated why, but it caps what compaction can save on its own. On its own, compaction took cost from $0.589 to $0.475, a 19.4% saving, at 47.2% F1 and 1.16x the latency.

Figure 4. Baseline against compaction. Spending on the review’s context falls 62%, but its reasoning output grows 28%, so the bill falls only 19%.

Finding 2: the open-weight model held on every stage

How far does changing models get you? That depends on which stage you change.

We started by moving Claude Haiku 4.5 onto one stage at a time. On exploration and fetching, it picked the same files Sonnet 5 did. Early tests on the other two were less kind: quality dipped with Haiku on analysis and dropped sharply with it on the review. Same model, very different results depending on where it sat. Kept on the two mechanical stages, Haiku cut the bill 15.4% at unchanged F1, and 34.6% with compaction.

That was as far as Haiku could go. It couldn’t hold quality on the review, where most of the money is, and at half Sonnet 5’s price it wouldn’t have moved the bill enough if it had. Reaching the review’s reasoning meant leaving the Claude family.

We then turned to an open-weight model: DeepSeek V4 Flash, the first one we tried, which we serve with vLLM on H100s. At Together AI’s list price it costs 14x less than Sonnet 5 per input token and 36x less per output token. With all four...