Agent Execution Tax

Fireworks AI

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Agent Execution Tax Agents Don't Fail on Intelligence. They Fail on Execution.

PUBLISHED 5/20/2026

Table of Contents

What 720 browser agent runs revealed about the real bottleneck in agentic AI. Deployment Readiness Scorecard The Agent Execution Tax

Definition

Applied to Our Data

How the Tax Compounds

Generalised Formulas Structured Output Reliability: The Root Cause

The Data

Why This Matters More Than You Think

Reliability-Adjusted Accuracy

Why Nobody Measures This What This Looks Like in Practice How We Measured This

The Setup

The Models

Scope of This Benchmark Cost Per Successful Task > Token Pricing Inference Latency: The Compounding Story What the Fireworks Serving Layer Contributes Per-Site Analysis: Where the Thesis Holds or Breaks

Universal Success

Universal Failure

The Differentiators

Gemini's One Win: Google Flights What This Means for AI Procurement

Procurement Scorecard Model Profiles: Three Models, Three Strategies

GLM-5: The Reasoning Powerhouse

MiniMax M2.5: The Best Value

Kimi K2.5: Fastest Inference in This Benchmark The Vision Question

Kimi K2.5 Vision: Infrastructure Constraint, Not Model Limitation Closing Appendix

A. Reproducibility

B. Benchmark Configuration

C. Evaluator Methodology

D. Full Per-Site Breakdown (Numeric Reference)

E. Data Files

Table of Contents

What 720 browser agent runs revealed about the real bottleneck in agentic AI.

A Notte × Fireworks AI benchmark report.

Foundation models keep getting smarter. They ace reasoning benchmarks, write fluent code, and pass professional exams. Yet when you put them inside an agent loop, where they must observe a webpage, decide what to do, and output a structured action ten times in a row, they fail roughly half the time. We ran 720 browser automation tasks across four LLMs to find out why. The answer was not intelligence. It was execution:  one model wasted nearly 1 in 5 LLM calls on malformed JSON that had to be retried.  That single reliability gap cascaded into higher latency, inflated cost, and lower task success, even though the model's raw reasoning capability was competitive. We call this overhead the  Agent Execution Tax : the ratio of wasted inference to productive inference. For the worst-performing model in our benchmark, that tax was 22.9%. For the best, it was zero. In agent systems, reliability compounds harder than intelligence. The models that won were not the ones with the best reasoning scores. They were the ones that reliably did what they were told, every time, in the format they were asked for. In production, that reliability is shaped not just by the model itself, but by the inference infrastructure serving it: structured output consistency, latency predictability, and stable execution under repeated agent loops. At 10,000 agent tasks per day, a modest production volume, the execution overhead of the worst-performing model costs over $40,000 per year in inference that produces no value. A model that looks cheaper per token can cost significantly more per outcome once retries, failures, and inflated call counts are factored in. Scope.  This is a text-only browser agent benchmark. Results measure structured output reliability and step efficiency in a multi-step agent loop — not general model intelligence, reasoning ability, or multimodal capability. See Scope of This Benchmark below for the full scope statement.

Deployment Readiness Scorecard

If you are evaluating models for an agent deployment, here is how they map to production constraints. If you need... Use Why Maximum task accuracy GLM-5 57.1% accuracy; 100% on Google Maps, HuggingFace, BBC News, Wolfram Alpha; strongest on structured data extraction and multi-step reasoning Lowest cost at scale MiniMax M2.5 $0.062 per successful task (2.3x cheaper than Gemini); RL-trained agent that takes the fewest steps (9.8 avg) and rarely retries (1.6%) Fastest real-time response Kimi K2.5 2.1s p50 LLM latency; zero parse retries across 852 calls; best for user-facing agents where perceived speed matters Rigorous procurement evaluation Reliability-Adjusted Accuracy Token pricing misleads at the model selection stage; cost per successful task and execution tax are the metrics that reflect what you actually pay for

One-line summary per model: • GLM-5:  Best accuracy, highest cost. Use for compliance workflows, research automation, and tasks where errors carry downstream consequences. • MiniMax M2.5:  Best value. Default choice for scaled production workloads. The $40k/year waste calculation makes it the economically dominant option at volume. • Kimi K2.5:  Best speed, zero execution overhead. Use for customer-facing agents, live demos, and any workflow where response latency affects user trust.

The Agent Execution Tax

A browser agent task looks simple from the outside: go to Amazon, search for a product, extract the price. Under the hood, it is a multi-step loop: observe page → LLM generates action (as JSON) → execute action → observe new page → repeat

A typical task takes 10 steps. Each step is an LLM call that must return valid structured output: a JSON object specifying which element to click, what text to type, or what data to extract. If the JSON is malformed, the framework retries. And that retry is invisible: it does not show up in task success rates or reasoning benchmarks. It only surfaces as inflated call counts, latency, and cost once you instrument the engine itself. Definition

Agent Execution Tax = (total_inference_calls − productive_calls) / productive_calls

Productive calls  are those that returned valid structured output on the first attempt. The tax measures how much additional inference you pay, relative to the useful work done. Every percentage point is money spent on inference that delivers nothing. Note the denominator: this is not the same as the raw retry rate (retries / total calls). An 18.6% retry rate translates to a 22.9% execution tax because the denominator shrinks when you remove the wasted calls. Applied to Our Data

Model Productive Calls Total Calls Execution Tax Kimi K2.5 852 852 0.0% GLM-5 869 884 0.6% MiniMax M2.5 815 828 1.6% Gemini 2.5 Flash 721 886 22.9%

Measured on instrumented runs (90 tasks per model). Zero parse failures (exhausted retries) recorded across all models. For every dollar of productive inference Gemini...