Fine Tuning Bottlenecks

The Fine-Tuning Bottleneck Isn't the Algorithm

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Fine Tuning Bottlenecks The Fine-Tuning Bottleneck Isn't the Algorithm

PUBLISHED 3/28/2026

Table of Contents The Destination Is Always an Agent The Bottlenecks That Show Up Every Time

Integration and Data Sovereignty

Iteration Velocity Use the Right Tool for the Job The Ladder from Managed to Full Control Where We Are Heading Start Fine-Tuning Today

Table of Contents

Teams default to chasing the latest training algorithm, but the real bottlenecks are integration, iteration speed, and knowing when to reach for SFT vs. RFT vs. DPO.

TL;DR: Integration friction and slow iteration cycles are the bottlenecks that actually stall fine-tuning — not the algorithm. We share the patterns we see across engagements, how teams like Cursor and Genspark broke through them, and where the workflow is heading: toward fully agentic fine-tuning loops that close themselves.

Most teams that come to us for fine-tuning are not struggling with the training algorithm. They are struggling with everything around it: getting reward functions to talk to internal APIs without leaking data, waiting days between experiments because each step lives in a different tool, and figuring out whether the problem even calls for SFT, RFT, or DPO. Over the past year, working with a select group of the most innovative startups, digital natives, and Fortune 500 companies, we have seen these patterns repeat across every engagement. The Destination Is Always an Agent

Every team that comes to us for fine-tuning is building a domain-specific agent. Code fixing, customer support, deep research, financial operations — the use case differs but the shape is the same. A generic frontier model hits a quality ceiling and the path forward is model-level customization. The ceiling is concrete. Genspark 's Deep Research agent was stuck at a 0.76 reward score on closed frontier models. They moved to RFT on open models via Fireworks and pushed past 0.82 — a jump that prompt engineering alone could not deliver. One large digital native company we worked with saw a 30% increase in task quality and a 2.5x reduction in latency after fine-tuning with RFT. Prompt engineering can only get you so far, to reach a new capability tier you need fine tuning. Within a single account, we saw use cases ranging from escalation detection to reward modeling to AI-powered search — all running concurrently. That breadth inside one organization tells you fine-tuning is ongoing infrastructure for building agentic systems , not something you do once and move on from. The Agent Journey

Every team follows the same arc: a generic model hits a quality ceiling, fine-tuning closes the gap, and the result is a domain-specific agent in production. The Bottlenecks That Show Up Every Time

Across these engagements — different industries, model sizes, use cases — the same problems keep coming back. The interesting thing is that none of them are about the training algorithm itself. They are all about what surrounds it. Integration and Data Sovereignty

The most consistent blocker is integration. Reward functions, internal graders, and evaluation APIs have to stay inside the customer's environment. Sensitive business logic and proprietary data cannot leave for third-party scoring. Fireworks addresses this at two levels. For teams that need full data isolation, Training API lets you run training loops where the data never leaves your environment — you control the Python process, the data stays on your side, and only weight updates flow through the platform. For managed fine-tuning, secure bring-your-own-bucket storage and remote environments keep evaluators executing inside the customer's VPC. One team was constrained to specific non-Chinese open-source models for compliance. Model availability and geopolitical requirements shape the fine-tuning workflow just as much as the training algorithm does. The platform has to support a broad set of base models. Data Sovereignty Architecture

Reward functions, graders, and training data stay inside the customer environment. The training platform connects securely without data leaving the VPC. Iteration Velocity

The training job is rarely the bottleneck. Cycle time is. Teams spend weeks setting up training infrastructure, curating noisy datasets, running a job, then discovering through offline evals that the model still falls short on quality. By the time they have iterated on the data, adjusted hyperparameters, and retrained, another week has passed. The actual GPU time is a fraction of the total cycle. The teams that move fastest have collapsed that gap from weeks to hours. High-frequency job submission, fast feedback on what improved, minimal manual setup between experiments. One team ran over 100 jobs in the span of weeks. Another submitted dozens of RFT jobs with near-continuous iteration. These cadences look more like CI pipelines than ML experiments. Our work with Cursor on Composer 2 is the clearest example — Fireworks powered the distributed RL training infrastructure that helped Composer 2 beat top frontier closed-source models on coding benchmarks, with a new checkpoint shipping every ~5 hours thanks to the tight inference-training loop . Collocating fine-tuning and deployment on a single platform is what makes this possible. A trained LoRA adapter deploys in minutes — no model export, no separate serving stack. The eval-protocol CLI runs evaluations against the live deployment. The cost estimator lets teams plan iteration budgets before committing GPU hours. There is a hidden multiplier here too: hyperparameter optimization . Train/test split discipline, grid search, and learning rate tuning still have outsized impact on final model quality. Most product teams building agents do not have a dedicated ML engineer to get this right, and sloppy experiment setup is one of the biggest reasons fine-tuning "doesn't work." This is one of the areas where we think the tooling has the most room to improve — imagine a system that watches your eval metrics across runs and adjusts the training configuration automatically, instead of requiring an ML expert to babysit each experiment. We're building exactly this, and it's closer than you think. Iteration Velocity

The difference between teams that ship and teams that stall:...