K3 LoRA Training

Fireworks AI

Kimi K3 on Fireworks: Frontier Intelligence You Can Own

Blog

K3 Lora Training Make Kimi K3 Yours: LoRA Training on Fireworks

PUBLISHED 7/26/2026

Table of Contents First, a quick primer Cost Efficiency at Scale: Pay-Per-Token Training and Flexible Serving Why Fireworks Serverless Training How LoRAs learn reliability for your tasks

Task 1: teach K3 a new objective (Countdown)

Task 2: teach K3 to work through a problem step by step (Frozen Lake) Your Model Is Only as Good as Your Reward Try it

Beyond LoRA

Table of Contents

Kimi K3 is now available for Multi-LoRA serving and training in private preview with Fireworks Serverless Training. Kimi K3 is a massive mixture-of-experts model with roughly 2.8 trillion total parameters, the first open-weights model to reach the 3T parameter scale. Training a model this massive used to be a big project. As of today it is a one-line change on Fireworks. The secret? LoRA adapters: cheap to train, cheap to serve, and powerful enough for meaningful K3 post-training. With Fireworks Training, you can take the best open model in the world and cheaply teach it to nail your exact task, without paying frontier prices or touching a GPU cluster. First, a quick primer

LoRA (Low-Rank Adaptation) • Instead of training every single parameter of a model, you train a small side-car of weights (called an adapter ) on top of the model. • It's the cheapest way to train and easy to store: the adapter is only megabytes. • Best for behavior and style: personas, tone, format, structured outputs, a new objective, a tool-using loop. • Complexity is set by the adapter's rank (r) . Even a rank-1 adapter, a single direction of change, can meaningfully steer behavior, which is why LoRA punches so far above its size.

Serving a LoRA on Fireworks Once trained, an adapter loads into an inference deployment for evaluation or production. We offer two ways: Live merge folds a single adapter into the model at deploy time, so it runs at full base-model speed with no overhead. Multi-LoRA hosts many adapters on one deployment and routes each request to the right one.

Either way, the base weights sit in memory once, so with many adapters on top the marginal cost of each new fine-tune is close to zero. See the deploying LoRAs guide . Cost Efficiency at Scale: Pay-Per-Token Training and Flexible Serving

Fine-tuning a frontier model is genuinely hard. It takes work across three layers: the research itself, trainer implementation and optimization, and infrastructure and GPU management. Serverless training on Fireworks is your new infrastructure team, collapsing all three so a researcher can operate without a dedicated infra team behind them. There's no trainer to spin up and no reserved GPU cluster to book. There's no trainer to spin up and no reserved GPU cluster to book. The loop is yours: your code calls forward_backward, steps the optimizer, and samples your latest weights in the same session. Fireworks runs those operations on a shared pool of GPUs. You keep the training logic, we handle the hardware. Nothing to start, nothing to tear down. Nothing to start, nothing to tear down. "At HUD, we run agents across thousands of concurrent environments to produce the best post-training data for some of the most demanding labs in the world. That means the training layer underneath has to be best-in-class. Our training also comes in bursts as we're constantly iterating on environments and rewards, and the last thing we want is to babysit a cluster or pay for idle GPUs between runs. Fireworks serverless training meets both requirements: we point our loop at one endpoint, attach to a warm pool in seconds, and pay per token instead of by the reserved hour. We scale to whatever the workload requires without thinking about GPU clusters, keeping the team focused on environments and reward quality, where the real work is." Lorenss Martinsons, Founder & Co-CEO at HUD Change the rank (r) to match the adapter's capacity to your task, and iterate. Preview accounts run up to 8 concurrent training jobs with generous rate limits, so you can test several ideas in parallel. Why Fireworks Serverless Training

You pay per token. You pay per token. A small reinforcement learning run, roughly 20 steps and 860K training tokens, costs about $65 all in, training plus sampling, and finishes in 30 to 60 minutes depending on pool load. Reinforcement learning is the primary use case: sample from the adapter, score it with your reward, take a step, repeat. And with full KV-cache awareness, multi-turn agent runs cost far less, since repeated context bills at just 20 percent of the standard prefill rate. See the serverless training guide for more. When you outgrow the shared pool, the same SDK runs on dedicated training , the same path for full-parameter training, DPO, sustained RL, and explicit control over checkpoints, deployments, and lifecycle. We also guarantee consistent numerics alignment between training and inference across both serverless and dedicated, so what you see in a training rollout is what you get in production. When your model is ready, deploy it straight to inference with one click, with no re-platforming and no handoff. How LoRAs learn reliability for your tasks

To see LoRA in action, we picked two tasks that look like the work people actually ship. Then we trained a single small adapter on top of K3 for each one and watched it learn. No full-parameter run. No touching the base weights. Just a small adapter and a few dozen steps of practice. Task 1: teach K3 a new objective (Countdown)

A common reason people fine-tune models is to teach it how to achieve a goal efficiently using knowledge it already has. The Countdown task hands the model a few numbers (45, 14, and 39) and a target number (20). The model has to hit the target using arithmetic (45 - 39 + 14 = 20). The model is graded on its final distance from the target: partial credit for getting close, full credit for hitting it. The model already knows how to add and multiply. What it has to learn is the goal itself: keep combining numbers until you land on the target, then stop. K3 starts off decently, getting ~80% of the way to its goal in most cases, but it can do better. It just has to learn this new objective and within about ten steps it locks in and holds near a perfect solve rate. This task shows us that when a task is mostly about learning behavior over facts, a small adapter can...