Agentic Kernels In Production

Agentic kernels in production Try GLM-5.3 today. Frontier intelligence at a fraction of the cost. Here

Model performance

Agentic kernels in production

Baseten's agentic kernel optimization framework cuts latency by 42.3% on Qwen-Image and by 15.2% on FLUX.2.

Authors

Brian Li

Faraz Shahsavan

Pankaj Gupta

Last updated August 28, 2026

Share

TL;DR We’ve built an agentic kernel development framework that identifies model-level optimization opportunities, generates improved kernels, and validates them in our serving stack. On our current models, we’ve improved end-to-end latency by 42.3% on Qwen-Image, 15.2% on FLUX.2, and a 5.5% increase in tok/s on MiniMax M3.

We’ve seen in recent years that agents have become surprisingly capable at kernel development, from ideation to generating kernels from scratch. Existing benchmarks such as KernelBench have made it easier to evaluate how well agents can optimize kernels on isolated general-purpose problems. However, there’s a gap between winning a kernel benchmark and shipping optimizations into production. A few reasons why: The best kernel configuration depends on the production workload. The kernel that wins on a general benchmark may lose on a specific deployment. Optimizations such as tile shapes, warp-specialization strategy, and CTA configurations respond differently to changes in tensor shape, batch size, sequence length, etc. Kernels like MoE and Attention make this especially visible.

A faster microbenchmark doesn’t necessarily translate to a faster model. Once your changes are integrated, interactions with downstream dependencies like CUDA graph capture and multi-stream execution can wipe out kernel-level gains or even result in a regression.

Optimizing kernels individually can miss higher-level opportunities. End-to-end traces often show that only a small subset of kernels have headroom for improvement. The lower-effort wins may come from restructuring the computation around them: fusing operations, eliminating redundant work, or removing pipeline bubbles.

Integrating a new kernel into a production serving engine is nontrivial. Unlike modifying a standalone torch model, serving engines have interconnected execution paths and dependencies. New kernels must be wired into the correct path, replace existing computation cleanly, and remain compatible with the surrounding runtime.

With this in mind, we’ve created a solution that bridges the gap between benchmarks and production. Given a model and serving engine, our framework can profile the full workload, reason about the best optimizations, and then generate and ship those kernels straight to production. The stack: model-level + kernel-level optimizations The optimization stack divides into two layers: ✕ Two-track optimization pipeline: model-level work profiles, ranks, and tests candidate changes for end-to-end latency improvements, and kernel-level work benchmarks kernels across variants. Both feed into engine integration and production. The dotted testing loop (microbenchmark, correctness check, ablation tests) determines whether a candidate gets archived or recorded as a dead end. Model-level optimization: Understands the full model workload, profiles where time is spent, and proposes changes such as fusion and redundant work elimination.

Per-kernel optimization: Takes generated and other performance-critical kernels identified in the trace, explores several implementations in parallel, and iterates on the strongest candidate.

The first layer helps expand the search space beyond one-for-one kernel improvements. Rather than only optimizing kernels in isolation, the framework can restructure the execution graph by removing redundant work, reducing intermediate materialization, or combining operations before generating and improving the underlying kernels. Learning across optimization runs Our framework also has a self-improving mechanism: kernels that pass correctness and end-to-end performance checks are retained as reusable candidates, while lessons from both successful and failed attempts are added to an evolving knowledge base alongside workload constraints and integration findings. This creates a self-improvement loop where each optimization iteration starts from accumulated experience, enabling the agent to generate stronger candidates and converge faster over time. ✕ Persistent knowledge architecture: successful optimizations get stored in the kernel database with their patch, test cases, benchmarks, and workload data, then feed into the next optimization loop. Both successful and unsuccessful optimizations get summarized into the knowledge base, with successes captured as reusable patterns and failures captured as caveats and root causes. Results + case studies Our initial experiment targeted diffusion models , namely Qwen-Image and FLUX.2 served with SGLang on B300 GPUs. The optimizations highlighted below were identified, proposed, and implemented entirely by our agentic framework. ✕ Median per-step denoising time across four model configurations (FLUX.2 FP8, FLUX.2 NVFP4, Qwen-Image FP8, Qwen-Image NVFP4). Every model gets faster moving left to right, with Qwen FP8 showing the largest drop, from 245.6 ms to 141.8 ms. Optimizations on both models Optimization #1: Pre-packed FP8 scales The FP8 paths in Qwen-Image and FLUX.2 were wasting launches converting scale metadata into DeepGEMM’s required format before matrix multiplications. Constant weight scales were repeatedly repacked through sequences of small kernel launches. We eliminate this overhead by changing the main FP8 activation producers to emit packed scales directly while also moving weight-scale packing to model load time. The numerical computation is unchanged, so outputs remain bit-identical. For example, at FLUX.2 attention projections: ✕ Baseline ✕ Optimized Another example at the Qwen-Image feed-forward layer: ✕ Baseline ✕ Optimized The optimization reduced end-to-end latency by 7.3% on Qwen-Image and 6.1% on FLUX.2 , with these gains persisting throughout the subsequent FP8 optimizations. Optimization #2: Fused QKV projection and epilogue Both models’ original attention paths compute the image query, key, and value projections independently, despite them all using the same input. This resulted in repeated activation quantization and GEMM setup throughout every attention block. The optimization merges the three FP8 projections into one GEMM, then fuses bias addition, QK normalization,...