How To Build A Day Zero Api For Kimi K3

How to build a day-0 API for Kimi K3 Kimi K3 is here. Try it now

Model performance

How to build a day-0 API for Kimi K3

Baseten has day-0 support for Kimi K3 on our Model APIs. Kimi K3 is a new 2.8T-parameter open frontier model.

Authors

Model Performance Team

Last updated July 27, 2026

Share

Baseten has day-0 support for Kimi K3 on our Model APIs. We want to thank Moonshot AI for sharing the Kimi K3 weights with us for early access, as well as the Inferact and RadixArk teams for their collaboration throughout the development process. ✕ Kimi K3 is available today on Baseten Model APIs with vision input and full 1M-token context window. Kimi K3 is a new open frontier model. At 2.8T parameters, it is much larger than any previous open model, introducing a number of challenges to building a performant inference API. New architectural techniques allow Kimi K3 to scale beyond the trillion-parameter threshold of prior frontier open models: Kimi Delta Attention (KDA) and Attention Residuals (AttnRes) as a scalable backbone for the Kimi Architecture.

Extremely sparse experts, with just 16 of 896 experts active at a time, organized using Stable LatentMoE.

A new vision encoder for processing image inputs and mapping visual information into latent space.

This article describes the technical work required to run Kimi K3’s novel model architecture and huge weights at scale in time for launch day. Milestone 1: Generate a token After receiving early access to the Kimi K3 weights from the Moonshot AI team, our first priority was to simply get the model up and running. Generating our first tokens of Kimi K3 required: Provisioning hardware: Given the size of Kimi K3, we decided to run the model on NVIDIA GB300 NVL72 systems.

Loading weights: In MXFP4, Kimi K3 weights are over 1.4TB of data.

Bringing up an inference engine: We worked with the teams behind vLLM and SGLang to run pre-release builds of the inference engines for Kimi K3.

Often, when building day-0 APIs, an early step is porting the weights to NVFP4 to improve performance and compatibility with NVIDIA Blackwell and our inference stack. However, Kimi K3 uses native MXFP4 weights with MXFP8 activations, and we were able to use these weights directly. With the weights in hand, we worked closely with Inferact on vLLM and RadixArk on SGLang. Before running Kimi K3 on the Baseten Inference Stack, we needed to establish a baseline in collaboration with the leading open-source inference engines. Adding support for a model to an inference engine is non-trivial. It requires implementing core modeling code, optimized kernels for new architectures like KDA, and building frontend compatibility for Kimi K3 across everything from tokenization to tool calling. vLLM’s early access image for NVIDIA Blackwell GPUs helped us establish basic feature completeness, pass initial evals, and set a baseline performance target. This work around accommodating Kimi K3 architectural features like KDA, AttnRes, and Stable LatentMoE formed a solid foundation for us to build on. We also did extensive validation of the vLLM inference engine for Kimi K3, and made contributions back to the open-source engine based on our work. SGLang’s early access image provided a reference for fast, reliable serving of Kimi K3. SGLang has a history of strong support for vision language models, and Kimi K3 is no exception. With the RadixArk team, we focused on frontend compatibility and kernel optimization, and our engineering team contributed fixes for frontend bugs around tool call handling and structured outputs to support release readiness. Thank you to the Inferact and RadixArk teams for building side by side with us throughout the preview window. This work provided both an essential foundation for our Kimi K3 API and an opportunity to contribute back to the open-source community. Milestone 2: Validate the inference engine Kimi K3 is the smartest open model ever. It’s essential to actually deliver on that intelligence during inference. ✕ Kimi K3 benchmarks show strong performance on agentic tasks, which depend on accurate tool calls and high-quality model outputs. Quality validation can occur at different levels of rigor. Simple sanity checks, like calling the model with a known prompt or running a light benchmark like gsm8k or BFCL and checking if results are within the margin of error, are useful checkpoints during the development process to make sure that things aren’t going off the rails. But releasing a public API requires more rigorous benchmarking. The Moonshot AI team operates Kimi Vendor Verifier , which helps inference providers ensure accurate, high-fidelity serving of model weights. Passing Kimi Vendor Verifier was an essential early milestone in our API development, and it was an extremely helpful tool throughout the development process. There are lots of opportunities to mess up in serving models. While the popular narrative is that quantization is the root of all quality issues, in practice that is not true. Most quality issues, especially with tool calling and other structured model behaviors, come from the inference server frontend. ✕ The frontend sits in front of the inference engine and processes the inputs and outputs The frontend is deterministic code that runs on the CPU in front of the inference loop. It is responsible for accepting inputs and returning outputs. The frontend must: Operate and validate the API

Tokenize prompts and detokenize outputs

Render the chat template

Parse reasoning and tool calls

Format the output to ChatCompletions, messages, or another standard

These tasks are subtly different from model to model, and it is all too common to introduce bugs and performance degradation when building quickly for day-0 support. Robust checks like Kimi Vendor Verifier evaluate performance across common failure modes, like tool calling, to ensure the model is served with a high degree of fidelity across both the inference loop and the API surface. As we continued to develop the API, we used Kimi Vendor Verifier at subsequent milestones to ensure that performance optimizations had not introduced bugs that would degrade accuracy. Milestone 3: Find the right configuration Inference engines offer a wide range of configuration options to tune performance to different models, hardware, traffic shapes, and latency/throughput tradeoffs. These options, and the interactions between them, are complex. To discover the right configuration, we do a...