Making Kimi K3 Tokenization 18x Faster For Million Token Agentic Workloads

Making Kimi K3 tokenization 18x faster for million-token agentic workloads Kimi K3 is here. Try it now

Model performance

Making Kimi K3 tokenization 18x faster for million-token agentic workloads

We built the Baseten Tokenizer (Basetenkenizer) to optimize tokenization for long input sequences, starting with Kimi K3.

Authors

Michael Feil

Last updated July 27, 2026

Share

For years, inference engineers have been able to disregard tokenization time as negligible. Short input sequences take a couple of milliseconds to tokenize, a tiny fraction of the time required for prefill and decode. This has changed. Open frontier models like Kimi K3 support input sequences of up to one million tokens. These long input sequences are increasingly common in agentic workloads, where an agent loop repeatedly appends tool results, observations, retrieved documents, intermediate state, reasoning traces, and updated instructions back into the next request. ✕ Input sequence lengths and prefix cache hit rates increase quickly in agentic loops We built the Baseten Tokenizer (Basetenkenizer) to optimize tokenization for long input sequences, starting with Kimi K3. Within the Baseten Inference Stack, we migrated from a custom Python tiktoken implementation to the Rust-based Basetenkenizer, for the day zero release. On the long input sequences where tokenization time matters most, the complete serving path is up to 18x faster than tiktoken while preserving exact token IDs. ✕ Baseten Tokenizer is more than 6x faster than tiktoken for short sequences and 18x faster for million-token sequences, with exact token ID parity. This post describes the implementation and the benchmarking process for Baseten Tokenizer. Why Kimi K3 tokenization is non-trivial Most LLMs use a byte-pair encoding (BPE) tokenizer wrapped in a standard chat template. This tokenizer design dates back to the earliest Transformer models like GPT-2 and RoBERTa. Kimi K3 ships with a tiktoken.model BPE vocabulary along with a regex pre-tokenization for structural tokens like , , , and . A Python chat renderer emits typed encode_segments, and long-text chunking behavior is inherited from the custom tiktoken path. In structural parts of the rendered prompt, a string like should encode as a control token. In user or tool text, the exact same string must be treated as ordinary text. For example, if a user writes: please explain the literal token The characters must not become a structural marker. It has to be encoded as normal user text. Losing this distinction changes the prompt. The old tokenizer handled this by rendering chat into segments: EncodeSegment(text= "" , allow_special= True ) EncodeSegment(text= "message role=\"user\"" , allow_special= False ) EncodeSegment(text= "" , allow_special= True ) EncodeSegment(text= "literal user text " , allow_special= False ) To maintain exact token IDs, we needed to implement the same behavior in the Baseten Tokenizer. To make it easy to compare token IDs with other libraries, we are also releasing a Kimi K3 tokenizer.json on Hugging Face today. One native call for the whole serving path The old Kimi path looped over those segments in Python. It also reproduced tiktoken's long-input safety rules — splitting text into at most 400k characters, then splitting runs of whitespace or non-whitespace at 25k characters — and finally assembled Python lists of token IDs. Every segment and every chunk paid for a trip across the Python/native boundary. Baseten Tokenizer accepts the ordered (text, allow_special) segments in one native call, returning the tokens in an array interface: encoding = tokenizer.encode_segments( [(segment.text, segment.allow_special) for segment in segments], add_special_tokens= False , tiktoken_safe= True , )

input_ids = encoding.into_numpy(ids= True )[ "ids" ] The chat renderer stays in Python, outside the measured encode region. Special-token selection, safe chunking, regex pre-tokenization, BPE, ordered assembly, and the NumPy handoff all happen behind one native boundary. Serving code never materializes a million Python integers. Where the speed comes from Our implementation combines several optimizations: Specialized pre-tokenization scanners: Recognizes Kimi's split regex pattern ahead of time and dispatches to a handwritten function instead of a general regex engine. For those not interested in writing this, the Kimi model family's regex can be matched and reformulated for the PCRE2 JIT library.

A stack-resident BPE merge tier: Short pre-tokens of up to 32 bytes run in a stack-allocated linked list with a linear minimum-rank scan and dense first-round byte-pair table, avoiding heap and priority queue overhead.

Multi-core semantics: Identical pre-tokens are routed to the same CPU core on cold, long inputs, ensuring repeated words are merged once and served from cache before output order is restored. encode_segments has a limit of 400,000 characters, a legacy limitation of tiktoken. Basetenkenizer uses this as an opportunity to schedule each 400,000-character chunk on a separate core, making long-context ISL faster than Gigatoken.

Native typed segments and safe chunking: Per-span special-token policies and compatibility rules run in Rust, preventing the creation of Python work lists.

Zero-copy NumPy ownership transfer: The token array is handed to Python without intermediate lists of integers, and encodings are only materialized when accessed.

Smart-pointer PyO3 bindings: Uses abi3-py310 bindings, with strings borrowed as Cow values in Rust to improve efficiency.

These improvements compound on long context and multiple turns. Benchmark results for Kimi K3 serving We evaluated this performance gain against the workload profile it is designed to help the most: a long input sequence with a high prefix cache hit rate. ✕ When prefill is fast due to cache hit, optimizing tokenization noticeably lowers TTFT for long input sequences. This time savings compounds over multiple turns. The primary workload is a rendered Kimi K3 chat prompt containing 54 typed segments. Every measured round uses a new deterministic corpus of  the same character length, mixing prose, code, JSON, Unicode scripts, punctuation, numbers, unique request IDs and hashes, and literal K3 control strings inside user text. No measured prompt is encoded twice, so BPE caches start cold and only benefit from naturally recurring vocabulary — as they would in real traffic. Basetenkenizer parallelizes...