Fine Tuning Qwen3 Tts For High Quality Voice Cloning

Fine-tuning Qwen3-TTS for high-quality voice cloning Kimi K3 is here. Try it now

AI engineering

Fine-tuning Qwen3-TTS for high-quality voice cloning

A guide to fine-tuning Qwen3-TTS for high-quality voice cloning, comparing ICL, speaker-embedding-only, and fine-tuning approaches with a full training recipe.

Authors

Ian Carrasco

Last updated July 31, 2026

Share

Building expressive speech experiences that capture the mannerisms and nuance of human speech requires richer, more extensive datasets across a wide range of speech scenarios. Models like Qwen3-TTS already ship with a remarkable ability to perform zero-shot voice cloning from a few seconds of audio. But richer cloning capabilities require fine-tuning over a larger corpus of utterance-level examples. Closed-source providers typically call this "professional voice cloning," as opposed to "instant voice cloning." It requires a larger upfront dataset curation step, including sourcing a voice actor. We’ve published a modified version of the official published Qwen3-TTS training recipe in our ml-cookbook for creating single-speaker fine-tuned checkpoints on Baseten Training, which can then be easily deployed on Baseten Cloud as a dedicated deployment. Three approaches to voice cloning: ICL vs. speaker-embedding-only vs. fine-tuning Qwen3-TTS was designed with strong voice cloning capabilities in mind, but extracting the best quality and performance from the model requires fine-tuning. Qwen3-TTS has two primary instant voice cloning modes: in-context learning (ICL) and speaker-embedding-only. In-context learning (ICL) ICL mode tokenizes your reference audio and reference text and includes them as a prefix for every request. This teaches the relationship between reference audio and reference text at inference time, but it costs a considerable prefill overhead (~100s of tokens) for stronger voice conditioning. ICL mode can also cause speaker inconsistency due to inherent non-determinism in attention and the current prompt being generated. For longer, non-streaming generations, this is often a solid approach, since those cases avoid frequent reconditioning per utterance. Speaker-embedding-only mode Speaker-embedding-only mode passes your reference audio through a smaller speaker embedding model (ECAPA-TDNN) that produces a roughly 10-token embedding, added as a prefix before generation. This yields lower time to first audio (TTFA) at the expense of better speaker capture due to the reduced dimensionality of the speaker embedding. However, the smaller speaker embedding means lower prefill overhead, especially at higher concurrency. Fine-tuning The goal of fine-tuning is to capture a richer text-audio alignment relationship without the additional overhead of ICL-based cloning. We accomplish this by training on significantly more text-audio pairings via supervised fine-tuning (SFT) and conditioning the model to generate based on an accumulated speaker embedding across a subset of the pairs. ✕ The input "shapes" for the three voice cloning approaches: ICL mode stacks a speaker embedding, reference audio, reference text, and the text prompt; speaker-embedding-only mode drops the reference audio and text, using just the embedding plus prompt; and fine-tuning drops the speaker input entirely, since voice identity is baked into the model weights. Each step down removes conditioning inputs (and their inference overhead). Different prompt constructions based on cloning paradigm

Fine-tuning recipe Step 1: Building the dataset Performing TTS fine-tuning commonly requires a dataset of single-speaker (audio, text) pairs. You can produce these by having the target speaker read a pre-existing transcript, or by generating the dataset with an ASR model in the loop, which only requires audio as input. Given a long-form audio file, we transcribe it while simultaneously emitting character-level timestamps. These timestamps let us split the transcript and reference audio at closing punctuation, producing utterance-level paired training examples. We repeat this until the full audio is processed. For the published example, we used the LJ Speech dataset , which contains roughly 24 hours of single-speaker audio (but we fine-tuned on only about 1.5 hours as a more realistic starting point). ✕ The dataset-building pipeline: a long-form audio file runs through Whisper for ASR, which produces character-level timestamps used to split the audio at closing punctuation. The result is a table of utterance-level pairs, matching each transcript segment to its corresponding audio clip, ready for training. Step 2: Training the model Qwen3-TTS uses a multi-codebook approach to produce audio features that are then decoded into a waveform. Raw waveform audio is too high-rate to model directly; predicting it sample by sample would explode sequence lengths and compute. Instead, vector quantization is applied iteratively to the upstream residuals to produce a compact residual vector-quantized (RVQ) representation: 12 codec frames per second, each spanning 16 codebooks. As in LLM training, we use cross-entropy loss between the tokenized ground-truth audio and generated speech, split into two components: talker loss and sub-talker loss. Talker loss is the cross-entropy at the first codebook, which carries most of the phonetic and prosodic content. Sub-talker loss aggregates the remaining 15 codebook layers, which capture more fine-grained acoustic detail. The weighting of sub-talker loss relative to talker loss is tunable and an area for further exploration; our recipe used the upstream default of 0.3. We also updated how the target speaker embedding is derived. Instead of using a single reference clip, we compute it as a centroid: the frozen speaker encoder runs over 64 training clips, and the embeddings are averaged. A single clip is a noisy estimate of voice identity. Per-clip embeddings of the same speaker typically agree at roughly 0.7 cosine similarity with each other, but at 0.85+ with their centroid, so the mean across clips is a more stable target. Sensitivity to the selected reference clip is a known issue with zero-shot cloning, and this approach helps address it systematically. The centroid embedding is then baked directly into the fine-tuned checkpoint, eliminating the need for any reference audio processing at inference time. We extended the published recipe with linear warmup and cosine learning-rate decay. TTS fine-tuning, like other modalities, is...