stepfun-ai/Step-Audio-R1

stepfun-ai/Step-Audio-R1

Language: Python

License: Apache-2.0

Stars: 673

Forks: 48

Open issues: 20

Created: 2025-11-11T02:38:13Z

Pushed: 2026-04-29T04:15:21Z

Default branch: main

Fork: no

Archived: no

README:

Step-Audio-R1/R1.5

🔥🔥🔥 News!!

• Apr 29, 2026: 🚀 We release the technical report of Step-Audio-R1.5 (ArXiv; [PDF](Step-Audio-R1.5.pdf)).
• Apr 29, 2026: 📦 We open-source three in-house benchmarks from Step-Audio-R1.5 under benchmarks/Step-Audio-R1.5/: step_caption, step_spqa, and step_dialogue_understanding.
• Jan 14, 2026: 🚀 We release the inference code and model weights of Step-Audio-R1.1 (HuggingFace; ModelScope)
• Nov 27, 2025: 🎉 We release the inference code and model weights of Step-Audio-R1 (HuggingFace; ModelScope)
• Nov 27, 2025: 🎮 We released the HF Space Playground
• Nov 19, 2025: 🎉 We release the Demo Page
• Nov 19, 2025: 👋 We release the technical report of [Step-Audio-R1](Step-Audio-R1.pdf).

WeChat Developer Group

📑 Open-source Plan

• [x] Inference Code (vLLM)
• [x] Online demo (Gradio)
• [x] Model Checkpoints

📚 Open Benchmarks

We release three standalone evaluation benchmarks from Step-Audio-R1.5 under benchmarks/Step-Audio-R1.5/:

• step_caption
• step_spqa
• step_dialogue_understanding

Please see benchmarks/Step-Audio-R1.5/README.md and the benchmark-local READMEs for dataset details.

Overview of R1.5

To-Do List

• [x] Technical report release (ArXiv; [PDF](Step-Audio-R1.5.pdf))
• [x] Open-source benchmark release (benchmarks/Step-Audio-R1.5/)
• [ ] Inference code for Step-Audio-R1.5

Introduction

Step-Audio-R1.5 is our latest audio reasoning model, described in the technical report and mirrored in this repository as [Step-Audio-R1.5.pdf](Step-Audio-R1.5.pdf).

While recent reasoning-oriented audio models benefit from extended Chain-of-Thought on objective benchmarks, they are often optimized with verifiable reward signals that compress rich auditory interaction into isolated labels. Step-Audio-R1.5 is designed to move beyond that limitation: instead of only maximizing benchmark correctness, it aims to preserve prosodic naturalness, emotional continuity, and immersive long-turn spoken interaction.

RLHF for Audio Reasoning

Step-Audio-R1.5 marks a shift from purely verifiable-reward-style optimization toward Reinforcement Learning from Human Feedback (RLHF) in audio reasoning.

This transition is motivated by a simple observation: strong objective scores alone do not guarantee a natural conversational experience. By incorporating preference-driven alignment for spoken interaction, Step-Audio-R1.5 maintains robust analytical reasoning while substantially improving the overall interactive feel of long-form audio dialogue.

Benchmark Results

The figure below compares the average score across all eight speech-to-text benchmarks. Step-Audio-R1.5 substantially improves over Step-Audio-R1 and remains highly competitive with other mainstream large models.

Overview of R1.1

Introduction

Step-Audio R1.1 (Realtime) is a major upgrade to Step-Audio-R1, designed for interactive spoken dialogue with both real-time responsiveness and strong reasoning capability.

Unlike conventional streaming speech models that trade intelligence for latency, R1.1 enables *thinking while speaking*, achieving high intelligence without sacrificing speed.

Mind-Paced Speaking (Low Latency)

Based on the research *Mind-Paced Speaking*, the Realtime variant adopts a Dual-Brain Architecture:

• A Formulation Brain responsible for high-level reasoning
• An Articulation Brain dedicated to speech generation

This decoupling allows the model to perform Chain-of-Thought reasoning during speech output, maintaining ultra-low latency while handling complex tasks in real time.

Acoustic-Grounded Reasoning (High Intelligence)

To address the *inverted scaling* issue—where reasoning over transcripts can degrade performance—Step-Audio R1.1 grounds its reasoning directly in acoustic representations rather than text alone.

Through iterative self-distillation, extended deliberation becomes a strength instead of a liability. This enables effective test-time compute scaling and leads to state-of-the-art performance, including top-ranking results on the AA benchmark.

Overview of R1

Introduction

Step-Audio-R1 is the first audio language model to successfully unlock test-time compute scaling. It decisively solves the "inverted scaling" anomaly plaguing existing models, where performance paradoxically degrades with longer reasoning chains.

We identify the root cause of this failure as Textual Surrogate Reasoning: conventional models, due to text-based initialization, rely on linguistic abstractions (analyzing transcripts) rather than genuine acoustic properties. To resolve this modality mismatch, we introduce Modality-Grounded Reasoning Distillation (MGRD), an iterative training framework that shifts the model's reasoning focus from textual surrogates to acoustic analysis.

This new approach allows us to create Step-Audio-R1, which:

• Is the first audio reasoning model that successfully benefits from test-time compute scaling.
• Surpasses Gemini 2.5 Pro and is comparable to Gemini 3 across comprehensive audio benchmarks.
• Transforms extended deliberation from a liability into a powerful asset for audio intelligence.

Model Architecture

Step-Audio-R1 builds on the architecture of our previous StepAudio 2 and consists of three main components:

1. Audio Encoder: We use the pre-trained Qwen2 audio encoder. It operates at a 25 Hz frame rate and is frozen during training. 2. Audio Adaptor: A simple adaptor (identical to Step-Audio 2) connects the encoder to the LLM and downsamples the feature frame rate to 12.5 Hz. 3. LLM Decoder: We use Qwen2.5 32B as the core reasoning component. It directly takes the latent audio features from the…