inclusionAI/Ling-V2

inclusionAI/Ling-V2

Description: Ling-V2 is a MoE LLM provided and open-sourced by InclusionAI.

Language: Python

License: MIT

Stars: 271

Forks: 17

Open issues: 6

Created: 2025-09-04T06:54:36Z

Pushed: 2025-10-04T06:15:38Z

Default branch: main

Fork: no

Archived: no

README:

Ling-V2

🤗 Hugging Face&nbsp&nbsp | &nbsp&nbsp🤖 ModelScope

Introduction

Today, we are excited to announce the open-sourcing of __Ling 2.0__ — a family of MoE-based large language models that combine __SOTA performance__ with __high efficiency__. The first released version, Ling-mini-2.0, is compact yet powerful. It has __16B total parameters__, but only __1.4B__ are activated per input token (non-embedding 789M). Trained on more than __20T tokens__ of high-quality data and enhanced through multi-stage supervised fine-tuning and reinforcement learning, Ling-mini-2.0 achieves remarkable improvements in complex reasoning and instruction following. With just 1.4B activated parameters, it still reaches the top-tier level of sub-10B dense LLMs and even matches or surpasses much larger MoE models.

Strong General and Professional Reasoning

We evaluated Ling-mini-2.0 on challenging general reasoning tasks in coding (LiveCodeBench, CodeForces) and mathematics (AIME 2025, HMMT 2025), as well as knowledge-intensive reasoning tasks across multiple domains (MMLU-Pro, Humanity's Last Exam). Compared with sub-10B dense models (e.g., Qwen3-4B-instruct-2507, Qwen3-8B-nothinking) and larger-scale MoE models (Ernie-4.5-21B-A3B-PT, GPT-OSS-20B/low), Ling-mini-2.0 demonstrated outstanding overall reasoning capabilities.

7× Equivalent Dense Performance Leverage

Guided by Ling Scaling Laws, Ling 2.0 adopts a __1/32 activation ratio__ MoE architecture, with empirically optimized design choices in expert granularity, shared expert ratio, attention ratio, aux-loss free + sigmoid routing strategy, MTP loss, QK-Norm, half RoPE, and more. This enables small-activation MoE models to achieve over __7× equivalent dense performance__. In other words, __Ling-mini-2.0 with only 1.4B activated parameters (non-embedding 789M) can deliver performance equivalent to a 7–8B dense model__.

High-speed Generation at 300+ token/s

The highly sparse small-activation MoE architecture also delivers significant training and inference efficiency. In simple QA scenarios (within 2000 tokens), __Ling-mini-2.0 generates at 300+ token/s (on H20 deployment)__ — more than __2× faster__ than an 8B dense model. Ling-mini-2.0 is able to handle __128K context length__ with YaRN, as sequence length increases, the relative speedup can reach __over 7×__.

Open-sourced FP8 Efficient Training Solution

Ling 2.0 employs __FP8 mixed-precision training__ throughout. Compared with BF16, experiments with over 1T training tokens show nearly identical loss curves and downstream benchmark performance. To support the community in efficient continued pretraining and fine-tuning under limited compute, we are also open-sourcing our __FP8 training solution__. Based on tile/blockwise FP8 scaling, it further introduces FP8 optimizer, FP8 on-demand transpose weight, and FP8 padding routing map for extreme memory optimization. On 8/16/32 80G GPUs, compared with LLaMA 3.1 8B and Qwen3 8B, __Ling-mini-2.0 achieved 30–60% throughput gains with MTP enabled, and 90–120% throughput gains with MTP disabled__.

A More Open Opensource Strategy

We believe Ling-mini-2.0 is an ideal starting point for MoE research. For the first time at this scale, it integrates 1/32 sparsity, MTP layers, and FP8 training — achieving both strong effectiveness and efficient training/inference performance, making it a prime candidate for the small-size LLM segment. To further foster community research, in addition to releasing the post-trained version, we are also open-sourcing __five pretraining checkpoints__: the pre-finetuning Ling-mini-2.0-base, along with four base models trained on 5T, 10T, 15T, and 20T tokens, enabling deeper research and broader applications.

Model Downloads

You can download the following table to see the various stage of Ling-mini-2.0 models(1.43B activated of 16.26B total params). If you are located in mainland China, we also provide the model on ModelScope.cn to speed up the download process.

| Model | Context Length | Download | |:----------------------:| :----------------: |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | Ling-mini-base-2.0-5T | 4K | 🤗 HuggingFace 🤖 ModelScope | | Ling-mini-base-2.0-10T | 4K | 🤗 HuggingFace 🤖 ModelScope | | Ling-mini-base-2.0-15T | 4K | 🤗 HuggingFace 🤖 ModelScope | | Ling-mini-base-2.0-20T | 4K | 🤗 HuggingFace 🤖 ModelScope | | Ling-mini-base-2.0 | 32K -> 128K (YaRN) | 🤗 HuggingFace 🤖 ModelScope | | Ling-mini-2.0 | 32K -> 128K (YaRN) | 🤗 HuggingFace 🤖 ModelScope |

And the Ling-flash-2.0 models(6.1B activated of 100B total params) can be downloaded from the following table.

| Model | Context Length | Download | |:----------------------:| :----------------: |:-------------------------------------------------------------------------------------------------------------------------------------------------------------:| | Ling-flash-base-2.0 | 32K -> 128K (YaRN) | 🤗 HuggingFace 🤖 ModelScope | | Ling-flash-2.0 | 32K -> 128K (YaRN) | 🤗 HuggingFace 🤖 ModelScope |…