openbmb/BitCPM-CANN-8B

GitHub Repo | Technical Report

👋 Join us on Discord and WeChat

Introduction

BitCPM-CANN is the first end-to-end 1.58-bit (ternary) large language model training system natively built on Huawei Ascend NPU. The system integrates quantization-aware training (QAT) into the Megatron-LM framework with MindSpeed acceleration, covering the full training stack from custom ternary operators to distributed parallel training on Ascend 910B.

We train a family of four models—BitCPM-CANN-0.5B/1B/3B/8B—and evaluate them against their full-precision MiniCPM4 counterparts across 11 benchmarks. The 1B/3B/8B models retain 95.7%–97.2% of full-precision performance, while enabling approximately 6× memory reduction at inference time. QAT introduces only 5% training throughput overhead (148 vs. 155 TFLOP/s per NPU).

Key Features

• 🔬 1.58-Bit Ternary Quantization: Compresses model weights to ternary values {-1, 0, 1}, achieving ~90% bit-width reduction compared to BF16.
• 🖥️ Native Ascend NPU Training: First publicly reported 1.58-bit training effort on domestic NPU platform at 8B scale, establishing reusable low-bit training infrastructure for the Ascend ecosystem.
• ⚡ Minimal Training Overhead: Only 5% throughput degradation compared to full-precision training on Ascend 910B.
• 📦 ~6× Inference Memory Reduction: Enables longer contexts, more serving replicas, and edge deployment on consumer devices.

Important Note

> The models in this repository are in pseudo-quantized (fake quantization) format. This means the weights are stored in standard floating-point format with ternary values already applied during training. You can load and run inference with these models exactly the same way as full-precision models—no special quantization libraries or custom kernels are required.

BitCPM-CANN Model Family

| Model | HuggingFace | GGUF | |-------|-------------|------| | BitCPM-CANN-0.5B | openbmb/BitCPM-CANN-0.5B | openbmb/BitCPM-CANN-0.5B-gguf | | BitCPM-CANN-1B | openbmb/BitCPM-CANN-1B | openbmb/BitCPM-CANN-1B-gguf | | BitCPM-CANN-3B | openbmb/BitCPM-CANN-3B | openbmb/BitCPM-CANN-3B-gguf | | BitCPM-CANN-8B | openbmb/BitCPM-CANN-8B | openbmb/BitCPM-CANN-8B-gguf |

Usage

Inference with Transformers

Since BitCPM-CANN models are in pseudo-quantized format, you can use them exactly like standard full-precision models:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
torch.manual_seed(0)

path = 'openbmb/BitCPM-CANN-8B'
device = "cuda"
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map=device, trust_remote_code=True)

# User can directly use the chat interface
responds, history = model.chat(tokenizer, "Write an article about Artificial Intelligence.", temperature=0.7, top_p=0.7)
print(responds)

# User can also use the generate interface
# messages = [
# {"role": "user", "content": "Write an article about Artificial Intelligence."},
# ]
# prompt_text = tokenizer.apply_chat_template(
# messages,
# tokenize=False,
# add_generation_prompt=True,
# )
# model_inputs = tokenizer([prompt_text], return_tensors="pt").to(device)

# model_outputs = model.generate(
# **model_inputs,
# max_new_tokens=1024,
# top_p=0.7,
# temperature=0.7
# )
# output_token_ids = [
# model_outputs[i][len(model_inputs[i]):] for i in range(len(model_inputs['input_ids']))
# ]

# responses = tokenizer.batch_decode(output_token_ids, skip_special_tokens=True)[0]
# print(responses)

Evaluation Results

Main Results

BitCPM-CANN models are evaluated against their full-precision MiniCPM4 counterparts across 11 benchmarks spanning commonsense reasoning, domain knowledge, and mathematics & reasoning.

| Task | 8B FP | 8B Ternary | 3B FP | 3B Ternary | 1B FP | 1B Ternary | 0.5B FP | 0.5B Ternary | |------|-------|------------|-------|------------|-------|------------|---------|--------------| | ARC-c | 87.46 | 86.10 | 80.34 | 78.98 | 64.41 | 67.12 | 51.86 | 50.51 | | ARC-e | 95.06 | 93.47 | 92.77 | 88.36 | 79.89 | 79.01 | 71.78 | 65.08 | | BoolQ | 84.89 | 83.39 | 79.85 | 77.89 | 68.38 | 65.50 | 62.29 | 43.55 | | PIQA | 80.52 | 78.78 | 70.57 | 72.69 | 66.16 | 65.45 | 60.99 | 58.49 | | WinoGrande | 63.30 | 61.17 | 58.41 | 52.96 | 51.62 | 53.28 | 51.07 | 51.54 | | CMMLU | 80.62 | 78.92 | 78.11 | 76.53 | 74.57 | 67.42 | 65.22 | 60.49 | | C-Eval | 81.36 | 77.50 | 75.85 | 75.89 | 73.25 | 65.96 | 66.11 | 60.74 | | MMLU | 75.83 | 70.65 | 66.95 | 64.41 | 57.71 | 57.71 | 55.55 | 50.73 | | MMLU-Redux | 77.14 | 69.85 | 65.82 | 60.07 | 54.80 | 54.16 | 48.00 | 43.79 | | BBH | 76.72 | 70.70 | 68.29 | 68.30 | 64.40 | 60.40 | 49.87 | 47.44 | | GSM8K | 91.51 | 85.75 | 81.64 | 79.45 | 63.15 | 61.56 | 52.08 | 39.42 | | Average (11 tasks) | 81.31 | 77.84 | 74.42 | 72.32 | 65.30 | 63.42 | 57.71 | 51.98 | | Retention | | 95.7% | | 97.2% | | 97.1% | | 90.1% |

Key Observations

• 1B and above achieve ≥95.7% retention: The 3B model achieves the highest retention at 97.2%, demonstrating that ternary QAT at this scale introduces minimal capability loss.
• 0.5B reveals scale-dependent sensitivity: The smallest model retains 90.1%, indicating that quantization perturbation is more damaging when model capacity is limited.
• 1:1 alignment with MiniCPM4: The matched evaluation enables direct substitution decisions—deployments can replace specific full-precision models with their ternary counterparts with clearly quantified trade-offs.

Training Efficiency

| Configuration | TFLOP/s per NPU | Overhead | |---------------|-----------------|----------| | Full-precision | 155 | — | | Ternary QAT | 148 | 4.5% |

System-level throughput on 2-node 16-card Ascend 910C:

• 3B model: ~2700 tokens/s per card
• 8B model: ~1340 tokens/s per card

Technical Approach

BitCPM-CANN uses a ternary quantizer that maps each weight group to {-1, 0, 1} scaled by a group-wise factor, trained with Straight-Through Estimator…