Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct

ArcticSpeculator

Build a fastest OSS vllm-based speculative decoding system for your own model, using ArcticTraining and ArcticInference!

We compare the throughput (tokens/s) of existing vllm-based speculative decoding systems for Llama3.1-70B-Instruct on 8xH100 as below:

| method | ShareGPT | HumanEval | |--------------------------------------|----------------|--------------| | VLLM V1 Baseline | 84.1 | 84.1 | | VLLM V1 Eagle | 102.2 | 112.0 | | VLLM V1 Eagle3 | 77.7 | 85.3 | | VLLM V0 MLP-Speculator (IBM) | 77.9 | 66.7 | | ArcticSpeculator | 172.4 | 203.7 |

For more details about ArcticSpeculator and how to use it:

• ❄️ [Using Arctic-Inference and Arctic-Training for improving real-world speculative decoding Performance (blog)]()
• 🚀 Getting started guide using ArcticTraining

We also release ArcticSpeculator checkpoints we trained with ArcticTraining to run with ArcticInference:

| model | ArcticSpeculator | |---- | ---- | | Llama-3.1-70B-Instruct | Arctic-LSTM-Speculator-Llama-3.1-70B-Instruct | | Llama-3.3-70B-Instruct | Arctic-LSTM-Speculator-Llama-3.3-70B-Instruct | | Qwen2.5-32B-Instruct | Arctic-LSTM-Speculator-Qwen2.5-32B-Instruct | | Llama-3.1-8B-Instruct | Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct|