togethercomputer/GPTQModel

togethercomputer/GPTQModel

Description: LLM model quantization (compression) toolkit with HW acceleration support for Nvidia, AMD, Intel GPU and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

License: NOASSERTION

Stars: 0

Forks: 0

Open issues: 1

Created: 2026-08-05T18:46:26Z

Pushed: 2026-08-06T01:18:15Z

Default branch: main

Fork: yes

Parent repository: ModelCloud/GPTQModel

Archived: no

README:

GPT-QModel

LLM model quantization (compression) toolkit with hw acceleration support for NVIDIA CUDA, AMD ROCm, Huawei Ascend NPU, Intel XPU, and Intel/AMD/Apple CPUs via HF, vLLM, and SGLang.

Latest News

• 08/04/2026 7.4.0 main: ✨ Added axk2 (A.X-K2) model support
• 08/04/2026 7.4.0 main: 🚀🔥⚡ Added Swordfish Blackwell (>= sm100) GPTQ/AWQ kernel from AlpinDale: Paper.
• 07/24/2026 7.3.1 main: ✨ Added solar_open and solar_open2 model support
• 07/23/2026 7.3.1 main: ✨ Added Intern S2 PreView model support
• 07/23/2026 7.3.1 main: ✨ Added inkling_mm_model model support
• 07/22/2026 7.3.1 main: ✨ Added Poolside Laguna S 2.1 model support
• 07/14/2026 7.3.0-dev main: ✨ Added nemotron_h_puzzle model support
• 07/07/2026 7.3.0-dev main: ✨ Added deepseek_vl model support
• 07/06/2026 7.3.0-dev main: ✨ Added deepseek_ocr2 model support
• 04/07/2026 7.3.0-dev main: ✨ Added deepseek_vl_v2 model support
• 02/07/2026 7.2.0-dev main: ✨ Added lfm2 and lfm2_vl model support
• 06/30/2026 7.2.0-dev main: ✨ Added minimax_m3_vl / MiniMax M3 model support
• 06/25/2026 7.2.0-dev main: ✨ Added cohere2_moe model support
• 05/25/2026 7.1.0-dev main: ✨ Added hy_3 and ministral3 model support
• 05/25/2026 7.1.0-dev main: ✨ Added hunyuan_v1_dense and hunyuan_v1_moe model support
• 05/21/2026 7.1.0-dev main: ✨ Added nemotron_labs_diffusion model support
• 05/20/2026 7.1.0-dev main: ✨ Added interns1, ovis2_5, ovis2_6_moe and ovis2_6_next model support
• 05/15/2026 7.1.0-dev main: ✨ Added mimo_v2 model support
• 05/13/2026 7.1.0-dev main: ✨ Added minicpmv_4_6 and DeepSeek V4 model support
• 05/07/2026 7.1.0-dev main: ✨ Added GLM-4.5V, GLM-4.6V, Zamba and Zamba2 model support
• 04/29/2026 7.1.0-dev main: ✨ Added PoolSideAI Laguna model support for fused Laguna MoE checkpoints. Added ERNIE 4.5 VL MoE, Ling-2.6-flash and NVIDIA Nemotron 3 Nano Omni model support.
• 04/28/2026 7.0.0: 🚀 Added Huawei Ascend NPU support through native torch kernels for GPTQ, AWQ, ParoQuant, GGUF, QQQ, and EXL3. Added internvl_chat, gemma3n, GLM-OCR, GLM-ASR, and falcon_mamba model support.
• 04/16/2026 6.1.0: 🚀🔥⚡ CUDA kernels are now fully JIT-compiled, shrinking the wheel by about 300x and building only what you use; Marlin now supports NVIDIA Turing+ GPUs, Machete kernel validation now covers supported GPUs, GLM 5/5.1 joins the lineup, and LazyTurtle plus AWQ / multi-GPU MoE fixes make large-model quantization easier, lighter, and smoother.
• 04/03/2026 6.0.3: 🎉 New quantization methods: ParoQuant, GGUF, FP8, EXL3, and FOEM: First-Order Error Matters. Added PrismML/Bonsai 1bit model quantization (inference only), faster ParoQuant/AWQ kernels, ParoQuant optimization scope control: module (Paro Lite) or layer (Paro reference), plus Gemma4, MiniCPM-O, MiniCPM-V, and GLM4 MoE Lite model support.
• 03/19/2026 5.8.0: ✨HF Transformers 5.3.0 support with auto-defusing of fused models via pypi pkg: Defuser. Qwen 3.5 family support added. New fast HF cpu kernels for GPTQ/AWQ added. Experimental INT8 cpu kernel added for GPTQ.

Archived News

• 02/09/2026 5.7.0: ✨New MoE.Routing config with Bypass and Override options to allow multiple brute-force MoE routing controls for higher quality quantization of MoE experts. Combined with FailSafeStrategy, GPT-QModel now has three separate control settings for efficient MoE expert quantization.

AWQ qcfg.zero_point property has been merged with a unified sym symmetry property; zero_point=True is now sym=False. Fixed AWQ sym=True packing/inference and quantization compatibility with some Qwen3 models. Exaone 4.0 support.

• 12/31/2025 5.7.0-dev: ✨New FailSafe config and FailSafeStrategy, auto-enabled by default, to address uneven routing of MoE experts resulting in quantization issues for some MoE modules. Smooth operations are introduced to FailSafeStrategy to reduce the impact of outliers in FailSafe quantization using RTN by default. Different FailSafeStrategy and Smoothers can be selected. Threshold to activate FailSafe can also be customized.

New Voxtral and Glm-4v model support, plus audio dataset calibration for Qwen2-Omni. AWQ compatibility fix for GLM 4.5-Air.

• 12/17/2025 5.6.2-12 Patch: Fixed uv compatibility. Both uv and pip installs will now show UI progress for external wheel/dependency downloads. Fixed macOS and AWQMarlin kernel loading import regressions. Resolved most multi-arch compile issues on Ubuntu, Arch, RedHat and other distros. Fixed multi-arch build issues and Tritonv2 kernel launch bug on multi-GPUs. Fixed 3-bit Triton GPTQ kernel dequant/inference and license property compatibility issue with latest pip/setuptools.
• 12/9/2025 5.6.0: ✨New HF Kernel for CPU optimized for AMX, AVX2 and AVX512. Auto module tree for auto-model support. Added AfMoE and Dots1 model support. Fixed pre-layer pass quantization speed regression. Improved HF Transformers, PEFT and Optimum support for both GPTQ and AWQ. Fixed many AWQ compatibility bugs and regressions.
• 11/9/2025 5.4.0: ✨New Intel CPU and XPU hardware-optimized AWQ TorchFusedAWQ kernel. Torch Fused kernels now compatible with torch.compile. Fixed AWQ MoE model compatibility and reduced VRAM usage.
• 11/3/2025...