amazon/GKA-primed-HQwen3-32B-Instruct

GKA-primed-HQwen3-32B-Instruct

GKA-primed-HQwen3-32B-Instruct is a Hybrid language model consisting of 50% Attention layers and 50% Gated KalmaNet (GKA) layers, primed from Qwen3-32B using the Hybrid Model Factory Priming pipeline. The model is instruction-tuned and supports context lengths up to 128K tokens.

GKA (pronounced as gee-ka) is a State-Space Model layer inspired by the Kalman Filter that solves an online ridge regression problem at test time, with constant memory and linear compute cost in the sequence length.

By combining Attention with GKA, our Hybrid model achieves up to 2× faster inference at long contexts while closely matching the base Transformer's quality.

Links

• 📄 Gated KalmaNet paper (CVPR 2026)
• 💻 GitHub: Hybrid Model Factory

Why Hybrid?

Each Primed Hybrid model is initialized from a base Transformer by converting a portion of its Attention layers into State-Space Model (SSM) layers that maintain a fixed-size recurrent state instead of a growing KV cache. At a 50% Hybrid ratio, roughly half the KV cache (which grows linearly with sequence length) is replaced with fixed-size SSM state. The practical benefits:

• Higher throughput at long contexts — less memory on KV cache means more memory for batching
• More concurrent sequences — ~2× as many concurrent sequences before hitting memory limits
• Growing advantage with context length — at long contexts, Attention dominates the forward pass while SSM layers remain negligible in cost. Since the Hybrid model makes roughly half as many Attention calls as the base Transformer, the throughput advantage grows with context length

Increasing hybridization ratio, replacing more Attention layers with SSM layers, further reduces memory and increases throughput, typically at the expense of performance.

Model Overview

• Type: Causal Language Model (Hybrid Attention + SSM)
• Base Model: Qwen3-32B
• Hybrid Layer Type: Gated KalmaNet (GKA)
• Hybrid Ratio: 50% (32 Attention + 32 GKA layers)
• Parameters: ~32B
• Context Length: 128K natively
• Precision: bfloat16
• License: Apache 2.0

Note, this is an Instruct-tuned model and is not a thinking model, that is, it does not natively produce chain-of-thought thinking tokens in its generation trace.

Benchmark Results

Below we report benchmark performance for all our instruct-tuned Primed models. All Hybrid models use a 50% Hybrid ratio and are Primed from Qwen3-32B.

We consider the following Transformer as a baseline:

• Qwen3-32B (Long): The Qwen model fine-tuned on our priming data, extending its native context length from 32K to 128K. All Primed Hybrid models use the same training hyperparameters and data as this baseline, making it a fair comparison for differing architectures.

On both long- and short-context benchmarks, our Primed Hybrid models closely match the performance of the Transformer model while having [considerably lower deployment costs](#inference-efficiency), showcasing the efficacy of the Priming process.

Long-Context Benchmarks

Evaluated on HELMET, MRCR, and BABILong across context lengths from 8K to 128K, using a weighted average with geometrically increasing weights for longer contexts.

The plot below shows performance averaged over context lengths from 8K to 128K.

How close are the Hybrid models to the Transformer baseline on long context tasks? Primed GKA and GDN hybrids have competitive long-context capabilities with a gap of ~2.5-3 points on average with the Transformer [Qwen3-32B (Long)], while being [1.5–2× faster at inference](#inference-efficiency) on long contexts.

Short-Context NLP Benchmarks

Evaluations on Tulu3-dev from OLMES. All tasks are over a short-context length (≤ 8K). Each category in the table below averages the following Tulu3-dev subtasks: 1. Math: GSM8K, MATH. 2. Knowledge: MMLU, PopQA, TruthfulQA. 3. Coding: HumanEval, HumanEval+. 4. Reasoning: BigBenchHard. 5. Instruction Following: IFEval.

| Model | Math | Knowledge | Coding | Reasoning | Instruction Following | Average | |------------------------------------|-------|----------|--------|-----------|-----------------------|---------| | Qwen3-32B [Long] | 74.43 | 54.47 | 94.54 | 82.89 | 81.52 | 77.56 | | GKA-primed-HQwen3-32B-Instruct | 74.02 | 53.95 | 93.43 | 80.31 | 78.74 | 76.09 | | GDN-primed-HQwen3-32B-Instruct | 73.65 | 54.35 | 94.40 | 80.99 | 79.3 | 76.54 |

How close are the Hybrid models to the Transformer baseline on short context tasks? Our Primed Hybrid models are within ~1-1.5 points of the average performance of the Transformer [Qwen3-32B (Long)] using [ [!NOTE] > For applications to complex reasoning and coding problems check out our Primed Hybrid Reasoning models.

About Gated KalmaNet (GKA)

Gated KalmaNet is a State-Space Model layer that is more expressive than both Mamba2 and Gated DeltaNet. GKA achieves this by employing the Kalman Filter to compute the optimal state at each time-step based on the entire past. In contrast, SSMs like Mamba2 and GDN rely on instantaneous objectives (that rely *solely* on the current input and loss estimate of the past) to compute their state.

Unlike other SSM-based hybrid layers, GKA gives you a runtime knob for trading compute against speed — with no retraining nor architecture changes. The num_iter parameter controls how many iterations the GKA solver runs during inference. No other hybrid layer type offers this: GDN and Mamba2 have fixed compute per layer, so their speed is fixed a priori. GKA lets you slide along the compute–latency curve per deployment, making it uniquely suited for scenarios where different endpoints or traffic tiers have different latency budgets.

For details on controlling GKA's compute–speed tradeoff at serving time via num_iter, see GKA Compute Control, and for more details on the modeling choices see the [GKA…