Hot Chips 2026: SN50 RDU Dataflow at Scale

Hot Chips 2026: SN50 RDU Dataflow at Scale

BACK TO RESOURCES

Blog

Hot Chips 2026: Dataflow at Scale

By

Raghu Prabhakar, Chief Architect, SambaNova

--> September 2, 2026

At Hot Chips 2026, the industry converged on a new bottleneck: AI is moving from one-shot answers to agents that reason, call tools, and keep working. That means far more generated tokens, and far more time spent in decode.

TL;DR

Agentic AI generates more tokens and spends most of inference in memory-bound decode, so system bandwidth matters more than peak compute.

Across six frontier-model configs SambaNova analyzed, decode was 75% to 97% of model-inference time.

Model Bandwidth Utilization (MBU) measures how much installed bandwidth actually moves weights and KV-cache: installed bandwidth times MBU equals tokens per second.

The SN50 RDU pairs a dataflow execution model and 432 MB on-chip SRAM with scale-up and scale-out Ethernet to keep bandwidth productive across chips.

Scaling modeled DeepSeek-R1 from 64 to 256 RDUs lifts per-user speed from 200 to 500 tokens per second (TPS) while MBU holds between 44% and 51%.

That shift makes the hardware challenge much more interesting. The next generation of inference will not be won with one big chip. Frontier models already span multiple chips, and their speed depends on how efficiently processors, memory, and networks operate as one system. A fast chip matters. A fast system matters more.

Our Hot Chips talk, “Dataflow at Scale: the SN50 RDU,” focused on that full-system problem: How much installed bandwidth reaches the model; how dataflow keeps work moving inside the RDU ; and how the network preserves that efficiency as models scale across chips, nodes, and racks.

Raghu Prabhakar, Chief Architect at SambaNova, presents the SN50 RDU at Hot Chips 2026.

From Peak Bandwidth to Useful Model Bandwidth

Previous SambaNova blogs explain the decode bottleneck and why disaggregated inference maps compute-heavy prefill to GPUs and memory-bound decode to RDUs. Our talk started with the metric that connects that architecture to delivered token speed: Model Bandwidth Utilization (MBU).

This idea is well-established, not new. The MosaicML engineering team at Databricks introduced MBU as achieved model-data bandwidth divided by a system’s peak memory bandwidth. Peak high bandwidth memory (HBM) tells you what the hardware could move. MBU tells you how much of that bandwidth is actually moving model weights and key-value (KV) cache data during token generation. For bandwidth-bound decode, installed bandwidth multiplied by MBU gives effective model bandwidth, which is the part that translates into tokens per second.

In the six frontier-model configurations SambaNova analyzed, decode represented 75% to 97% of model-inference time. At the high end, the DeepSeek-V3 case (8K input, 1K output, B300 FP4, and one active request) spent 97% of model-inference time in decode. The exact split varies by model, context, precision, and concurrency, but the system lesson is consistent: Peak compute alone does not predict the speed users will feel from an agent.

MBU connects peak HBM bandwidth to the model-data movement that drives token generation. Source: Raghu Prabhakar, Hot Chips 2026. Concept reference: MosaicML/Databricks.

This distinction gets more important as systems grow. Adding accelerators increases theoretical bandwidth, but per-user token speed will not rise with it if synchronization, memory traffic, or communication overhead drives utilization down. The goal is not simply more bandwidth. The goal is more bandwidth that the model can use.

How Dataflow Turns Bandwidth into Tokens

The SN50 RDU starts with a dataflow execution model rather than a sequence of separately launched kernels. The compiler maps the model across a tiled mesh of compute and memory units. SN50’s 432 MB of distributed on-chip SRAM keeps frequently used data close to compute and allows intermediate values to flow between operations without repeated trips to HBM.

Persistent decoder execution and operator fusion keep the model moving across token steps instead of repeatedly stopping and resuming work at kernel boundaries. Double-buffered memory units begin moving the next model tile while the current computation runs. Collective communication can terminate in SRAM, avoiding unnecessary HBM traffic and allowing compute, memory access, and communication to overlap.

That overlap is the link between MBU and scale. Useful bandwidth is not a property of memory alone; it is the result of keeping the entire execution pipeline productive. Once a model spans multiple RDUs, the network becomes part of that pipeline.

Inference at Scale Needs Scale-Up and Scale-Out

Dense frontier models and mixture-of-experts models span many accelerators. Their fabrics must carry frequent collectives and dynamic token movement without turning communication into the next bottleneck.

The SambaNova SN50 uses three connected network domains. Inside an eight-socket node, seven integrated 800G Ethernet links form a fully connected topology, which gives each RDU a direct path to other RDUs in the node. Beyond the node, two additional 800G ports per RDU connect to the inter-node scale-up fabric. Across those domains, the SN50 integrates ten 800G Ethernet ports with an aggregate 2 TB/s per RDU. In the 64-socket example, two 64-port 800G switches connect every node to both switches, with one link from every RDU going to each switch.

Scale-out has a different job. Each SN50 RDU uses a 400G RoCEv2 NIC to connect scale-up domains across a rail-optimized Ethernet network. This layer supports disaggregated inference and KV-cache transfer between scale-up domains. A separate front-end data center network remains available for hosts, storage, and management.

SN50 combines a fully connected eight-socket node, 800G inter-node scale-up, and a 400G RoCEv2 scale-out path between scale-up domains. Source: Raghu Prabhakar, Hot Chips 2026.

The separation matters because the traffic is different. Scale-up carries the tightly coupled communication required to split one model across RDUs. Scale-out connects larger domains and heterogeneous inference stages. Each tier is built around the communication pattern it has to sustain.

Why Tensor and Expert Parallelism Need Different Network Behavior

Tensor parallelism splits large tensor operations across RDUs. Every step depends on collectives, such as reduce-scatter, all-gather, and...