5 Lessons from Building a Multi-plane Network Fabric for Agentic AI

Multi-Plane Networking for Agentic AI: 5 Lessons | CoreWeave

Announcement

Webinar

Podcast

GTC 2026

CoreWeave recognized as a Visionary in the Gartner® Magic Quadrant™ for Cloud AI Infrastructure. Read the report

Products

Data and storage

Infrastructure control

Runtime acceleration

Model and agent development

Mission control

Solutions

Pricing

Resources

About us

Contact us Login

Contact us Login

Clear

Training a foundation model is, from the east-west fabric's point of view, a well-behaved problem. Thousands of GPUs run the same collective at roughly the same time, over and over, for weeks. It's brutal on bandwidth, but it's rhythmic. You can design around that rhythm. Agentic AI has a less predictable traffic pattern. An agent breaks a task into steps, calls a retriever, calls a tool, calls another model, sometimes calls another agent, and waits on each hop before it knows what the next one is. That serial path is an agent chain, and its latency is the sum of every hop rather than the worst one. Multiply that across thousands of concurrent chains running alongside training jobs and standard inference on the same fleet, and the traffic stops looking like a batch job. It starts looking like a latency-sensitive, many-hop conversation happening everywhere on the fabric at once. Agentic traffic breaks assumptions that training never tested Unlike general-purpose clouds retrofitted for AI, CoreWeave designed and built its network for AI from the ground up. Agents themselves aren't new, but running them at fleet scale introduces networking demands that earlier AI workloads never made. Meeting those demands meant rebuilding the backend network from the topology up, so one fleet can carry sustained training collectives, steady inference, and bursty agent traffic without any one of the three degrading the others. We built our network on the NVIDIA Spectrum-X Ethernet platform, BlueField-3 DPUs, and ConnectX-9 SuperNICs. We then paired that hardware with our own stack for telemetry, orchestration, scheduling, security, and observability through CoreWeave Kubernetes Service (CKS) , SUNK , and CoreWeave Mission Control . Today that fabric connects fleets across more than 50 data centers. Here's what building it, at fleet scale, taught us. 1. Non-blocking is what keeps your agents moving when training spikes A fabric can pass every point-to-point bandwidth test you throw at it and still stall your agent chains. Oversubscription is usually why. When leaf switches carry less uplink capacity to the spine than the nodes beneath them can generate, the shortfall stays invisible until two traffic types want the same link at the same moment. Training alone can tolerate some of that. A fabric mixing synchronized training bursts with thousands of small, sequential agent requests cannot, because it needs high aggregate bandwidth and low, predictable per-flow latency simultaneously. An oversubscribed link struggles to sustain both simultaneously. The lesson we applied is: don't oversubscribe. Every leaf carries enough uplink capacity to sustain full line rate from every connected node, at every layer, at a 1:1 ratio. That discipline holds on every fabric we've built, and it's why CoreWeave operates some of the largest NVIDIA Quantum InfiniBand and RDMA over Converged Ethernet (RoCE) fabrics running today. Keep one tenant's burst off another tenant's chain NVIDIA Spectrum-X Ethernet coordinates the switch and the SuperNIC as one fabric rather than treating congestion as a switch-level problem. Adaptive routing and telemetry-based congestion control keep one tenant's synchronized training burst from degrading another tenant's latency-sensitive agent chain. Congestion thresholds tuned for NVIDIA Collective Communications Library (NCCL) traffic reduce the packet loss that collective operations are prone to triggering, which matters for training runs and for the fan-out-then-synchronize pattern that an agent orchestration step often resembles. We also move on new silicon early. CoreWeave was one of the first cloud providers to deploy the NVIDIA Spectrum-X SN6600-LD , the industry's first fully liquid-cooled 102.4 Tb/s Ethernet switch, running it as the switching fabric for NVIDIA Vera Rubin NVL72 . That gives customers access to higher-capacity networking as new AI systems come online. 2. Multi-plane means your fleet can double without costing you a hop Staying non-blocking as a fleet grows is harder than establishing it on day one. A flat, two-tier leaf-spine topology eventually runs out of radix, the number of ports a switch can practically support. The traditional fix is a third tier, which buys capacity by adding hops. More hops mean more places for jitter to accumulate, often when agentic workloads can least afford either. Instead of adding another tier, multi-plane topology avoids the trade. Splitting each GPU's SuperNIC connectivity across two or more independent network planes gives the fabric scale without giving it depth. That's what lets a flat, two-tier design reach 128,000 GPUs, 64 times the reach of a single-plane fabric, with no redesign along the way. Multi-plane operates at the scale of the fabric. Multi-rail applies the same discipline at the scale of the node. Each GPU connects through more than one SuperNIC, and in a rail-optimized design, every node's Nth SuperNIC connects to the same leaf switch as every other node's Nth SuperNIC. Traffic on a given rail never crosses into another leaf's domain to reach its destination. The result is GPU-to-GPU paths that stay short and predictable instead of hashing across whichever leaf happens to be reachable. Multi-plane keeps the fabric from running out of room as it grows. Multi-rail keeps every individual GPU's path into that fabric short and consistent. 3. Hardware-accelerated load balancing gives you the bandwidth you already bought Non-blocking bandwidth and a topology built to scale still don't guarantee that traffic uses all of the bandwidth. Left to static hashing, a handful of large flows pile onto the same path while other paths sit idle. That imbalance is how a well-designed fabric becomes congested under real collective and agentic traffic. Load balancing has traditionally run in software, with NCCL deciding how to spread a collective's traffic across available links. While this approach works, it comes with two...