What Comes Next: Operating and Evolving the Production AI Factory

NVIDIA, CoreWeave, and AI Factory Operations at Scale

Announcement

Webinar

Podcast

GTC 2026

CoreWeave recognized as a Visionary in the Gartner® Magic Quadrant™ for Cloud AI Infrastructure. Read the report

Products

Data and storage

Infrastructure control

Runtime acceleration

Model and agent development

Mission control

Solutions

Pricing

Resources

About us

Contact us Login

Contact us Login

Clear

This is the second in a series from CoreWeave and NVIDIA on the production AI factory lifecycle. In part one, “ Why AI Factories Need Proof Before Production ,” we explored how co-design and validation prepare the integrated stack for deployment. This post covers what comes next: operating at scale and preparing for the next generation of accelerated computing. The same discipline that designs and validates the stack carries into what happens after a workload goes live, catching the health signals, holding the scheduling steady, keeping recovery ready before it's needed. This is where that discipline gets tested, starting with what changes once a system moves from validated to operational. It's no longer about whether it works. It's about whether it keeps working, for the two-week run to simulate a production-ready environment.  A cluster that passes every benchmark can still lose real hours to interruptions, or a checkpoint that silently didn't save. What actually matters at that point isn't whether the hardware is fast. It's how much of that speed turns into finished work instead of recovery time. Operating at scale Day two starts the moment day one ends Catching an issue before it costs a run is what day-two operations actually are. Continuous health signals, workload visibility, scheduling control, and fast response the moment something breaks. Long-running distributed training jobs don't leave room to notice an interruption a day late. Why goodput governs AI factory economics At the scale these systems run, one number governs the economics. How much of the compute customers are paying for turns into workload progress. That's goodput, the share of time the system spends doing useful work rather than recovering from interruptions. On long-running, distributed training workloads, CoreWeave has demonstrated up to 96% goodput on NVIDIA Hopper GPUs , always as an upper bound rather than a guarantee. Everything below that line is lost productivity, useful work the system could not deliver because resources went to interruptions and recovery rather than model progress. The variables driving goodput What drives that number comes down to two variables—throughput and resiliency—and a 1,024-NVIDIA Hopper cluster benchmark isolates both. Throughput. Model FLOPs utilization (MFU) measures how much of a GPU's peak throughput actually turns into model progress. CoreWeave measured 20% higher MFU on Hopper-based systems than other  publicly reported benchmarks show. MFU only tells part of the story. What determines how fast a training run finishes is effective throughput, which multiplies that utilization rate against the GPU's peak capability.  MFU is also workload-dependent, it varies with model architecture (dense versus mixture-of-experts), sequence length, and parallelism, so this figure reflects its test context rather than a universal result. Resiliency. Mean time to failure improved roughly 10x, with an effective training time ratio of up to 98 percent.  Restarts and lost checkpoints stop setting the timeline when interruptions drop by that much. On CoreWeave Cloud, those gains come from two layers working together. Reliability, availability, and serviceability (RAS) built into the GPU, CPU, and rack architecture, and CoreWeave's software layer that detects, contains, and recovers from issues. Those gains come from fewer interruptions and faster recovery; failures still happen, they just cost less time.

On the NVIDIA side, resiliency improves with each generation. NVIDIA Blackwell introduced a dedicated RAS engine that applies AI-based predictive maintenance across thousands of hardware and software data points.  Reliability that begins in silicon and compounds upward, working alongside topology-aware scheduling on rack-scale systems, is how the integrated stack keeps useful work flowing, rather than being added on after the fact. NVIDIA Vera Rubin carries that further with a second-generation RAS engine for proactive maintenance and real-time health checks without downtime.  Silicon can flag a degrading component. Something above it has to decide what to do about it.  At the system level, modular cable-free trays make Vera Rubin assembly 90x faster than Blackwell and simplifies serviceability, and software-defined NVLink routing reroutes around faults to keep operation continuous with less maintenance overhead. How unified software and silicon improve AI factory reliability That decision layer is CoreWeave Mission Control , which provides reliability, transparency, and actionable insights for the teams running these clusters. Its job is to catch and contain infrastructure failures before they cost a customer the run, through straggler detection, automated node draining, fleet and rack lifecycle management, and recovery workflows. Doing that means processing over 200 million metrics samples per second across all customer environments, because a lag has to be seen before it can be drained.  SUNK , CoreWeave's Slurm on Kubernetes offering, dynamically reallocates GPUs across training and inference workloads on the same cluster as demand shifts, so freed capacity goes back to work the moment it's available, as detailed in our NVIDIA Vera Rubin NVL72 deep dive , rather than sitting idle between jobs. These are the capabilities behind CoreWeave's MLPerf Training v6.0 result on NVIDIA GB300 NVL72, where Mission Control held a consistent performance baseline and SUNK's NVIDIA NVLink-domain-aware placement kept communication local across the NVL72 domain. Preparing for the next generation Craig Falls, Head of Quantitative Research at Jane Street, put it this way : "Our research depends on infrastructure that's both powerful and reliable, and CoreWeave has delivered on this as we've scaled across NVIDIA Hopper and Blackwell . Their ability to deliver highly performant clusters with full cluster observability and a support team that engages deeply on hard problems gives us the confidence to partner...