Agents on Live Hardware: Reason Where You Must, Engineer Everywhere Else

Data center anomaly detection with AI agents

Announcement

Webinar

Podcast

GTC 2026

CoreWeave Forge is here: connect your AI loop and compound every improvement. Read more

Infrastructure

CoreWeave Forge

Solutions

Resources

Log in

| Sign up Contact us

Contact us

Clear

Data center aisle with server racks and a sealed containment pod, overlaid with floating panels of live power and cooling metrics.

An AI data center is a physical system. Power draw, inlet temperature, coolant flow, fan speed, all sampled continuously. Somewhere in that stream is the node that might interrupt a training run. We wanted agents to find it first. The question was never whether to use them. It was where. Point an agent at every part of a problem and you get a system that is slow, and capable of giving you a different answer on Tuesday than it gave on Monday. So we gave them two different jobs: offline, building a deterministic anomaly detector, and live, reasoning only about what it flags. That split turned out to be the most useful thing we learned. An agent can build the tool, or be the tool The familiar version is a coding assistant: describe a problem, and the agent writes something that solves it. When it finishes, the agent is done and the code is what you keep. That is one of two ways to use an agent, and the difference that matters is whether the agent is still there when the system runs. You can use an agent to build the tool . The agent works offline, explores the data, interprets the requirements, and assembles a pipeline from conventional machine learning and data processing components. What ships to production is that pipeline. The agent is not there when it runs. Or you can use an agent as the tool . The agent sits inside the running system. It is called on live data, reasons about what it sees, and produces an answer. Take the agent away and the system stops working. Neither is the lesser choice, and the interesting problems need both. Ours did.

Agents to build the tool Agents as the tool

What the agent does Works offline: explores the data, assembles and tunes the pipeline Runs inside the live system, reasoning over each case

What runs in production Conventional ML and data processing code The agent: model, tools, and context

Best suited for Repetitive, well-defined, constantly running Open-ended, varied, each case genuinely new

Same input, same output? Always Not guaranteed

Cost profile Expensive to build once, then cheap on every data point Cheap to set up, then a real cost on every call

How you explain a result Inspect the code and the model Read the reasoning the agent produced

Figure 1: The two agent modes, side by side.

‍ Detecting anomalies in data center telemetry needs both. It breaks into two questions, and they are not the same kind of question. Find anomalous events. Then explain them, and guide the resolution. The first repeats millions of times a day and should give the same answer every time. The second needs judgment, because no procedure written in advance covers every case.

Figure 2: Workflow schematic. Agents building the detection pipeline on the left, agents reviewing and analyzing flagged anomalies on the right. ‍ Part 1: Agents to build anomaly detection tools Anomaly detection runs continuously, across every channel, forever: five million signals a day. No operations team can watch that. The failures that matter are also subtle: a node degrades slowly, across several channels at once, in a pattern that looks unremarkable on any one of them. Putting an agent in the live path would mean asking it to judge every one of those signals. The problem is that the same input should always produce the same output, and language models work the other way: they sample from a distribution, and even at low temperature the same prompt can return a different answer on a different run. Useful when you want a fresh perspective, disqualifying when you are deciding whether a node is failing. An operations team needs a detector that behaves the same way every time. So we used agents to build the detector instead. What the agent gets is a harness rather than a prompt: a working environment with tools to profile the telemetry, engineer features, train candidate models, and score its own results against a held-out set of labeled incidents. The harness is where the design lives. It fixes what the detector has to achieve, hands the agent the means to get there, and checks every candidate against the same held-out incidents. The agent can rewrite the code. It cannot move the goalposts. It repeats that cycle until the metrics stop improving, and we review what comes out before anything reaches production. Setting that up is where the engineering goes. The features that matter have to come from somewhere, so we sat with the engineers who run the fleet and worked out which signals actually carry a failure and how to represent them. The definition of a real incident is ours too, and that is what the agent scores itself against. Point an agent at a complex system with no requirements and you get something that detects plenty and means nothing. Given that kind of environment, AI agents are genuinely good at this work. One placed first out of 685 submissions on a materials-science benchmark, reasoning about the underlying physics rather than brute-forcing the search. The full write-up is in Tabular ML didn't die. It was just waiting for agents . The interesting part is that a non-deterministic process produces a deterministic artifact. The agent varies. What it hands over does not: feature definitions, model weights, thresholds, post-processing. Ordinary code and a fixed model, versioned, diffable, and reviewed by a person before it goes near production. Run it twice on the same signal and you get the same answer, forever, on every one of the five million that arrive each day. This is not a one-off step, and that is the shift worth noticing. A deterministic tool used to be expensive enough to build that you shipped it and left it alone for a year. When an agent does the building, the same tool gets rebuilt whenever the data moves. New telemetry arrives, or a failure mode presents differently, and the agents revisit the pipeline and refine it. It stops being frozen. An engineer stays involved throughout. They decide what the tool has to do, check that what comes back measures the right thing, and send it round again when it misses. The agent makes each round fast. The...