Everyone's adopting AI agents. Almost no one can prove the ROI. Here's why

Everyone's adopting AI agents. Almost no one can prove the ROI. Here's why | Databricks Blog Skip to main content

Summary

Productivity used to mean how much work one person could do alone. In an agentic world, it means how much a person can do with a team of AI assistants working alongside them, without sacrificing quality.

The AI "ROI gap" exists because the people buying AI and the people using it are often different, and most organizations are still in a trial-and-error phase of figuring out which use cases actually work.

The agents that create real value are the ones with business context: your data, your history, your constraints. Without it, an agent can only give generic advice, not decisions your company can act on.

Amber Roberts, Staff Technical Marketing Manager at Databricks, has spent the last several years watching how AI actually gets used inside organizations, not just how it's pitched. In this conversation, she talks through why "faster" is the wrong way to think about agentic AI , why so many companies are struggling to show ROI on their AI investments, and what has to be true for an AI agent to be useful at work instead of just impressive in a demo. Why speed is the wrong way to measure AI productivity Most conversations about AI productivity are still about speed: doing the same tasks faster. Why is that the wrong lens for thinking about agentic AI, and what other KPIs should teams consider? Amber Roberts: Productivity is tied to whatever a team's actual KPIs are, and that's different depending on where you sit. For some teams, the work is detail-oriented and speed isn't the biggest issue. For others, latency is the whole ballgame. Even the word "speed" means different things: speed of development versus speed of inference, for example. Engineering productivity and marketing productivity aren't the same thing, and they shouldn't be measured the same way. So I'd push back on the idea that there's one right KPI to swap in. It depends on the team, and honestly, it depends on the person. Someone who's learning a domain from scratch is going to use AI differently, and value different things, than someone who already knows exactly what they want to build and just wants to automate the tedious parts. I'm skeptical of headlines that go all in on "this is the wrong way to think about it." There's a lot of gray areas. Even people on the same team can have different priorities for what they want out of AI, depending on their own expertise. The AI ROI gap: why adoption is outpacing proof There's a growing body of research showing companies are adopting AI fast but struggling to show real ROI. Why does that gap exist, and what does it reveal about how most organizations are deploying AI agents today? Amber Roberts: The people buying AI and the people using it day to day are often different people. The buyers don't want to be late to the game. Their job depends on staying at the cutting edge of technology, even if they're not the ones implementing it. If they see competitors buying and deploying AI, they're going to want to make a deal happen. That's true of any new technology: nobody wants to be the one left behind. Then, over six months to a few years, you start to see whether it actually built ROI, whether it was worth what was spent. That's the phase a lot of organizations are in right now: looking at what they bought, rolling it out broadly, and figuring out which use cases it's effective for. A lot of it is trial and error. But that trial and error is still better than being six months behind the competition. Redefining productivity for an agentic world How do you define productivity in an agentic world, and how is that different from how enterprises have measured it for the last 20 years? Amber Roberts: The classic definition of productivity is how effectively inputs like labor, time, and capital turn into outputs like goods and services. What's interesting is that AI doesn't output a good, it outputs a service, optimizing something behind the scenes. The key shift is this: productivity used to be measured by how much work one person could do. Now it's how much work a person can do with a team of assistants at their disposal. We see this even in how we evaluate people, in interviews or take-home assessments. It's no longer impressive to see how much someone produced alone in four hours, because there's a visible difference between that and what someone produces when they're leveraging AI well. That's the new baseline. But it doesn't replace deep knowledge of the subject matter. It just means the same instinct, the same taste, the same expertise, is now applied at a different scale. Where I see the biggest jump is when people take everything that used to eat their day, the recurring, schedulable, "hire an assistant to do this" work, and automate it so they can focus on the harder, more creative parts of the job. The parts that require actual pushback and judgment, because AI's default mode is agreeable. It'll say "sounds good" to almost anything. That's useful for an assistant, but it's also exactly why the parts of a job that require someone to argue, to push back, to say "no, that's not right," matter more now, not less. That's also the part that gets missed when people report productivity gains without a quality metric attached. Productivity is technically linked to efficiency, and it assumes you're not sacrificing quality to get there. If a company says they're 40% more efficient, that's really only meaningful if quality held steady. Otherwise you might just be taking the same amount of time to produce the same result, with more moving parts. That tracks with what we're seeing in content specifically: when you push volume up, without extra attention to the details that make something good, quality tends to slip. The tools don't always keep up. Why business context beats raw task speed A lot of AI tools can execute a task. What does it actually mean for an agent to "understand business context," and why does that matter more than speed? Amber Roberts: I think about it like comparing an AI agent to a real human assistant. If you ask an assistant to do something and they don't have your context, your schedule, or a sense of what's happened over the last few...