Built for Mass Scale: Hard-Won Lessons from Teams Running High Volume Inference Workloads in Production

Built for Mass Scale: Hard-Won Lessons from Teams Running High Volume Inference Workloads in Production | DigitalOcean

© 2026 DigitalOcean, LLC. Sitemap .

AI/ML

Built for Mass Scale: Hard-Won Lessons from Teams Running High Volume Inference Workloads in Production

By Hasan Nabulsi

Content Marketing Manager

Updated: June 30, 2026 5 min read

<- Back to blog home

Moving AI from a flashy demo to a high-volume production environment is a transition filled with hidden technical debt and infrastructure challenges. There’s a difference between calling the OpenAI API in a weekend prototype and serving 50,000 concurrent users who need sub-200ms latency, graceful fallbacks, and reliable output every single time. It is rarely a “model problem.” Instead, it is a problem of decisions, trade-offs, and architecture.

At DigitalOcean Deploy 2026 , we hosted a panel of engineering leaders from Workato , Hippocratic AI , and ISMG. Moderated by Karnik Modi, DigitalOcean’s Senior Manager of Engineering, panelists shared the lessons they’ve learned while running inference workloads at scale.

The session focused on managing P99 latency spikes in real-time interactions, restricting agent permissions to prevent “admin” vulnerabilities, and ensuring infrastructure is policy-aware before production traffic hits. These insights move beyond model performance to address the orchestration and security guardrails required for reliable, mass-scale AI.

Watch the full recorded session from Deploy 2026 :

View YouTube video

The Built for Mass Scale Panelists

Each panelist represents a company operating at the frontier of production AI, where the gap between a working prototype and a reliable system serving real users is the entire challenge. From orchestrating autonomous agents across thousands of enterprise applications to running real-time clinical voice conversations where latency is a patient-safety issue to deploying AI-powered intelligence across a global cybersecurity media network, these teams have confronted the infrastructure, governance, and architectural decisions that only surface at scale.

Oscar Wu — AI Research Technical Lead, Workato Research Lab

Workato is an enterprise integration platform that connects over 14,000 applications and has orchestrated more than one trillion automated tasks, and its AI focus has shifted to agentic orchestration‚—building, deploying, and governing autonomous AI agents that can reason, act, and execute multi-step workflows across enterprise systems without writing code. At production scale, Workato’s AI Research Lab confronts the hard problems of agent governance, tool selection accuracy across large tool inventories, and keeping inference fast and cost-efficient under sustained load. Workato’s AI Research Lab runs its inference workloads on DigitalOcean’s AI-Native Cloud, where it achieved 67% lower inference costs and 77% faster time-to-first-token on NVIDIA Hopper GPUs.

Debo Datta — Co-founder, Hippocratic AI

Hippocratic AI builds safety-focused generative AI voice agents for healthcare, handling patient-facing tasks like post-discharge follow-up, chronic care management, medication review, and clinical trial coordination. Because each clinical conversation can span hundreds of turns in real time, the company’s core infrastructure challenge is maintaining sub-second latency at scale so that voice interactions remain empathetic and natural rather than robotic. Hippocratic AI was a design partner for DigitalOcean’s inference engine and runs its patient-facing workloads on the platform, powering over 20 million patient interactions with 40% lower latency.

Dan Grosu — CTO/CISO, ISMG

ISMG (Information Security Media Group) operates 38 media properties focused on cybersecurity, IT, and AI, serving security professionals across sectors like banking, healthcare, and government worldwide. The company has built its own enterprise AI platform, Apollo, which processes thousands of pages of interview transcripts and conference sessions to produce cybersecurity market intelligence and strategic assessments.

AI Has Gone From “Secret Sauce” to Standard Infrastructure

The panelists agreed that the conversation around AI has changed over the last 18 months. It’s not enough to simply “have AI”; the focus is now on whether your stack can actually support it at scale.

“AI is no longer a competitive edge. It’s competitive infra. The question has shifted from ‘Should we ship AI features?’ to ‘Is our enterprise stack actually going to work with AI agents?’” — Oscar Wu, Workato

When AI becomes the infrastructure, the GPU bill that powers it stops being an experimental line item and starts being a core operating cost. For companies like Hippocratic AI, which focuses on healthcare, AI is the core product, but it functions as a significant cost center due to the massive GPU requirements. The challenge is turning that cost into a safe, reliable revenue generator.

What Works at Ten Requests Fails at a Million

Everything looks efficient in a controlled environment. However, the panelists identified specific “bottlenecks” that only appear once real production traffic hits.

The Agentic Identity Crisis

When an AI agent has access to 5 tools, it works perfectly. When it has access to over 50, it starts to falter. Oscar noted that agents often begin picking the wrong tools because names sound similar or they lack a governed policy for execution.

The Latency Trap

At Hippocratic AI, latency is more than a technical metric—it’s a patient safety concern. Their safety-focused LLM handles multi-turn clinical conversations where delays erode the empathy and trust required for effective care.

“When volumes are low, you don’t really test the limits. Only when you scale do you notice the P99 latency. If you’re on a clinical phone call with 200 turns and your latency isn’t great, you are experiencing a slowdown in every single call.” — Debo Datta, Hippocratic AI

Plan for the Architecture That Hasn’t Shipped Yet

Scaling AI inference is a constant cycle of redesigning your stack. As new architectures emerge, the winners will be teams whose foundations are agent-ready, policy-aware, and structured for trust.

“Let AI surprise you. If you have your data organized, you are in the prime position to leverage AI to great success.” — Dan Grosu, ISMG

Tighten Agent Permissions to Shrink the Blast Radius

As inference becomes more distributed, the security stakes rise. Dan from ISMG...