Patching at Fleet Scale, Twice: How DigitalOcean Closed Januscape and the AMD Safe RET Issue Without Customer Impact

Patching at Fleet Scale, Twice: How DigitalOcean Closed Januscape and the AMD Safe RET Issue Without Customer Impact | DigitalOcean

© 2026 DigitalOcean, LLC. Sitemap .

Trust & Security

Patching at Fleet Scale, Twice: How DigitalOcean Closed Januscape and the AMD Safe RET Issue Without Customer Impact

By Tim Lisko

Deputy CISO

Updated: August 24, 2026 12 min read

<- Back to blog home

Setting the stakes

In early July, security researcher Hyunwoo Kim discovered Januscape (CVE-2026-53359), a flaw in KVM’s handling of nested virtualization that could allow a malicious guest to escape into the host hypervisor. It was disclosed publicly on July 6 via the Linux oss-security mailing list . For a cloud provider, a guest-to-host escape is the most serious class of vulnerability there is: the hypervisor is the boundary that keeps each customer’s workloads isolated from each other, and from our infrastructure itself.

We responded, patched the entire fleet in eight days with zero confirmed customer-facing impact, and drafted a post about how we did it. Then, before we could hit publish, it happened again. In late July we learned of a second and unrelated vulnerability affecting our entire AMD hypervisor fleet, that could not be livepatched. Roughly 1,600 hypervisors needed a kernel update and a reboot.

So now this story is about two responses, three weeks apart. The first built the muscle. The second proved it was repeatable, at a larger scale, and on a harder constraint. Here’s how both played out, and why two of the most serious vulnerability classes in cloud computing ended up feeling like just another couple of weeks for us.

Act one: Januscape

The fast path: fleet-wide livepatching

Our response kicked off the same night the vulnerability was disclosed. When public exploit code surfaced late in the evening of July 6, the Kernel Engineering team was paged and dug in immediately. Engineers reproduced the exploit in an isolated environment, confirmed which kernel lines were affected, and built the first working livepatch before 1 AM, roughly 45 minutes after answering the page. Livepatching lets us fix a running kernel in place, with no reboot, no migration, and no observed disruption to the customer. A few hours later, patches for the kernel versions (6.1 and 6.12) that run the majority of our hypervisor fleet were ready to ship. For the remainder, we had to follow a different approach; more on that below.

The rollout followed the discipline we apply to any change that touches every host we operate: patches were built, validated in our staging environment, and promoted through a gradual production ramp with health checks and commitment to pause at any sign of regression. Within roughly 48 hours of the patches being built, thousands of hypervisors were protected and the 6.x fleet reached full coverage shortly after.

For most of the fleet, the story was that the vulnerability was quietly closed before most customers had finished reading the CVE announcement. The exception is where the harder work began.

The long tail: hosts with no upstream fix

Every large fleet has a long tail, and how you handle the long tail under pressure says more about your engineering culture than how you handle the easy 95%.

A few hundred dedicated hypervisors were running a 5.10-series kernel that the upstream fix didn’t cover. As there was no upstream patch to cherry-pick for that kernel line, the Kernel Engineering team wrote an entirely new livepatch implementation from scratch. It passed initial testing, but as it didn’t correspond to an upstream-reviewed fix, there was a level of risk above what we normally accept for a change to a running production kernel.

Faced with a working-but-unproven patch on one side and a slower-but-durable path on the other, our team made the call to evacuate and upgrade the affected hosts entirely . This meant live-migrating customer workloads off each machine, repaving with a new OS and modern kernel, and returning it to service fully patched. This is the heaviest maintenance operation we can perform on a host: a full repave with each machine out of service for the duration of the rebuild. The custom livepatch was held in reserve and hardened in parallel rather than rushed to production.

Our customers choose our dedicated fleet because they have the highest expectations for isolation and availability, and the least tolerance for unanticipated downtime. Shipping an unproven kernel patch to the customer segment least able to absorb a destabilizing event would have traded our security risk for their availability risk. The solution was clear: take the path that is slower for us, addresses the security risk, and is designed to be invisible to customers.

Turning a kernel problem into a capacity problem

Evacuating hosts means live migrating running workloads, and in a hot, heavily utilized dedicated fleet, spare capacity is scarce by design. At the maintenance pace the fleet’s capacity could support, full remediation was projected to take an unacceptable amount of time at three and a half weeks. To move faster, without sacrificing any quality of work, the Security, Capacity, Product, and Fleet Management teams got creative:

Temporary refleeting : Servers were moved from an adjacent premium pool into the constrained fleet to serve as evacuation targets, with a plan to return them after the burn-down.

Reclaiming stranded capacity : Teams audited the region for hosts sitting idle (machines held for completed projects, stale reservations, hosts stranded in standby after prior maintenance), validated their health, and released them back into production as landing space.

Raising maintenance concurrency : With real evacuation targets in place, the number of hosts that could be safely drained in parallel nearly tripled.

As a result, the projected remediation time dropped to less than 1.5 weeks, and the actual burn-down ran faster still. All of it moved through our standard live-migration machinery, with no customer-visible interruption observed and capacity buffers preserved throughout.

In order to keep the accelerated remediation running smoothly, an incident was formally declared. This creates a single coordination point, clear ownership, structured status reporting, and an auditable record for every capacity and fleet decision made under pressure. Additionally, our CloudOps team watched the evacuation pipeline 24/7. Automated fleet maintenance at elevated concurrency...