Video Captioning at scale: 600 TB in 95 minutes with Anyscale on CoreWeave

Video Captioning at Scale: 600 TB in 95 Minutes | CoreWeave Blog

Announcement

Webinar

Podcast

GTC 2026

CoreWeave recognized as a Visionary in the Gartner® Magic Quadrant™ for Cloud AI Infrastructure. Read the report

Products

Data and storage

Infrastructure control

Runtime acceleration

Model and agent development

Mission control

Solutions

Pricing

Resources

About us

Contact us Login

Contact us Login

Clear

Data processing is now a GPU workload Large language models (LLMs) have moved well past chat. Physical AI , drug discovery, recommendations, and creative generation all need the same thing first: internet-scale multimodal data that has been found, filtered, and turned into something a model can train on. That work no longer fits on just CPUs. Video, audio, LiDAR, and satellite data have to be decoded, filtered, featurized, and normalized, and at exabyte scale that means GPUs . Standing up the infrastructure to do it usually costs a team weeks or months before they see a single result. Anyscale and CoreWeave built a video captioning pipeline across millions of clips to find out how fast that path could actually be. From account signup to a running production job: under 24 hours. Here’s how it went, and what the numbers looked like at 1,600 GPUs. Two infrastructure layers for video captioning Two layers made this run, and each one made the other quicker. They’re worth separating before the timeline. What Ray handles Ray OSS and Anyscale own the workload layer. Ray schedules tasks and actors across the cluster, streams data through the pipeline, and scales the number of workers to match demand. What Ray can’t see is the hardware. It doesn’t know that a node is thermally throttling, or that one straggler GPU is quietly holding up a stage. What CoreWeave handles underneath The CoreWeave layer handles the AI infrastructure. CoreWeave Kubernetes Service runs Kubernetes directly on bare metal nodes , with GPU drivers, network and storage interfaces, and observability plug-ins already installed, which is why the Anyscale install took an hour instead of an entire quarter. CoreWeave Mission Control runs continuously across the fleet, evaluating node and cluster health, catching stragglers before they stall a job, and replacing unhealthy nodes automatically. Storage sits at the same layer. CoreWeave AI Object Storage (CAIOS) holds the dataset as a single global namespace with no egress, request, or transaction fees, and the CAIOS Local Object Transport Accelerator (LOTA) runs on every CoreWeave node to bypass the standard gateway and cache fetched objects on local NVMe. That keeps repeated reads, model weights especially, off the network and close to the GPU. Ray Data decides what runs where and keeps the pipeline streaming. CoreWeave keeps the AI infrastructure healthy, the storage fast enough to feed it, and the cluster whole when a node drops. GPUs are easy to allocate and hard to keep busy, and the gap between those two states is infrastructure the workload layer never has to compensate for. From account signup to production job in under 24 hours Standing up GPU infrastructure at this scale is normally measured in weeks: procurement, Kubernetes install, driver and network validation, storage plumbing, and finally a scheduler that can keep the GPUs fed. This took less than a day. Hour 0. CoreWeave provisioned the reservation: a dynamic cluster scaling to 1,600 NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, 200 AMD Turin 9655P nodes at 192 vCPUs each, and CoreWeave AI Object Storage as the data lake. Hour 1. Anyscale went onto CKS as an operator install because CKS already handles the managed control plane, GPU drivers, networking, and node lifecycle. Hour 2. Orchestration test: CKS ran the node layer while the Anyscale operator ran Ray cluster lifecycle, GPU autoscaling, workload placement, and the developer interfaces into the reservation. Hour 10. We authored the pipeline in Ray Data as a single streaming job, with CPU decode and filter feeding GPU captioning. We developed it interactively against the live cluster in Anyscale Workspaces. Hour 15. We tuned it for GPU utilization and throughput using Anyscale Observability, CoreWeave AI Object Storage bandwidth, and faster model loads. Hour 20. We promoted the same code to a fault-tolerant Anyscale Job, no rewrite, with retries and autoscaling for the full-scale run.

The pipeline itself had two stages. CPU stage: video decode, scene-boundary splitting, and filtering. GPU stage: caption and annotation inference with Qwen3-VL-8B over the resulting clips.

From Ray Core to Ray Data Our validation started small. Before we pointed anything at the full 1,600 GPU reservation, we ran the pipeline against a 600 GB dataset on a fixed pool of 256 GPUs, and that’s where the first bottleneck surfaced. The first pass at video captioning used Ray Core. Remote functions read the video dataset and decoded keyframes on the CPU: @ray.remote(num_cpus=1) def read_and_decode_shard(path: str) -> List[Dict[str, Any]]:

Decoding logic here

Inference ran on Ray actors. An actor is a Python class that Ray pins to a long-lived worker process, so the expensive state gets initialized once and reused across calls instead of rebuilt on every request. Here each actor held one GPU and one loaded vLLM engine. We created and managed those actors by hand. Our driver maintained a fixed list for round-robin scheduling, so this implementation did not automatically flex with demand: @ray.remote(num_gpus=1) class CaptionActor:

vLLM inference logic here

Manual actor management

actors = [CaptionActor.remote() for _ in range(num_gpus)]

Ray Data took that management away. It streams execution and scales resources to workload demand. Reading directly from a Parquet manifest decoupled ingestion from video decoding and enabled better pipelining:

Direct, optimized data ingestion

ds = ray.data.read_parquet(input_path, **read_kwargs)

Applying decode logic via a simple map operation

ds = ds.flat_map(decode_row, num_cpus=1)

Batched model inference came built-in. Instead of standing up and orchestrating vLLM engines through Ray actors, we passed the config to a batch processor and the pipeline collapsed to read_parquet, flat_map, processor, write_parquet: vlm_processor = build_processor( vLLMEngineProcessorConfig(**config_kwargs), preprocess=vlm_preprocess, postprocess=vlm_postprocess, ) ds = vlm_processor(ds)

Scale the dataset 1,000x and CoreWeave AI...