World Model Data Pipeline

World Model Data Pipeline

← Back to Blog

Jul 10, 2026

World Model Data Pipeline

World Model Data Pipeline

A key ingredient of training a new world model is the data. At Reka Labs, we have a dedicated data team responsible for preparing petabytes of video and image data for our model training. Here we outline key components of our data pipeline and some challenges we face. In follow-up blog posts, we will delve deeper into specific parts of the data team’s charter. Data Pipeline Outline & Key Challenges Preparing data for world model training requires many processing stages, starting with sourcing, then moving through various steps of analyzing, filtering, transforming, and annotating video or image data. When the final artifact is delivered, it is ready to be used by the research team training our “omni” world model from scratch. The key challenge with the data pipeline is the tyranny of the project management iron triangle: cost, time, and quality — pick two. On one hand, we want deep analysis of the data to ensure only high-quality videos reach the model (quality), but, on the other hand, it is computationally expensive (cost). While training our models, we are quickly learning what works and what doesn’t, resulting in fast turnaround and the need for rapid adjustments to the data pipeline, shortening our timelines (time). Managing these three goals is a fine balancing act. Furthermore, we are operating both on a few large datasets and on hundreds of small, heterogeneous datasets, and different modalities. Because of the large volume of data and short timelines, we must approach the data processing problem with a scalability mindset. We constantly analyze where the next bottleneck is and put just the right amount of effort into addressing it until something else becomes the most pressing issue. On the other hand, the heterogeneity of the datasets increases the challenge of automation. Each new dataset brings new corner cases, requiring us to be ever vigilant about tooling reliability issues, and to constantly adjust our AI harnesses. Stage 1: Sourcing The first stage of our data pipeline is data sourcing . This stage immediately poses a series of challenges. First, we must discover datasets with the right characteristics required by model training, and in the appropriate volume. Second, we need to standardize the dataset’s metadata, format, and location, so that its processing through the data pipeline can start in earnest. We developed a formal specification of what the data pipeline tooling expects, and each dataset must be converted to match the specification. We call this process data ingestion . One of the key data providers for us is Reka’s Claru , a data marketplace that provides purpose-built datasets for frontier robotics, embodied AI, and world models. Overall, the sourcing stage is in many ways the hardest to automate, and requires a lot of human review. We developed detailed guidance for our AI agents to help deal with the dataset heterogeneity, but pretty much every single new dataset poses a new corner case we haven’t covered yet, requiring us to constantly update our data ingestion AI harness. Stage 2: Analysis There are multiple layers of data analysis. During ingestion, while we are reformatting a given dataset’s metadata to match our formal spec, we are also analyzing it with ffmpeg to produce a standard set of basic metadata. We obtain information about each input video such as width, height, number of frames, fps, codec used, and whether the video is interlaced. We also do a series of “health checks,” ensuring that e.g., the video has correct key frames and is not corrupted. This process is very fast and compute-efficient. Later on in the pipeline, we do more heavyweight analysis that inspects the video contents and requires GPU computation. For example, we run shot boundary detection or compute various other quality scores. We also leverage Claru for datasets with strict quality requirements, such as those used for supervised fine-tuning. The aforementioned iron triangle of competing goals fully manifests itself in the analysis stage of our pipeline. Often, we must rapidly deploy new, augmented data analysis, and it is extremely expensive to perform a full analysis on every input video. Stage 3: Filtering One of the key benefits of doing analysis is data filtering — corrupted or poor-quality videos never reach the model. What counts as “poor quality” is decided by a bank of 9+ Quality Analysis models, each scoring every clip on a different aspect such as aesthetic quality, presence of text, and others; the combined scores bin each clip into a quality rank. The same stage also computes video embeddings, which we use to deduplicate near-identical clips. This is an interesting chicken-and-egg problem: to save processing costs, we want to filter out videos, but to filter out a video, we must analyze it. Since running a full analysis is computationally expensive, we first analyze each given video “only a little bit” to determine which videos even deserve full analysis. Overall, this poses an important challenge of proper sequencing of the various processing steps to maximize cost savings. Stage 4: Transformation Training a model on videos requires GPU efficiency, which in turn requires frame-level precision of the input video lengths and distribution, as well as appropriate resolution. This means input videos must be segmented into appropriate lengths, also taking into account the scene boundaries detected during the analysis stage. Furthermore, some videos may require post-processing, such as removal of black bar padding on the sides, resolution scaling, or re-encoding to an appropriate codec. Transforming input videos at our scale ultimately runs into the same challenges as the analysis stage, but even more severe ones. We are not only inspecting the videos, but also their re-encodings. It is also worth noting that many of the video transformations we do rely on ffmpeg , and the exact ffmpeg incantations we developed to transform videos at scale, at low cost, and to handle all the myriad corner cases coming from our varied datasets are one of the many key elements of our pipeline. Stage 5: Annotation World models cannot be trained only on videos and images; they must also be annotated with appropriate information. For example, we require a description of what’s happening in the video, or a taxonomy of what is in the video, e.g., people, cars, nature....