Reka Inverse Dynamics Model For Interactive World Models

Reka Inverse Dynamics Model for Interactive World Models

← Back to Blog

Oct 2, 2026

Reka Inverse Dynamics Model for Interactive World Models

Reka Inverse Dynamics Model for Interactive World Models

Model weights (Apache 2.0): RekaAI/Reka-Inverse-Dynamics-Model on Hugging Face Simulated environments are essential for robotics. They enable safe training and testing of robots. However, even the most sophisticated ones fail to capture the “noise” of the real world because of idealised physics and unnatural visuals.

Simulated environments are essential for robotics. They enable safe training and testing of robots. However, even the most sophisticated ones fail to capture the “noise” of the real world because of idealised physics and unnatural visuals.

Interactive World Models are different. They are visually more realistic, and can model more complex phenomena such as folding materials or squeezing towels. By embedding digital replicas into these dynamic environments, robots can execute commands through direct interaction. Here is the challenge though. True interactivity requires a world model to accurately respond to low-level commands (e.g., moving left or right). To solve this, we introduce the Reka Inverse Dynamics Model (RIDM), which extracts low-level commands directly from real-world video data to act as ‘captions’ describing physical changes in consecutive frames. RIDM is a compact, efficient model that processes short clips and extracts low-level motor commands like moving forward, sideways, turning or tilting. Even though the model is entirely trained on video games, by decoupling visuals from motion, its predictions transfer to real-world videos.

Predicted key probability W forward 0.85 A move left 0.34 S back 0.01 D move right 0.76 Shift walk 0.01

turn ( yaw ) 16.9° left ← left right → tilt ( pitch ) 1.4° up ← down up → First window of the clip

Blue bars show five movements; each carrying the name of the game control that produces it. The green dials show the camera turn and the camera tilt, in degrees.

1

Learn from the game

W

A

S

D

turn 12°

recorded gameplay

video, the keys held, the camera angles

video and the true actions

train the IDM

objective: from two frames, predict the keys held and how far the camera moved

IDM

2

Label our real video

our real video

no action labels at all

video only, no actions

IDM

the weights from step 1

what the camera did

forward

turn 12°

predicted labels

3

Train the world model

forward

turn 12°

the same video, now labelled

the actions came from the IDM

video and actions together

train the world model

on video and action pairs

world-language-action

understands actions, generates video

the same weights, carried over

the video, and the labels it predicted

Figure 1 — How RIDM can be used to train interactive world models.

1

Learn from the game

W

A

S

D

turn 12°

recorded gameplay

video, the keys held, the camera angles

video and the true actions

train the IDM

objective: from two frames, predict the keys held and how far the camera moved

IDM

the same weights, carried over

2

Label our real video

our real video

no action labels at all

video only, no actions

IDM

the weights from step 1

what the camera did

forward

turn 12°

predicted labels

the video, and the labels it predicted

3

Train the world model

forward

turn 12°

the same video, now labelled

the actions came from the IDM

video and actions together

train the world model

on video and action pairs

world-language-action

understands actions, generates video

Figure 1 — How RIDM can be used to train interactive world models.

The data We recorded gaming footages at 1280 × 720, 48 frames per second, with 11 ground-truth key commands, the mouse movement and the exact camera angles for every frame. The annotations are provided by the engine. What it looks like

Pool

Keys held

Turn

Tilt

Must travel

Rows

Forward

one pool with Turn, about half each

exactly W

2° or less

2° or less

walking pace or more

104,646

Turn

the same pool as Forward, not a second 104,646

none at all

10° or more

2° or less

1 unit or less

104,646

Sideways or back

A and D step sideways, S steps back

one of A, D, S

2° or less

2° or less

2 units or more

53,429

Tilt

the scarcest motion we train on

none at all

2° or less

10° or more

1 unit or less

1,554

Sideways + turn

added last, to tell a step from a turn

one of A, D

5° or more

no demand

2 units or more

50,000

Figure 2 — Gameplay clips with the associated ground-truth annotations.

Forward

one pool with Turn, about half each

Keys held

exactly W

Turn

2° or less

Tilt

2° or less

Must travel

walking pace or more

Rows

104,646

Turn

the same pool as Forward, not a second 104,646

Keys held

none at all

Turn

10° or more

Tilt

2° or less

Must travel

1 unit or less

Rows

104,646

Sideways or back

A and D step sideways, S steps back

Keys held

one of A, D, S

Turn

2° or less

Tilt

2° or less

Must travel

2 units or more

Rows

53,429

Tilt

the scarcest motion we train on

Keys held

none at all

Turn

2° or less

Tilt

10° or more

Must travel

1 unit or less

Rows

1,554

Sideways + turn

added last, to tell a step from a turn

Keys held

one of A, D

Turn

5° or more

Tilt

no demand

Must travel

2 units or more

Rows

50,000

Figure 2 — Gameplay clips with the associated ground-truth annotations.

The model Our model disentangles visuals from motion by projecting inputs into optical flow space. Since our goal is to use RIDM to automatically annotate real-world footages, not 3d gameplays, the method needs to transfer from games to real-world. We also compare that approach with a direct one that is working in the pixel space. To differentiate both, we use names: flow-based and pixel-based model. Training both models on random gaming footages will not work either. Successful approach requires data filtering and resampling. For instance, some actions such as backward or sideways strafe movements are much more rare than simple forward movements. Suitable resampling helps in learning these less common movements. Both models are compact. The flow-based model has 2.8m (2,785,499) parameters; with only 1.8m (1,795,337) trainable parameters and about 990k frozen that constitutes RAFT-small. The pixel-based model is fully trainable and has around 9.8m (9,836,063) parameters. Pixels

the IMPALA trunk, the one VPT uses for its policy

every pixel

IMPALA trunk...