World Motion Models

A set of SE(3) trajectories approximates the 4D world.

Much of what we experience in the world can be approximated by a single SE(3) trajectory or a composition of several trajectories. For example, in the scene above, a single trajectory captures the motion of the camera or rigid cube. Combining trajectories lets us describe articulated furniture or a robot arm, or use skinning to represent the linked motion of a human, hand, or animal. Even general non-rigid, unstructured objects, such as the deforming bear and towel, fit this picture: a small set of SE(3) basis motions can be blended to approximate continuous deformation (DynMF, MoSca, SoM).

State and action

We encode the where, when, and what of each world element as tokens, organize them into a table, and learn a sequence model over this representation.

A pose token at every step

Each token is an SE(3) pose paired with a per-trajectory feature, encoding the where (T ∈ SE(3)) and the what (F ∈ ℝ^D) at each time step. The grid above represents a dynamic 3D scene as an M × N token tensor: rows index the entity and columns index the time step. Time indices are either consecutive within the causal window, or disjoint virtual indices serving as non-causal context (goal frames, retargeting exemplars).

WMM unifies flexible sequence modeling in a single diffusion model, conditioning on any partially known context.

Training

During training, we corrupt each token independently with a per-token noise level; WMM then predicts the clean tokens.

Inference

At inference, one trained WMM can condition on any partially known context in the token table and generate the unknown tokens. By changing which tokens are observed, the same model flexibly supports all the modes below.

• (i) Future Prediction P(s′ | s) / Policy P(a | s)

  Observe history, predict future tokens: state tokens forecast the scene; action tokens make the same weights serve as a policy.

  Noisy Clean

  Action

  Robot State

  Env State

  Time →

• (ii) Action-Conditioned Prediction P(s′ | s, a)

  Keep actions observed and predict future states: WMM becomes a learned forward dynamics simulator, or world model, that supports planning and model-predictive control (MPC).

  Noisy Clean

  Action

  Robot State

  Env State

  Time →

• (iii) Motion Planning / Infilling

  Observe sparse keyframes and generate the missing motion between them for planning or interpolation.

  Noisy Clean

  Action

  Robot State

  Env State

  Time →

• (iv) Retargeting, Inverse Dynamics P(a | s′, s), etc.

  Observe a partial entity set and generate the rest, enabling retargeting, partner prediction, or inverse dynamics.

  Noisy Clean

  Action

  Robot State

  Env State

  Time →

Our transformer architecture enhances conditioning on flexible per-token noise levels and scales to thousands of tokens through factored attention and registers.

One general architecture, trained the same way, handles all of the tasks below.

Learned Retargeting — Human–Humanoid Joint Modeling

Context: exemplars causal temporal window

human
motion robot
links

Time →

known generated

WMM can retarget human motion to humanoid robots (and vice versa). It models the joint distribution of the human motion and its physically plausible, retargeted humanoid motion. Most importantly, we do not have to tell WMM which human joints correspond to which robot joints: a few paired context exemplars suffice, so the approach is morphology-free. At inference, WMM generates the humanoid motion from these exemplars and the human motion.

→

Inference speed FPS, higher better

GMR

35.0

PHUMA

2.3

Ours

87.0

Tracking success rate higher better

GMR

0.79

PHUMA

0.83

Ours

0.87

Faster and safer retargeting: compared with optimization-based methods — GMR is real-time but less accurate and unsafe; PHUMA is more physically plausible but slow and can fail — our data-driven prior, trained on a curated retargeting dataset, is both faster and safer. We score quality by feeding each kinematic trajectory to a frozen low-level controller (TWIST) and rolling it out in physics simulation.

motion example ◀ 1 / 2 ▶

GMR PHUMA Ours Kinematics
(output) Physics Sim
(via TWIST)

Input human

WMM can be used to retarget noisy motion reconstructed from videos:

video example ◀ 1 / 2 ▶

Trained on multiple humanoids: swap the context exemplar at inference — human+G1 or human+H1 — and the same model retargets different humanoids, each with a different number of links and morphology.

Motion Planning on Many Morphologies

Arm pose

Arm pose

Obstacle

Time →

known generated

WMM can plan motion for robot manipulators. It plans collision‑free motion by modeling the motion of each link of the robot and the obstacle. Given the first and last robot states together with the obstacle’s motion in the middle, WMM generates the intervening robot motion.

→

One model, many morphologies: a single WMM models robot arms with different morphologies.

LangTable — WMM as Policy and Forward Dynamics for Model Predictive Control (MPC)

Action

Robot State

Env State

Time →

known generated

WMM can act directly as a robot policy. It predicts actions alongside future states. LangTable is a language-conditioned tabletop pushing benchmark. WMM models the SE(3) state of every block together with the end-effector — its state (green) and action (red) — and pins one context time step to the successful ending frame. At inference, WMM observes only a short history and generates the rest; we read the policy directly from the predicted action row.

Policy rollout results — please see our paper for quantitative evaluation.

‹ ›

Task: “move the red moon below the yellow pentagon”

Task: “pull the red pentagon apart from the clump”

Task: “push the red pentagon up and to the left”

Task: “put the yellow pentagon to the bottom right”

Task: “slide the green star next to the red moon”

Action

Robot State

Env State

Time →

known generated

Plain policy MPC

success rate (%) ↑

76

80

+4 pp

mean steps ↓

92.7

87.2

−5.5

WMM can also serve as a world model for planning. Given the actions as well, the same WMM runs as a forward dynamics simulator and predicts the resulting future states (mask, right). This unlocks model-predictive control: the model first imagines a successful goal (below, left), then scores the action chunk’s final frame against it with a value function based on their difference and optimizes the chunk with both sampling-based and differentiable gradient-based MPC. This enables faster rollouts at a higher success rate versus the plain policy (note the rapid push of the cube in the video below).

Task: “push the blue cube to the top right”

Value Function
(Distance) ↔

TraceGen — Dense 3D Scene Future Prediction

keypoint 1

keypoint 2

keypoint 400

Time →

first frame (t = 0) generated, 32 steps

WMM can predict the 3D future of general scenes. It handles dense prediction, querying hundreds of trajectories at once. This formulation subsumes existing scene-future-prediction works such as MoMaps, TraceGen, and PointWorld. At inference, the input is the first-frame RGB‑D image; the prediction is the future of the tracks at dense query positions.

→

Known input: first-frame RGB‑D

Generated output: future 3D tracks

Results — each scene shows 2D camera tracks (top) and 3D point‑cloud tracks (bottom); the two columns compare the TraceGen baseline and Ours.

TraceGenOurs

‹ ›

Epic-Kitchens · scene 1

DROID · scene 1

Epic-Kitchens · scene 2

DROID · scene 2

TraceGen Ours

Droid ↓ lower is better (×100)

0.206

0.151

MSE

27% improvement

1.289

1.129

MAE

12% improvement

0.285

0.200

Endp.

30% improvement

EpicKitchen ↓ lower is better (×100)

0.445

0.322

MSE

28% improvement

2.721

2.130

MAE

22% improvement

0.791

0.556

Endp.

30% improvement