Marionette: Predicting World States, Rendering Geometry, Painting Appearance
Marionette is a world model for interactive games with articulated characters. It uses a two-stage autoregressive dynamics model to predict an explicit 276-dimensional 3D world state, a zero-parameter graphics bridge to convert predicted state into pose-control videos, and a control-conditioned video-diffusion observation model to synthesize photorealistic RGB observations.
Marionette explicitly models evolving world state, delegates geometric computation to a fixed zero-parameter renderer, and uses a neural model for appearance synthesis. The system comprises three components: a two-stage autoregressive dynamics model predicting a 276-dimensional 3D world state (multi-entity articulated skeletons, metric root trajectories, rotations), a zero-parameter graphics bridge converting state to pose-control videos with closed-form world-space geometry and occlusion, and a control-conditioned video-diffusion model synthesizing photorealistic RGB observations. Experiments establish two properties of Marionette, though the specific properties are not detailed in the provided evidence.
The architecture separates world-state prediction from rendering and appearance synthesis, reducing error accumulation over long horizons by using a fixed zero-parameter renderer for geometry and occlusion. The explicit 276-dimensional state includes articulated skeletons and metric trajectories, enabling interpretability and controllability. The use of a control-conditioned video-diffusion model for appearance suggests a hybrid approach combining deterministic geometry with generative texture synthesis.
This approach could improve consistency and controllability in interactive game world models, potentially enabling more reliable long-horizon simulation and game AI. It may influence research directions in world models for embodied AI and simulation, shifting focus from end-to-end pixel prediction to structured state prediction with modular rendering.
Potential applications include game development tools, interactive simulation, and virtual environment generation. The modular design may reduce computational cost and improve quality, but no commercial deployment or business metrics are provided in the evidence.
Next observable signals include publication of detailed experimental results, code or model release, and follow-up work applying similar state-render-appearance decomposition to other domains such as robotics or autonomous driving. Adoption by game development or simulation platforms would indicate practical impact.