PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control
PhysStream is an autoregressive model for physics-grounded image-to-video synthesis that incorporates structured scene memory (positional maps and object tracking maps derived online from previously generated frames) and supports fine-grained motion control via sparse velocity-increment signals. It is trained in two stages: a bidirectional model finetuned with motion-control conditioning, then a causal autoregressive model trained with additional structured scene memory. PhysStream enables interactive, mid-generation control over multi-object tabletop rigid-body scenes, reducing motion distribution distance (FVMD) by 33%.
PhysStream introduces a streaming approach to physics-grounded video generation, allowing interactive control during generation rather than requiring a full control schedule upfront. It uses structured scene memory and sparse velocity-increment signals to encode physical dynamics, improving physical consistency. The model is trained in two stages and demonstrates a 33% reduction in FVMD on multi-object tabletop rigid-body scenes.
The two-stage training strategy—first finetuning a bidirectional model with motion-control conditioning, then training a causal autoregressive model with structured scene memory—suggests that explicit memory of object positions and tracking improves physical consistency in autoregressive generation. The use of sparse velocity-increment signals rather than pixel-space positions indicates a shift toward learning underlying dynamics rather than direct spatial control.
This research addresses a gap in controllable video generation: enabling mid-generation, fine-grained physical control. Such capability could be valuable for simulation, robotics, and interactive media, where users need to adjust physical parameters on the fly. The focus on tabletop rigid-body scenes suggests near-term applications in industrial or scientific visualization.
If the approach generalizes, it could enable new products for interactive simulation, game development, or robotics training where users need real-time physical control over generated video. The reduction in motion distribution distance suggests improved realism, which may lower the cost of generating physically plausible synthetic data.
Next observable signals include whether the model scales to more complex scenes beyond tabletop rigid bodies, whether the structured scene memory approach is adopted by other video generation models, and whether the interactive control capability leads to commercial tools or APIs. Further validation on standard benchmarks and comparison with other controllable methods would strengthen the claim.