Video Generative Models as Geometry Learner
GeoNeXt repurposes pretrained video generative models for geometry estimation, formulated as a next-frames prediction task. It jointly models images and geometry targets (depth and surface normals) and is evaluated for zero-shot monocular depth and surface normal estimation across diverse datasets, outperforming previous task-specific approaches.
The paper introduces GeoNeXt, a method that adapts pretrained video generative models for geometry estimation. Unlike prior approaches that either train task-specific models independently or fine-tune image diffusion backbones with substantial labeled data, GeoNeXt treats geometry estimation as a next-frames prediction task. This formulation leverages structured knowledge and richer priors from video models, enabling joint modeling of images and geometry targets. Experiments demonstrate improved zero-shot monocular depth and surface normal estimation across diverse datasets compared to previous task-specific methods.
GeoNeXt reformulates geometry estimation as next-frame prediction in a pretrained video generative model, allowing joint learning of depth and surface normals with image conditioning. This approach exploits temporal and structural priors from video models, reducing the need for large labeled geometry datasets and improving zero-shot generalization.
Using video generative models for geometry estimation could lower data requirements and improve robustness in 3D perception tasks. This may accelerate adoption in applications like autonomous driving, robotics, and augmented reality, where accurate depth and surface normals are critical but labeled data is expensive.
GeoNeXt offers a data-efficient way to obtain depth and surface normal estimates from monocular video, potentially lowering costs for 3D scene understanding. This could benefit companies building autonomous systems, AR/VR, and robotics by reducing reliance on expensive depth sensors or large labeled datasets.
Next signals to watch include whether GeoNeXt is extended to multi-view geometry, dynamic scenes, or other geometric targets like optical flow. Also monitor if the approach is adopted by industry labs or integrated into larger video generation frameworks, and whether it reduces the need for specialized depth sensors.