Depth Pro: Sharp Monocular Metric Depth in Less Than a Second: Monocular Depth Estimation Reaches New Heights: 2.25 Megapixel Sharp Depth Map in 0.3 Seconds
Submitted in October 2024. Apple team releases Depth Pro, a zero-shot monocular metric depth estimation foundation model. Core contribution: outputs absolute scale depth maps without camera intrinsics, generating high-resolution depth maps of 2.25 megapixels (approximately 1920x1200) in 0.3 seconds with sharp boundaries. The model is based on an efficient multi-scale vision transformer, trained on real and synthetic data, and achieves state-of-the-art single-image focal length estimation.
Depth Pro achieves significant breakthroughs in speed, resolution, accuracy, and generalization in monocular depth estimation. It outputs metric depth without camera intrinsics and has fast inference (0.3 seconds/2.25MP), making it suitable for real-time applications such as mobile devices, AR/VR, and autonomous driving. Compared to previous methods, Depth Pro shows qualitative improvements in boundary sharpness and high-frequency details, thanks to its multi-scale ViT architecture and innovative training strategy. The model is open-sourced and is expected to become a new benchmark in the field.
The model uses an efficient multi-scale vision transformer (ViT) as the backbone, achieving high-resolution output through multi-scale feature fusion. Training combines real datasets (e.g., NYUv2, KITTI) and synthetic datasets (e.g., Hypersim) to ensure both metric accuracy and boundary tracking. Key innovations include: 1) absolute scale prediction without camera intrinsics; 2) a dedicated boundary accuracy evaluation metric; 3) a single-image focal length estimation module. On multiple benchmarks, Depth Pro surpasses previous methods in metrics such as RMSE and δ1, with inference speed several times faster than similar models. Limitations: may still face challenges with extreme lighting and transparent objects.
This technology directly impacts industries such as AR/VR, robotics, autonomous driving, photography, and film production. For example, AR devices can achieve more realistic virtual-real fusion; robots can grasp objects more accurately; autonomous driving can improve obstacle detection. Apple's open-source strategy may accelerate industry adoption and promote the proliferation of mobile depth sensing applications. At the same time, it may reduce reliance on dedicated depth sensors (e.g., LiDAR), changing the hardware market landscape.
Recommendations for AR/VR and robotics companies: 1) immediately evaluate Depth Pro's performance in their own scenarios; 2) consider integrating it into products to replace or assist depth sensors; 3) watch for Apple's potential mobile-optimized version. For chip manufacturers, hardware acceleration optimization can be targeted for this model architecture.
Future focus: 1) deployment optimization on mobile and edge devices; 2) extension to dynamic scenes and video streams; 3) fusion with other sensors (e.g., IMU) to improve robustness; 4) whether Apple will integrate it into iOS or Vision Pro; 5) application innovations from the open-source community.