content type paperpublished May 2025
Matrix3D: Large Photogrammetry Model All-in-One
AuthorsYuanxun Lu†, Jingyang Zhang, Tian Fang, Jean–Daniel Nahmias, Yanghai Tsin, Long Quan‡, Xun Cao†, Yao Yao†, Shiwei Li
We present Matrix3D, a unified model that performs several photogrammetry subtasks, including pose estimation, depth prediction, and novel view synthesis using just the same model. Matrix3D utilizes a multi-modal diffusion transformer (DiT) to integrate transformations across several modalities, such as images, camera parameters, and depth maps. The key to Matrix3D’s large-scale multi-modal training lies in the incorporation of a mask learning strategy. This enables full-modality model training even with partially complete data, such as bi-modality data of image-pose and image-depth pairs, thus significantly increases the pool of available training data. Matrix3D demonstrates state-of-the-art performance in pose estimation and novel view synthesis tasks. Additionally, it offers fine-grained control through multi-round interactions, making it an innovative tool for 3D content creation.
† Nanjing University
‡ Hong Kong University of Science and Technology (HKUST)
Related readings and updates.
4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities
October 18, 2024research area Computer Visionconference NeurIPS
*Equal Contributors
Current multimodal and multitask foundation models like 4M or UnifiedIO show promising results, but in practice their out-of-the-box abilities to accept diverse inputs and perform diverse tasks are limited by the (usually rather small) number of modalities and tasks they are trained on. In this paper, we significantly expand upon the capabilities of 4M by training it on tens of highly diverse modalities and by performing…
Read moreFast and Explicit Neural View Synthesis
February 2, 2022research area Computer Vision, research area Methods and Algorithmsconference WACV
We study the problem of novel view synthesis from sparse source observations of a scene comprised of 3D objects. We propose a simple yet effective approach that is neither continuous nor implicit, challenging recent trends on view synthesis. Our approach explicitly encodes observations into a volumetric representation that enables amortized rendering. We demonstrate that although continuous radiance field representations have gained a lot of…
Read more
Discover opportunities in Machine Learning.
Our research in machine learning breaks new ground every day.