View publication

We consider the problem of online and real-time registration of partial point clouds obtained from an unseen real-world rigid object without knowing its 3D model. The point cloud is partial as it is obtained by a depth sensor capturing only the visible part of the object from a certain viewpoint. It introduces two main challenges: 1) two partial point clouds do not fully overlap and 2) keypoints tend to be less reliable when the visible part of the object does not have salient local structures. To address these issues, we propose DeepPRO, a keypoint-free and an end-to-end trainable deep neural network. Its core idea is inspired by how humans align two point clouds: we can imagine how two point clouds will look like after the registration based on their shape. To realize the idea, DeepPRO has inputs of two partial point clouds and directly predicts the point-wise location of the aligned point cloud. By preserving the ordering of points during the prediction, we enjoy dense correspondences between input and predicted point clouds when inferring rigid transform parameters. We conduct extensive experiments on the real-world Linemod and synthetic ModelNet40 datasets. In addition, we collect and evaluate on the PRO1k dataset, a large-scale version of Linemod meant to test generalization to real-world scans. Results show that DeepPRO achieves the best accuracy against thirteen strong baseline methods, e.g., 2.2mm ADD on the Linemod dataset, while running 50 fps on mobile devices.

Related readings and updates.

Pointersect: Neural Rendering with Cloud-Ray Intersection

Pointersect is a plug-and-play rendering algorithm for unseen point clouds without any per-scene optimization. It takes a point cloud (position and optionally color) as input and directly renders images from novel views. The same model also estimates surface normal and depth; it is differentiable and allows scene editing without any retraining. All results in the webpage are rendered by the same model, trained only on 48 meshes (shown at the…
See paper details

ICCV 2021

Apple sponsored the International Conference on Computer Vision (ICCV), which was held virtually from October 11 to 17.

See event details