PC2: projection-conditioned point cloud diffusion for single-image 3D reconstruction

Conference item

Reconstructing the 3D shape of an object from a single RGB image is a long-standing problem in computer vision. In this paper, we propose a novel method for single-image 3D reconstruction which generates a sparse point cloud via a conditional denoising diffusion process. Our method takes as input a single RGB image along with its camera pose and gradually denoises a set of 3D points, whose positions are initially sampled randomly from a three-dimensional Gaussian distribution, into the shape of an object. The key to our method is a geometrically-consistent conditioning process which we call projection conditioning: at each step in the diffusion process, we project local image features onto the partially-denoised point cloud from the given camera pose. This projection conditioning process enables us to generate high-resolution sparse geometries that are well-aligned with the input image and can additionally be used to predict point colors after shape reconstruction. Moreover, due to the probabilistic nature of the diffusion process, our method is naturally capable of generating multiple different shapes consistent with a single input image. In contrast to prior work, our approach not only performs well on synthetic benchmarks but also gives large qualitative improvements on complex real-world data. Data and code are available at https://lukemelas.github.io/projectionconditioned-point-cloud-diffusion/.

Authors: Melas-Kyriazi, L; Rupprecht, C; Vedaldi, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Host title: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• Journal: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• Pages: 12923-12932
• Publication date: 2023-08-22
• Acceptance date: 2023-02-27
• Event title: Conference on Computer Vision and Pattern Recognition (CVPR 2023)
• Event location: Vancouver, Canada
• Event website: https://cvpr2023.thecvf.com/Conferences/2023
• Event start date: 2023-06-18
• Event end date: 2023-06-22
• DOI: 10.1109/CVPR52729.2023.01242
• EISSN: 2575-7075
• ISSN: 1063-6919
• EISBN: 9798350301298
• ISBN: 9798350301304
• Language: English
• Keywords: ethics; point cloud compression; 3D from single images; computer vision; three-dimensional displays; solid modeling; geometry; shape