RealFusion: 360 reconstruction of any object from a single image

Conference item

We consider the problem of reconstructing a full 360° photographic model of an object from a single image of it. We do so by fitting a neural radiance field to the image, but find this problem to be severely ill-posed. We thus take an off-the-self conditional image generator based on diffusion and engineer a prompt that encourages it to “dream up” novel views of the object. Using the recent DreamFusion method, we fuse the given input view, the conditional prior, and other regularizers into a final, consistent reconstruction. We demonstrate state-of-the-art reconstruction results on benchmark images when compared to prior methods for monocular 3D reconstruction of objects. Qualitatively, our reconstructions provide a faithful match of the input view and a plausible extrapolation of its appearance and 3D shape, including to the side of the object not visible in the image.

Authors: Melas-Kyriazi, L; Laina, I; Rupprecht, C; Vedaldi, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Host title: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• Journal: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• Pages: 8446-8455
• Publication date: 2023-08-22
• Acceptance date: 2023-02-27
• Event title: Conference on Computer Vision and Pattern Recognition (CVPR 2023)
• Event location: Vancouver, Canada
• Event website: https://cvpr2023.thecvf.com/
• Event start date: 2023-06-18
• Event end date: 2023-06-22
• DOI: 10.1109/CVPR52729.2023.00816
• EISSN: 2575-7075
• ISSN: 1063-6919
• EISBN: 9798350301298
• ISBN: 9798350301304
• Language: English
• Keywords: FFR