Invisible stitch: generating smooth 3D scenes with depth inpainting

Conference item

3D scene generation has quickly become a challenging new research direction, fueled by consistent improvements of 2D generative diffusion models. Current methods generate scenes by iteratively stitching newly generated images with existing geometry, using pre-trained monocular depth estimators to lift the generated images to 3D. The predicted depth is fused with the existing scene representation through various alignment operations. In this work, we make two fundamental contributions to the field of 3D scene generation. First, we note that lifting images to 3D with a monocular depth estimation model is suboptimal as it ignores the geometry of the existing scene, thus prompting the need for alignment. We introduce a depth completion model to directly learn the 3D fusion process, resulting in improved geometric coherence of generated scenes. Second, we introduce a new benchmark to evaluate the geometric accuracy of scene generation methods. We show that the commonly used CLIP score between scene prompts and images is unsuitable for measuring the geometric quality of a scene and introduce a depth-based metric. Our benchmark thus offers an additional dimension to gauge the quality of generated scenes.

Authors: Engstler, P; Vedaldi, A; Laina, I; Rupprecht, C

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Host title: 2025 International Conference on 3D Vision (3DV)
• Pages: 457-468
• Publication date: 2025-03-25
• Acceptance date: 2024-11-06
• Event title: International Conference on 3D Vision 2025 (3DV2025)
• Event location: Singapore
• Event website: https://3dvconf.github.io/2025/
• Event start date: 2025-03-25
• Event end date: 2025-03-28
• DOI: 10.1109/3DV66043.2025.00047
• EISSN: 2475-7888
• ISSN: 2378-3826
• EISBN: 9798331538514
• ISBN: 9798331538521
• Language: English
• Keywords: geometry; solid modeling; ethics; visualization; extrapolation; three-dimensional displays; depth measurement; pipelines; benchmark testing; diffusion models