MVSplat360: feed-forward 360 scene synthesis from sparse views

Conference item

We introduce MVSplat360, a feed-forward approach for 360° novel view synthesis (NVS) of diverse real-world scenes, using only sparse observations. This setting is inherently ill-posed due to minimal overlap among input views and insufficient visual information provided, making it challenging for conventional methods to achieve high-quality results. Our MVSplat360 addresses this by effectively combining geometry-aware 3D reconstruction with temporally consistent video generation. Specifically, it refactors a feed-forward 3D Gaussian Splatting (3DGS) model to render features directly into the latent space of a pre-trained Stable Video Diffusion (SVD) model, where these features then act as pose and visual cues to guide the denoising process and produce photorealistic 3D-consistent views. Our model is end-to-end trainable and supports rendering arbitrary views with as few as 5 sparse input views. To evaluate MVSplat360's performance, we introduce a new benchmark using the challenging DL3DV-10K dataset, where MVSplat360 achieves superior visual quality compared to state-of-the-art methods on wide-sweeping or even 360° NVS tasks. Experiments on the existing benchmark RealEstate10K also confirm the effectiveness of our model. Readers are highly recommended to view the video results at donydchen.github.io/mvsplat360.

Authors: Chen, Y; Zheng, C; Xu, H; Zhuang, B; Vedaldi, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Curran Associates
• Host title: Advances in Neural Information Processing Systems 37: 38th Conference on Neural Information Processing Systems (NeurIPS 2024)
• Pages: 107064-107086
• Series: Advances in Neural Information Processing Systems
• Series number: 37
• Publication date: 2024-12-16
• Acceptance date: 2024-09-25
• Event title: 38th Conference on Neural Information Processing Systems (NeurIPS 2024)
• Event location: Vancouver Convention Centre, Vancouver, BC, Canada
• Event website: https://neurips.cc/virtual/2024/index.html
• Event start date: 2024-12-09
• Event end date: 2024-12-15
• ISSN: 1049-5258
• ISBN: 9798331314385
• Language: English
• Keywords: novel view synthesis; feed-forward 3DGS; 3D gaussians splatting; latent video diffusion model