Mesh based semantic modelling for indoor and outdoor scenes

Conference item

Semantic reconstruction of a scene is important for a variety of applications such as 3D modelling, object recognition and autonomous robotic navigation. However, most object labelling methods work in the image domain and fail to capture the information present in 3D space. In this work we propose a principled way to generate object labelling in 3D. Our method builds a triangulated meshed representation of the scene from multiple depth estimates. We then define a CRF over this mesh, which is able to capture the consistency of geometric properties of the objects present in the scene. In this framework, we are able to generate object hypotheses by combining information from multiple sources: geometric properties (from the 3D mesh), and appearance properties (from images). We demonstrate the robustness of our framework in both indoor and outdoor scenes. For indoor scenes we created an augmented version of the NYU indoor scene dataset (RGBD images) with object labelled meshes for training and evaluation. For outdoor scenes, we created ground truth object labellings for the KITTY odometry dataset (stereo image sequence). We observe a significant speed-up in the inference stage by performing labelling on the mesh, and additionally achieve higher accuracies.

Authors: Valentin, JPC; Sengupta, S; Warrell, J; Shahrokni, A; Torr, PHS

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Host title: 2013 IEEE Conference on Computer Vision and Pattern Recognition
• Pages: 2067-2074
• Publication date: 2013-10-03
• Acceptance date: 2013-02-24
• Event title: 26th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2013)
• Event location: Portland, Oregon, USA
• Event website: https://pamitc.org/cvpr13.html?redirect=true
• Event start date: 2013-06-23
• Event end date: 2013-06-28
• DOI: 10.1109/cvpr.2013.269
• ISSN: 1063-6919
• EISBN: 9781538656723
• Language: English