GeoNet++: Iterative geometric neural network with edge-aware refinement for joint depth and surface normal estimation

Journal article

In this paper, we propose a geometric neural network with edge-aware refinement (GeoNet++) to jointly predict both depth and surface normal maps from a single image. Building on top of two-stream CNNs, GeoNet++ captures the geometric relationships between depth and surface normals with the proposed depth-to-normal and normal-to-depth modules. In particular, the "depth-to-normal" module exploits the least square solution of estimating surface normals from depth to improve their quality, while the "normal-to-depth" module refines the depth map based on the constraints on surface normals through kernel regression. Boundary information is exploited via an edge-aware refinement module. GeoNet++ effectively predicts depth and surface normals with high 3D consistency and sharp boundaries resulting in better reconstructed 3D scenes. Note that GeoNet++ is generic and can be used in other depth/normal prediction frameworks to improve 3D reconstruction quality and pixel-wise accuracy of depth and surface normals. Furthermore, we propose a new 3D geometric metric (3DGM) for evaluating depth prediction in 3D. In contrast to current metrics that focus on evaluating pixel-wise error/accuracy, 3DGM measures whether the predicted depth can reconstruct high quality 3D surface normals. This is a more natural metric for many 3D application domains. Our experiments on NYUD-V2 [1] and KITTI [2] datasets verify that GeoNet++ produces fine boundary details and the predicted depth can be used to reconstruct high quality 3D surfaces.

Authors: Qi, X; Liu, Z; Liao, R; Torr, PHS; Urtasun, R

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Institute of Electrical and Electronics Engineers
• Journal: IEEE Transactions on Pattern Analysis and Machine Intelligence
• Volume: 44
• Issue: 2
• Pages: 969-984
• Publication date: 2020-09-01
• Acceptance date: 2020-07-11
• DOI: 10.1109/tpami.2020.3020800
• EISSN: 1939-3539
• ISSN: 0162-8828
• Pmid: 32870785
• Language: English
• Keywords: FFR