Ray-based visual odometry and 3D scene understanding

Thesis

Estimating camera motion and recovering scene geometry from monocular imagery are fundamental problems in computer vision and robotics. Learningbased approaches to visual odometry and depth estimation have shown promising performance, yet they often suffer from limitations in estimation accuracy, crossdomain generalization, and training stability. We hypothesize that a key factor underlying these challenges lies in the choice of representation. We support this hypothesis with an analysis showing that dense ray-based representations, instead of the widely used minimal 6-DoF, provide perturbation robustness, Lipschitz continuity, and information preservation—properties that directly target these three challenges.

This thesis progressively explores ray-based representations as an alternative formulation, starting from camera motion modeling and extending to full scene understanding. First, we revisit visual odometry from the perspective of camera representation and introduce a ray-bundle formulation for supervised learning, where motion is modeled as a spatially distributed set of rays that better aligns with spatial patch-level features. Extensive experiments demonstrate that this representation improves pose estimation accuracy, enhances generalization, and achieves more stable optimization.

Building on this, we extend the ray-based formulation to self-supervised visual odometry, where learning relies on implicit geometric constraints through a coupled pose–depth framework. By replacing conventional 6-DoF regression with ray-based representations in the pose branch, the proposed approach reduces trajectory drift, improves training dynamics, and enhances robustness across domains. As pose and depth are jointly optimized, the improved pose estimation also leads to more accurate monocular depth prediction.

Finally, we generalize the formulation from camera motion to the joint representation of motion and scene geometry. We introduce a length-aware raybased representation that unifies pose and depth into a single spatially structured form, and propose a ray-space self-supervised optimization strategy that operates directly in this domain, enabling joint optimization of motion and geometry within a unified space. This approach avoids the information loss associated with converting dense representations into lower-dimensional pose–depth forms, leading to improved accuracy and robustness across diverse environments.

Experiments on multiple benchmark datasets demonstrate that ray-based representations consistently improve motion and depth accuracy, generalization under domain shift, and training stability. These results suggest that modeling visual geometry in a spatially distributed ray-based form provides a principled and effective foundation for learning-based visual odometry and 3D scene understanding.

Authors: Xu, F

• DOI: 10.5287/ora-gaxeeg6mj
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English