Robust odometry and localisation in challenging environments with vision and lidar

Thesis

Odometry and localisation are two crucial modules for autonomous robot navigation. Numerous studies have demonstrated various systems operating in both indoor and outdoor environments. Despite the progress in accuracy and robustness, there remains significant potential to further enhance odometry and localisation systems, especially in challenging environments.

This thesis proposes several methods to enhance odometry and localisation systems utilising camera and lidar sensors, both individually and in combination. The aim is to address common failure modes and explore novel approaches for improvement.

The first contribution of this thesis is a multi-camera visual odometry system that integrates multiple cameras and an IMU within a factor graph framework. By tracking features across cameras and selecting a subset of features, the system achieves enhanced accuracy and robustness, able to operate in challenging environments such as underground mines and narrow spaces. The second contribution is a global lidar localisation system that leverages semantics and object instances. This work was the first to use panoptic segmentation directly on 3D lidar scans for indoor localisation. The system constructs a map with object instances and semantic classes, and estimates the lidar sensor poses online. The third contribution is a cross-modal global visual localisation system capable of operating with input maps from various 3D map representations. By rendering synthetic images from 3D maps, this system can localize monocular images and estimate 6 DoF poses. The fourth contribution is two datasets with multi-sensor modalities, aimed at promoting operations in challenging environments. The Hilti- Oxford datasets introduce a new method for obtaining millimetre-accurate ground truth poses and serve as the foundation for a major international competition.

This thesis particularly focuses on developing real-time operating systems, with each proposed algorithm capable of running on a mobile laptop. These algorithms have been extensively evaluated using real-world data and have demonstrated effective performance with both handheld devices and legged robot platforms.

Authors: Zhang, L

• DOI: 10.5287/ora-zv6ve15vp
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: vision and lidar; odometry; localisation
• Subjects: Robotics; Computer vision