Language-EXtended Indoor SLAM (LEXIS): a versatile system for real-time visual scene understanding

Conference item

Versatile and adaptive semantic understanding would enable autonomous systems to comprehend and interact with their surroundings. Existing fixed-class models limit the adaptability of indoor mobile and assistive autonomous systems. In this work, we introduce LEXIS, a real-time indoor Simultaneous Localization and Mapping (SLAM) system that harnesses the open-vocabulary nature of Large Language Models (LLMs) to create a unified approach to scene understanding and place recognition. The approach first builds a topological SLAM graph of the environment (using visual-inertial odometry) and embeds Contrastive Language-Image Pretraining (CLIP) features in the graph nodes. We use this representation for flexible room classification and segmentation, serving as a basis for room-centric place recognition. This allows loop closure searches to be directed towards semantically relevant places. Our proposed system is evaluated using both public, simulated data and real-world data, covering office and home environments. It successfully categorizes rooms with varying layouts and dimensions and outperforms the state-of-the-art (SOTA). For place recognition and trajectory estimation tasks we achieve equivalent performance to the SOTA, all also utilizing the same pre-trained model. Lastly, we demonstrate the system’s potential for planning. Video at: https://youtu.be/gRqF3euDfX8

Authors: Kassab, C; Mattamala, M; Zhang, L; Fallon, M

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Host title: 2024 IEEE International Conference on Robotics and Automation (ICRA)
• Pages: 15988-15994
• Publication date: 2024-08-08
• Acceptance date: 2024-01-10
• Event title: 2024 IEEE International Conference on Robotics and Automation (ICRA 2024)
• Event location: Yokohama, Japan
• Event website: https://2024.ieee-icra.org/
• Event start date: 2024-05-13
• Event end date: 2024-05-17
• DOI: 10.1109/ICRA57147.2024.10610341
• EISBN: 9798350384574
• ISBN: 9798350384581
• Language: English
• Keywords: autonomous systems; simultaneous localization and mapping; estimation; uncertainty; semantics; visualization; adaptation models