LAVT: Language-Aware Vision Transformer for referring image segmentation

Conference item

Referring image segmentation is a fundamental vision-language task that aims to segment out an object referred to by a natural language expression from an image. One of the key challenges behind this task is leveraging the referring expression for highlighting relevant positions in the image. A paradigm for tackling this problem is to leverage a powerful vision-language (“cross-madal”) decoder to fuse features independently extracted from a vision encoder and a language encoder. Recent methods have made remarkable advancements in this paradigm by exploiting Transformers as cross-modal decoders, concurrent to the Transformer's overwhelming success in many other vision-language tasks. Adopting a different approach in this work, we show that significantly better cross-modal alignments can be achieved through the early fusion of linguistic and visual features in intermediate layers of a vision Transformer encoder network. By conducting cross-modal feature fusion in the visual feature encoding stage, we can leverage the well-proven correlation modeling power of a Transformer encoder for excavating helpful multi-modal context. This way, accurate segmentation results are readily harvested with a light-weight mask predictor. Without bells and whistles, our method surpasses the previous state-of-the-art methods on Ref CoCo, RefCOCO+, and G-Ref by large margins.

Authors: Yang, Z; Wang, J; Tang, Y; Chen, K; Zhao, H

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Host title: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• Pages: 18134-18144
• Publication date: 2022-09-27
• Acceptance date: 2022-06-19
• Event title: Conference on Computer Vision and Pattern Recognition (CVPR 2022)
• Event location: New Orleans, Louisiana
• Event website: https://cvpr2022.thecvf.com/
• Event start date: 2022-06-19
• Event end date: 2022-06-24
• DOI: 10.1109/CVPR52688.2022.01762
• EISSN: 2575-7075
• ISSN: 1063-6919
• EISBN: 9781665469463
• ISBN: 9781665469470
• Language: English
• Keywords: FFR; grouping and shape analysis; segmentation; vision + language