Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers

Conference item

Most recent semantic segmentation methods adopt a fully-convolutional network (FCN) with an encoderdecoder architecture. The encoder progressively reduces the spatial resolution and learns more abstract/semantic visual concepts with larger receptive fields. Since context modeling is critical for segmentation, the latest efforts have been focused on increasing the receptive field, through either dilated/atrous convolutions or inserting attention modules. However, the encoder-decoder based FCN architecture remains unchanged. In this paper, we aim to provide an alternative perspective by treating semantic segmentation as a sequence-to-sequence prediction task. Specifically, we deploy a pure transformer (i.e., without convolution and resolution reduction) to encode an image as a sequence of patches. With the global context modeled in every layer of the transformer, this encoder can be combined with a simple decoder to provide a powerful segmentation model, termed SEgmentation TRansformer (SETR). Extensive experiments show that SETR achieves new state of the art on ADE20K (50.28% mIoU), Pascal Context (55.83% mIoU) and competitive results on Cityscapes. Particularly, we achieve the first position in the highly competitive ADE20K test server leaderboard on the day of submission.

Authors: Zheng, S; Lu, J; Zhao, H; Zhu, X; Luo, Z

• Publication status: Published
• Peer review status: Reviewed (other)
• Publisher: IEEE
• Journal: Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• Pages: 6877-6886
• Publication date: 2021-11-13
• Acceptance date: 2021-06-01
• Event title: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• Event start date: 2021-06-20
• Event end date: 2021-06-25
• DOI: 10.1109/cvpr46437.2021.00681
• EISSN: 2575-7075
• ISSN: 1063-6919
• EISBN: 978-1-6654-4509-2
• ISBN: 978-1-6654-4510-8
• Language: English
• Keywords: FFR