Moving object segmentation: all you need is SAM (and flow)

Conference item

The objective of this paper is motion segmentation – discovering and segmenting the moving objects in a video. This is a much studied area with numerous careful, and sometimes complex, approaches and training schemes including: self-supervised learning, learning from synthetic datasets, object-centric representations, amodal representations, and many more. Our interest in this paper is to determine if the Segment Anything model (SAM) can contribute to this task. We investigate two models for combining SAM with optical flow that harness the segmentation power of SAM with the ability of flow to discover and group moving objects. In the first model, we adapt SAM to take optical flow, rather than RGB, as an input. In the second, SAM takes RGB as an input, and flow is used as a segmentation prompt. These surprisingly simple methods, without any further modifications, outperform all previous approaches by a considerable margin in both single and multiobject benchmarks. We also extend these frame-level segmentations to sequence-level segmentations that maintain object identity. Again, this simple model achieves outstanding performance across multiple moving object segmentation benchmarks.

Authors: Xie, J; Yang, C; Xie, W; Zisserman, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Springer
• Host title: Computer Vision – ACCV 2024
• Series: Lecture Notes in Computer Science
• Series number: 15481
• Publication date: 2024-12-10
• Acceptance date: 2024-09-20
• Event title: 27th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2024)
• Event location: Marrakesh, Morocco
• Event website: https://conferences.miccai.org/2024/en/
• Event start date: 2024-10-06
• Event end date: 2024-10-10
• DOI: 10.1007/978-981-96-0972-7_17
• EISSN: 1611-3349
• ISSN: 0302-9743
• EISBN: 9789819609727
• ISBN: 9789819609710
• Language: English
• Keywords: video object segmentation; moving object discovery