Conference item
Omnimatte: associating objects and their effects in video
- Abstract:
- Computer vision is increasingly effective at segmenting objects in images and videos; however, scene effects related to the objects—shadows, reflections, generated smoke, etc.—are typically overlooked. Identifying such scene effects and associating them with the objects producing them is important for improving our fundamental understanding of visual scenes, and can also assist a variety of applications such as removing, duplicating, or enhancing objects in video. In this work, we take a step towards solving this novel problem of automatically associating objects with their effects in video. Given an ordinary video and a rough segmentation mask over time of one or more subjects of interest, we estimate an omnimatte for each subject—an alpha matte and color image that includes the subject along with all its related time-varying scene elements. Our model is trained only on the input video in a self-supervised manner, without any manual labels, and is generic—it produces omnimattes automatically for arbitrary objects and a variety of effects. We show results on real-world videos containing interactions between different types of subjects (cars, animals, people) and complex effects, ranging from semitransparent elements such as smoke and reflections, to fully opaque effects such as objects attached to the subject.
- Publication status:
- Published
- Peer review status:
- Peer reviewed
Actions
Access Document
- Files:
-
-
(Preview, Accepted manuscript, pdf, 4.2MB, Terms of use)
-
- Publisher copy:
- 10.1109/CVPR46437.2021.00448
Authors
- Publisher:
- IEEE
- Host title:
- Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR)
- Pages:
- 4505-4513
- Publication date:
- 2021-11-13
- Acceptance date:
- 2021-02-28
- Event title:
- Conference on Computer Vision and Pattern Recognition (CVPR 2021)
- Event location:
- Online
- Event website:
- http://cvpr2021.thecvf.com/
- Event start date:
- 2021-06-19
- Event end date:
- 2021-06-25
- DOI:
- EISSN:
-
2575-7075
- ISSN:
-
1063-6919
- EISBN:
- 978-1-6654-4509-2
- ISBN:
- 978-1-6654-4510-8
- Language:
-
English
- Keywords:
- Pubs id:
-
1173939
- Local pid:
-
pubs:1173939
- Deposit date:
-
2021-04-28
- ARK identifier:
Terms of use
- Copyright holder:
- IEEE
- Copyright date:
- 2021
- Rights statement:
- © 2021 IEEE.
- Notes:
- This paper will be presented at the Conference on Computer Vision and Pattern Recognition (CVPR 2021), 19th-25th June 2021. This is the accepted manuscript version of the article. The final version is available online from IEEE at: https://doi.org/10.1109/CVPR46437.2021.00448
If you are the owner of this record, you can report an update to it here: Report update to this record