Object-centric perception and description of video

Thesis

In this thesis, we explore video understanding from an object-centric perspective, treating objects as the fundamental units of computation. These objects persist over time, move through the scene, occlude and interact with one another, and, in edited media, often act out a story. We develop this perspective through methods for both low-level object perception and high-level object understanding.

The first direction of this thesis concerns the discovery of objects from motion, motivated by the Gestalt principle of common fate, which suggests that the perception of objectness can emerge from coherent motion. Following this intuition, we study moving object segmentation (MOS), the task of discovering, segmenting, and tracking multiple objects from motion cues. Our first work relies on optical flow as the primary motion signal, developing a Sim2Real MOS framework that transfers effectively to real-world videos and is further enhanced through appearance-based refinement. We then leverage large pretrained segmentation models and feed-forward 3D reconstruction backbones to achieve more accurate and geometrically grounded MOS, extending the task to temporally fine-grained, per-frame motion-state prediction.

The second direction focuses on understanding videos through the lens of characters, which are often the primary narrative objects in edited media. We first study the task of Audio Description (AD) generation for visually impaired audiences, leveraging advances in large multimodal models through training-free pipelines. Character information and film grammar are incorporated through structured prompting, enabling more accurate and narratively grounded descriptions. Finally, we address the problem of streaming character naming, developing a framework that infers character identities progressively from audiovisual cues and dialogue, providing a foundation for future character-centric video understanding.

Authors: Xie, J

• DOI: 10.5287/ora-y0d0mywgd
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: audio description generation; automatic character naming; edited media understanding; object-centric video understanding; moving object segmentation
• Subjects: Computer vision