Aligning source visual and target language domains for unpaired video captioning

Journal article

Training supervised video captioning model requires coupled video-caption pairs. However, for many targeted languages, sufficient paired data are not available. To this end, we introduce the unpaired video captioning task aiming to train models without coupled video-caption pairs in target language. To solve the task, a natural choice is to employ a two-step pipeline system: first utilizing video-to-pivot captioning model to generate captions in pivot language and then utilizing pivot-to-target translation model to translate the pivot captions to the target language. However, in such a pipeline system, 1) visual information cannot reach the translation model, generating visual irrelevant target captions; 2) the errors in the generated pivot captions will be propagated to the translation model, resulting in disfluent target captions. To address these problems, we propose the Unpaired Video Captioning with Visual Injection system (UVC-VI). UVC-VI first introduces the Visual Injection Module (VIM), which aligns source visual and target language domains to inject the source visual information into the target language domain. Meanwhile, VIM directly connects the encoder of the video-to-pivot model and the decoder of the pivot-to-target model, allowing end-to-end inference by completely skipping the generation of pivot captions. To enhance the cross-modality injection of the VIM, UVC-VI further introduces a pluggable video encoder, i.e., Multimodal Collaborative Encoder (MCE). The experiments show that UVC-VI outperforms pipeline systems and exceeds several supervised systems. Furthermore, equipping existing supervised systems with our MCE can achieve 4% and 7% relative margins on the CIDEr scores to current state-of-the-art models on the benchmark MSVD and MSR-VTT datasets, respectively.

Authors: Liu, F; Wu, X; You, C; Ge, S; Zou, Y

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Journal: IEEE Transactions on Pattern Analysis and Machine Intelligence
• Volume: 44
• Issue: 12
• Pages: 9255-9268
• Publication date: 2022-12-02
• Acceptance date: 2021-11-16
• DOI: 10.1109/tpami.2021.3132229
• EISSN: 1939-3539
• ISSN: 0162-8828
• Pmid: 34855588
• Language: English
• Keywords: taining; adversarial training; pipelines; data models; task analysis; pseudo supervised training; visualization; feature extraction; video captioning; decoding; unpaired video captioning; pipeline system; FFR