Learning human pose from unaligned data through image translation

Conference item

We introduce a method for learning 2D landmark detectors for human pose from unlabelled video frames and unpaired human keypoints. We formulate this as an imageto-image translation task, similar to CycleGAN, between appearance (RGB) images and skeleton images representing pose. We show CycleGAN confounds appearance and pose information resulting in sub-optimal landmark detection performance. To address this, we introduce a tight keypoint bottleneck which prevents leaking appearance information through skeleton pose images. To facilitate image reconstruction from the clean skeleton images so obtained, we condition the generator on an appearance image sampled from another frame in the video. We further show that the second cycle-consistency constraint in CycleGAN is detrimental to landmark detection performance; we discard it, simplifying the model significantly. Our model with the above components outperforms state-of-the-art unsupervised landmark detection methods on multiple 2D human pose benchmarks.

Authors: Jakab, T; Vedaldi, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publication date: 2020-01-01
• Acceptance date: 2019-03-02
• Event title: Learning from Unlabeled Videos (CVPR 2019 Workshop)
• Event location: Long Beach, California, USA
• Event website: https://sites.google.com/view/luv2019/home
• Event start date: 2019-06-16
• Event end date: 2019-06-16
• Language: English