SD4Match: learning to prompt stable diffusion model for semantic matching

Conference item

In this paper, we address the challenge of matching semantically similar keypoints across image pairs. Existing research indicates that the intermediate output of the UNet within the Stable Diffusion (SD) can serve as robust image feature maps for such a matching task. We demonstrate that by employing a basic prompt tuning technique, the inherent potential of Stable Diffusion can be harnessed, resulting in a significant enhancement in accuracy over previous approaches. We further introduce a novel conditional prompting module that conditions the prompt on the local details of the input image pairs, leading to a further improvement in performance. We designate our approach as SD4Match, short for Stable Diffusion for Semantic Matching. Comprehensive evaluations of SD4Match on the PF-Pascal, PF-Willow, and SPair-71k datasets show that it sets new benchmarks in accuracy across all these datasets. Particularly, SD4Match outperforms the previous state-of-the-art by a margin of 12 percentage points on the challenging SPair-71k dataset. Code is available at the project website: https://sd4match.active.vision/.

Authors: Li, X; Lu, J; Han, K; Prisacariu, VA

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Host title: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• Pages: 27548-27558
• Publication date: 2024-09-16
• Acceptance date: 2024-02-26
• Event title: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024)
• Event location: Seattle, WA, USA
• Event website: https://cvpr.thecvf.com/Conferences/2024
• Event start date: 2024-06-16
• Event end date: 2024-06-22
• DOI: 10.1109/cvpr52733.2024.02602
• EISSN: 2575-7075
• ISSN: 1063-6919
• EISBN: 9798350353006
• ISBN: 9798350353013
• Language: English
• Keywords: computer vision; adaptation models; accuracy; codes; semantics; benchmark testing; diffusion models