Personalizing retrieval using joint embeddings; or "The Return of Fluffy"

Conference item

The goal of this paper is to be able to retrieve images using a compound query that combines object instance information from an image, with a natural text description of what that object is doing or where it is. For example, to retrieve an image of ‘Fluffy the unicorn (specified by an image) on someone’s head’. To achieve this we design a mapping network that can ‘translate’ from a local image embedding (of the object instance) to a text token, such that the combination of the token and a natural language query is suitable for CLIP style text encoding, and image retrieval. Generating a text token in this manner involves a simple training procedure, that only needs to be performed once for each object instance. We show that our approach of using a trainable mapping network, termed π-map, together with frozen CLIP text and image encoders, improves the state of the art on two benchmarks designed to assess personalized retrieval.

Authors: Korbar, B; Zisserman, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: IEEE
• Host title: 2025 International Conference on Content-Based Multimedia Indexing (CBMI)
• Pages: 1-8
• Publication date: 2026-01-20
• Acceptance date: 2025-09-15
• Event title: 22nd International Conference on Content-based Multimedia Indexing (CBMI 2025)
• Event location: Dublin, Ireland
• Event website: https://www.robots.ox.ac.uk/~vgg/publications/2025/Korbar25/korbar25.pdf
• Event start date: 2025-10-22
• Event end date: 2025-10-24
• DOI: 10.1109/CBMI66578.2025.11339319
• EISBN: 9798331555009
• ISBN: 9798331555016
• Language: English
• Keywords: video search; personalization; retrieval