No representation rules them all in category discovery

Conference item

In this paper we tackle the problem of Generalized Category Discovery (GCD). Specifically, given a dataset with labelled and unlabelled images, the task is to cluster all images in the unlabelled subset, whether or not they belong to the labelled categories. Our first contribution is to recognise that most existing GCD benchmarks only contain labels for a single clustering of the data, making it difficult to ascertain whether models are leveraging the available labels to solve the GCD task, or simply solving an unsupervised clustering problem. As such, we present a synthetic dataset, named 'Clevr-4', for category discovery. Clevr-4 contains four equally valid partitions of the data, i.e based on object 'shape', 'texture' or 'color' or 'count'. To solve the task, models are required to extrapolate the taxonomy specified by labelled set, rather than simply latch onto a single natural grouping of the data. We use this dataset to demonstrate the limitations of unsupervised clustering in the GCD setting, showing that even very strong unsupervised models fail on Clevr-4. We further use Clevr-4 to examine the weaknesses of existing GCD algorithms, and propose a new method which addresses these shortcomings, leveraging consistent findings from the representation learning literature to do so. Our simple solution, which is based on `Mean Teachers' and termed μGCD, substantially outperforms implemented baselines on Clevr-4. Finally, when we transfer these findings to real data on the challenging Semantic Shift Benchmark suite, we find that μGCD outperforms all prior work, setting a new state-of-the-art.

Authors: Vaze, S; Vedaldi, A; Zisserman, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Neural Information Processing Systems Foundation
• Host title: Advances in Neural Information Processing Systems 36 (NeurIPS 2023)
• Publication date: 2024-02-27
• Acceptance date: 2023-09-21
• Event title: 37th Conference on Neural Information Processing Systems (NeurIPS 2023)
• Event location: New Orleans, USA
• Event website: https://nips.cc/Conferences/2023
• Event start date: 2023-12-10
• Event end date: 2023-12-16
• Language: English