Using mixup as a regularizer can surprisingly improve accuracy and out-of-distribution robustness

Conference item

We show that the effectiveness of the well celebrated Mixup can be further improved if instead of using it as the sole learning objective, it is utilized as an additional regularizer to the standard cross-entropy loss. This simple change not only improves accuracy but also significantly improves the quality of the predictive uncertainty estimation of Mixup in most cases under various forms of covariate shifts and out-of-distribution detection experiments. In fact, we observe that Mixup otherwise yields much degraded performance on detecting out-of-distribution samples possibly, as we show empirically, due to its tendency to learn models exhibiting high-entropy throughout; making it difficult to differentiate in-distribution samples from out-of-distribution ones. To show the efficacy of our approach (RegMixup), we provide thorough analyses and experiments on vision datasets (ImageNet & CIFAR-10/100) and compare it with a suite of recent approaches for reliable uncertainty estimation.

Authors: Pinto, F; Yang, H; Lim, SN; Torr, PHS; Dokania, PK

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Curran Associates, Inc
• Host title: Advances in Neural Information Processing Systems
• Volume: 35
• Pages: 14608–14622
• Publication date: 2023-04-01
• Acceptance date: 2022-09-15
• Event title: 36th Conference on Neural Information Processing Systems (NeurIPS 2022)
• Event series: Advances in Neural Information Processing Systems
• Event location: New Orleans, Louisiana, USA
• Event website: https://nips.cc/Conferences/2022
• Event start date: 2022-11-28
• Event end date: 2022-12-09
• ISSN: 1049-5258
• Commissioning body: Neural Information Processing Systems Foundation, Inc. (NeurIPS)
• ISBN: 9781713871088
• Language: English
• Keywords: FFR