Conference item
Rapid adaptation in online continual learning: are we evaluating it right?
- Abstract:
- We revisit the common practice of evaluating adaptation of Online Continual Learning (OCL) algorithms through the metric of online accuracy, which measures the accuracy of the model on the immediate next few samples. However, we show that this metric is unreliable, as even vacuous blind classifiers, which do not use input images for prediction, can achieve unrealistically high online accuracy by exploiting spurious label correlations in the data stream. Our study reveals that existing OCL algorithms can also achieve high online accuracy, but perform poorly in retaining useful information, suggesting that they unintentionally learn spurious label correlations. To address this issue, we propose a novel metric for measuring adaptation based on the accuracy on the near-future samples, where spurious correlations are removed. We benchmark existing OCL approaches using our proposed metric on large-scale datasets under various computational budgets and find that better generalization can be achieved by retaining and reusing past seen information. We believe that our proposed metric can aid in the development of truly adaptive OCL methods. We provide code to reproduce our results at https://github.com/drimpossible/EvalOCL.
- Publication status:
- Published
- Peer review status:
- Peer reviewed
Actions
Access Document
- Files:
-
-
(Preview, Accepted manuscript, pdf, 4.7MB, Terms of use)
-
- Publisher copy:
- 10.1109/ICCV51070.2023.01728
Authors
- Publisher:
- IEEE
- Host title:
- 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
- Pages:
- 18806-18815
- Publication date:
- 2024-01-15
- Acceptance date:
- 2023-07-13
- Event title:
- International Conference on Computer Vision (ICCV 2023)
- Event location:
- Paris, France
- Event website:
- https://iccv2023.thecvf.com/
- Event start date:
- 2023-10-01
- Event end date:
- 2023-10-06
- DOI:
- EISSN:
-
2380-7504
- ISSN:
-
1550-5499
- EISBN:
- 9798350307184
- ISBN:
- 9798350307191
- Language:
-
English
- Keywords:
- Pubs id:
-
1700126
- Local pid:
-
pubs:1700126
- Deposit date:
-
2024-03-15
Terms of use
- Copyright holder:
- IEEE
- Copyright date:
- 2023
- Rights statement:
- ©2023 IEEE.
- Notes:
- This is the accepted manuscript version of the article. The final version is available online from IEEE at https://dx.doi.org/10.1109/ICCV51070.2023.01728
If you are the owner of this record, you can report an update to it here: Report update to this record