Revisiting evaluation metrics for semantic segmentation: optimization and evaluation of fine-grained intersection over union

Conference item

Semantic segmentation datasets often exhibit two types of imbalance: class imbalance, where some classes appear more frequently than others and size imbalance, where some objects occupy more pixels than others. This causes traditional evaluation metrics to be biased towards majority classes (e.g. overall pixel-wise accuracy) and large objects (e.g. mean pixel-wise accuracy and per-dataset mean intersection over union). To address these shortcomings, we propose the use of fine-grained mIoUs along with corresponding worst-case metrics, thereby offering a more holistic evaluation of segmentation techniques. These fine-grained metrics offer less bias towards large objects, richer statistical information, and valuable insights into model and dataset auditing. Furthermore, we undertake an extensive benchmark study, where we train and evaluate 15 modern neural networks with the proposed metrics on 12 diverse natural and aerial segmentation datasets. Our benchmark study highlights the necessity of not basing evaluations on a single metric and confirms that fine-grained mIoUs reduce the bias towards large objects. Moreover, we identify the crucial role played by architecture designs and loss functions, which lead to best practices in optimizing fine-grained metrics. The code is available at https://github.com/zifuwanggg/JDTLosses.

Authors: Wang, Z; Berman, M; Rannen-Triki, A; Torr, PHS; Tuia, D

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Curran Associates
• Host title: Advances in Neural Information Processing Systems 36 (NeurIPS 2023)
• Volume: 36
• Pages: 60144-60225
• Series: Advances in Neural Information Processing Systems
• Publication date: 2024-07-01
• Acceptance date: 2023-10-27
• Event title: 37th Conference on Neural Information Processing Systems (NeurIPS 2023)
• Event location: New Orleans, Louisiana, USA
• Event website: https://neurips.cc/Conferences/2023
• Event start date: 2023-12-10
• Event end date: 2023-12-16
• ISSN: 1049-5258
• ISBN: 9781713899921
• Language: English