How to derive skill from the fractions skill score

Journal article

The fractions skill score (FSS) is a widely used metric for assessing forecast skill, with applications ranging from precipitation to volcanic ash forecasts. By evaluating the fraction of grid squares exceeding a threshold in a neighborhood, the intuition is that it can avoid the pitfalls of pixelwise comparisons and identify length scales at which a forecast has skill. The FSS is typically interpreted relative to a “useful” criterion, where a forecast is considered skillful if its score exceeds a simple reference score. However, the typical reference score used is problematic, since it is not derived in a way that provides obvious meaning, does not scale with neighborhood size, and may not be exceeded by forecasts that have skill. We, therefore, provide a new method to determine forecast skill from the FSS, by deriving an expression for the FSS achieved by a random forecast, which provides a more robust and meaningful reference score to compare with. Through illustrative examples, we show that this new method considerably changes the length scales at which a forecast would be regarded as skillful and reveals subtleties in how the FSS should be interpreted.

Authors: Antonio, B; Aitchison, L

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: American Meteorological Society
• Journal: Monthly Weather Review
• Volume: 153
• Issue: 6
• Pages: 1021-1033
• Publication date: 2025-06-01
• Acceptance date: 2025-03-14
• DOI: 10.1175/mwr-d-24-0120.1
• EISSN: 1520-0493
• ISSN: 0027-0644
• Language: English
• Keywords: forecast verification/skill; diagnostics; model errors; model evaluation/performance; numerical weather prediction/forecasting; error analysis