Journal article
Minimum sample size for external validation of a clinical prediction model with a binary outcome
- Abstract:
- In prediction model research, external validation is needed to examine an existing model's performance using data independent to that for model development. Current external validation studies often suffer from small sample sizes and consequently imprecise predictive performance estimates. To address this, we propose how to determine the minimum sample size needed for a new external validation study of a prediction model for a binary outcome. Our calculations aim to precisely estimate calibration (Observed/Expected and calibration slope), discrimination (C-statistic), and clinical utility (net benefit). For each measure, we propose closed-form and iterative solutions for calculating the minimum sample size required. These require specifying: (i) target SEs (confidence interval widths) for each estimate of interest, (ii) the anticipated outcome event proportion in the validation population, (iii) the prediction model's anticipated (mis)calibration and variance of linear predictor values in the validation population, and (iv) potential risk thresholds for clinical decision-making. The calculations can also be used to inform whether the sample size of an existing (already collected) dataset is adequate for external validation. We illustrate our proposal for external validation of a prediction model for mechanical heart valve failure with an expected outcome event proportion of 0.018. Calculations suggest at least 9835 participants (177 events) are required to precisely estimate the calibration and discrimination measures, with this number driven by the calibration slope criterion, which we anticipate will often be the case. Also, 6443 participants (116 events) are required to precisely estimate net benefit at a risk threshold of 8%. Software code is provided.
- Publication status:
- Published
- Peer review status:
- Peer reviewed
Actions
Access Document
- Files:
-
-
(Preview, Version of record, pdf, 2.1MB, Terms of use)
-
- Publisher copy:
- 10.1002/sim.9025
Authors
- Publisher:
- Wiley
- Journal:
- Statistics in Medicine More from this journal
- Volume:
- 40
- Issue:
- 19
- Pages:
- 4230-4251
- Place of publication:
- England
- Publication date:
- 2021-05-24
- Acceptance date:
- 2021-03-22
- DOI:
- EISSN:
-
1097-0258
- ISSN:
-
0277-6715
- Pmid:
-
34031906
- Language:
-
English
- Keywords:
- Pubs id:
-
1179487
- Local pid:
-
pubs:1179487
- Deposit date:
-
2022-08-14
Terms of use
- Copyright holder:
- Riley et al.
- Copyright date:
- 2021
- Rights statement:
- ©2021 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.
- Licence:
- CC Attribution (CC BY)
If you are the owner of this record, you can report an update to it here: Report update to this record