Journal article
Examining the effect of evaluation sample size on the sensitivity and specificity of COVID-19 diagnostic tests in practice: a simulation study
- Abstract:
-
Background: In response to the global COVID-19 pandemic, many in vitro diagnostic (IVD) tests for SARS-CoV-2 have been developed. Given the urgent clinical demand, researchers must balance the desire for precise estimates of sensitivity and specificity against the need for rapid implementation. To complement estimates of precision used for sample size calculations, we aimed to estimate the probability that an IVD will fail to perform to expected standards after implementation, following clinical studies with varying sample sizes.
Methods: We assumed that clinical validation study estimates met the ‘desirable’ performance (sensitivity 97%, specificity 99%) in the target product profile (TPP) published by the Medicines and Healthcare products Regulatory Agency (MHRA). To estimate the real-world impact of imprecision imposed by sample size we used Bayesian posterior calculations along with Monte Carlo simulations with 10,000 independent iterations of 5,000 participants. We varied the prevalence between 1 and 15% and the sample size between 30 and 2,000. For each sample size, we estimated the probability that diagnostic accuracy would fail to meet the TPP criteria after implementation.
Results: For a validation study that demonstrates ‘desirable’ sensitivity within a sample of 30 participants who test positive for COVID-19 using the reference standard, the probability that real-world performance will fail to meet the ‘desirable’ criteria is 10.7–13.5%, depending on prevalence. Theoretically, demonstrating the 'desirable' performance in 90 positive participants would reduce that probability to below 5%. A marked reduction in the probability of failure to hit ‘desirable’ specificity occurred between samples of 100 (19.1–21.5%) and 160 (4.3–4.8%) negative participants. There was little further improvement above sample sizes of 160 negative participants.
Conclusion: Based on imprecision alone, small evaluation studies can lead to the acceptance of diagnostic tests which are likely to fail to meet performance targets when deployed. There is diminished return on uncertainty surrounding an accuracy estimate above a total sample size of 250 (90 positive and 160 negative).
- Publication status:
- Published
- Peer review status:
- Peer reviewed
Actions
Access Document
- Files:
-
-
(Preview, Version of record, pdf, 2.1MB, Terms of use)
-
- Publisher copy:
- 10.1186/s41512-021-00116-4
Authors
Contributors
- Role:
- Contributor
- Institution:
- University of Oxford
- Division:
- MSD
- Department:
- Primary Care Health Sciences
- Role:
- Contributor
- Publisher:
- BioMed Central
- Journal:
- Diagnostic and Prognostic Research More from this journal
- Volume:
- 6
- Issue:
- 1
- Article number:
- 12
- Publication date:
- 2022-04-25
- Acceptance date:
- 2021-12-20
- DOI:
- EISSN:
-
2397-7523
- Pmid:
-
35468850
- Language:
-
English
- Keywords:
- Pubs id:
-
1259010
- Local pid:
-
pubs:1259010
- Deposit date:
-
2023-12-21
Terms of use
- Copyright holder:
- Sammut-Powell et al.
- Copyright date:
- 2022
- Rights statement:
- Copyright © 2022, The Author(s). This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
- Licence:
- CC Attribution (CC BY)
If you are the owner of this record, you can report an update to it here: Report update to this record