Discriminative Cut-Offs, Concurrent Criterion Validity, and Test–Retest Reliability of the Oxford Vaccine Hesitancy Scale

Journal article

Background: Validated instruments can be used to quantify vaccine hesitancy, but few provide transportable operational cut-offs or temporal stability metrics. We evaluated the Oxford Vaccine Hesitancy Scale (OVHS) to quantify discrimination for the self-reported historical COVID-19 vaccine refusal, derive and validate a single operational cut-off across populations, and assess the test–retest reliability. Methods: Five datasets (including one from a repeat administration) comprising 2451 assessments from 1989 demographically representative UK and US respondents were analyzed without pooling. Receiver operating characteristic (ROC) curves with 10,000 bootstrap replications (bias-corrected and accelerated and percentile Cis) were used to quantify discrimination. A Youden’s J near-optimal plateau algorithm constrained by a cross-dataset specificity floor (≥0.75) was used to select a transportable cut-off. A prevalence-agnostic aggregate Index of Union (Iuagg) provided secondary confirmation of this cut-off. Cut-off behaviour was visualized with a multi-sample two-graph ROC plot. The six-week test–retest reliability on a UK sample used a two-way mixed-effects, absolute-agreement ICC(A,1). Results: AUCs ranged 0.760–0.971 across datasets (derivation AUC: 0.960). The Youden plateau spanned scores 34–38; applying the specificity floor yielded an operational cut-off OVHS ≥ 35, which was confirmed by the Iuagg. At ≥35, sensitivity/specificity were 0.73–0.95/0.63–0.87 across samples; negative predictive value exceeded 0.90 in all cohorts. The test–retest reliability was good to excellent, with the OVHS total ICC(A,1) = 0.884 (95% CI 0.863–0.903); subscales ranged from 0.649 to 0.901 and the average-measure ICC(A,2) = 0.939. Conclusions: The OVHS demonstrates good-to-excellent discrimination for historical COVID-19 vaccine refusal and strong temporal stability. We found a single, transparent, and transportable operational cut-off (≥35). Our cut-off derivation framework is scale- and endpoint-agnostic and may generalize to other hesitancy instruments and decision cut-offs more broadly.

Authors: Kantor, J; Vanderslott, S; Morrison, M; Carlisle, RC

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: MDPI
• Journal: Vaccines
• Volume: 13
• Issue: 12
• Pages: 1200
• Article number: 1200
• Publication date: 2025-11-28
• Acceptance date: 2025-11-20
• DOI: 10.3390/vaccines13121200
• EISSN: 2076-393X
• ISSN: 2076-393X
• Language: English
• Keywords: test–retest reliability; psychometrics; vaccine hesitancy; Index of Union; ROC; Youden’s J; Oxford Vaccine Hesitancy Scale; cut-off; COVID-19