Tool to assess risk of bias in studies estimating the prevalence of mental health disorders (RoB-PrevMH)

Journal article

Objective There is no standard tool for assessing risk of bias (RoB) in prevalence studies. For the purposes of a living systematic review during the COVID-19 pandemic, we developed a tool to evaluate RoB in studies measuring the prevalence of mental health disorders (RoB-PrevMH) and tested inter-rater reliability.Methods We decided on items and signalling questions to include in RoB-PrevMH through iterative discussions. We tested the reliability of assessments by different users with two sets of prevalence studies. The first set included a random sample of 50 studies from our living systematic review. The second set included 33 studies from a systematic review of the prevalence of post-traumatic stress disorders, major depression and generalised anxiety disorder. We assessed the inter-rater agreement by calculating the proportion of agreement and Kappa statistic for each item.Results RoB-PrevMH consists of three items that address selection bias and information bias. Introductory and signalling questions guide the application of the tool to the review question. The inter-rater agreement for the three items was 83%, 90% and 93%. The weighted kappa scores were 0.63 (95% CI 0.54 to 0.73), 0.71 (95% CI 0.67 to 0.85) and 0.32 (95% CI −0.04 to 0.63), respectively.Conclusions RoB-PrevMH is a brief, user-friendly and adaptable tool for assessing RoB in studies on prevalence of mental health disorders. Initial results for inter-rater agreement were fair to substantial. The tool’s validity, reliability and applicability should be assessed in future projects

Authors: Tonia, T; Buitrago-Garcia, D; Peter, NL; Mesa-Vieira, C; Li, T

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: BMJ Publishing Group
• Journal: BMJ Mental Health
• Volume: 26
• Issue: 1
• Pages: e300694-e300694
• Publication date: 2023-10-29
• Acceptance date: 2023-09-29
• DOI: 10.1136/bmjment-2023-300694
• EISSN: 2755-9734
• ISSN: 2755-9734
• Language: English
• Keywords: Medicine; Statistics; Inter-rater reliability; Depression (economics); Clinical psychology; Psychology; Mathematics; Mental health; Psychiatry; Reliability (semiconductor); Kappa; Cohen's kappa; Anxiety