Evaluation of the accuracy of the large language model for risk of bias assessment in analytical studies_Chinese Journal of Evidence-Based Medicine

Authors：

SONG Tian , GUO Jing , LIU Bin , GONG Zhaoyuan , HU Mingzhi , CHEN Lin ,  CHE Qianzi ,  SHI Nannan

Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing 100700, P. R. China;

Corresponding author：

CHE Qianzi, Email: cheqianzi123@126.com; SHI Nannan, Email: 13811839164@vip.126.com

Keywords：

Large language models (LLM); Risk of bias; Analytical studies; Automatic analysis; Meta-analysis

DOI：

10.7507/1672-2531.202411028

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Objective To systematically review the accuracy and consistency of large language models (LLM) in assessing risk of bias in analytical studies. Methods The cohort and case-control studies related to COVID-19 based on the team's published systematic review of clinical characteristics of COVID-19 were included. Two researchers independently screened the studies, extracted data, and assessed risk of bias of the included studies with the LLM-based BiasBee model (version Non-RCT) used for automated evaluation. Kappa statistics and score differences were used to analyze the agreement between LLM and human evaluations, with subgroup analysis for Chinese and English studies. Results A total of 210 studies were included. Meta-analysis showed that LLM scores were generally higher than those of human evaluators, particularly in representativeness of exposed cohorts (△=0.764) and selection of external controls (△=0.109). Kappa analysis indicated slight agreement in items such as exposure assessment (κ=0.059) and adequacy of follow-up (κ=0.093), while showing significant discrepancies in more subjective items, such as control selection (κ=−0.112) and non-response rate (κ=−0.115). Subgroup analysis revealed higher scoring consistency for LLM in English-language studies compared to that of Chinese-language studies. Conclusion LLM demonstrate potential in risk of bias assessment; however, notable differences remain in more subjective tasks. Future research should focus on optimizing prompt engineering and model fine-tuning to enhance LLM accuracy and consistency in complex tasks.

Citation： SONG Tian, GUO Jing, LIU Bin, GONG Zhaoyuan, HU Mingzhi, CHEN Lin, CHE Qianzi, SHI Nannan. Evaluation of the accuracy of the large language model for risk of bias assessment in analytical studies. Chinese Journal of Evidence-Based Medicine, 2025, 25(5): 562-567. doi: 10.7507/1672-2531.202411028 Copy

1.	Borenstein M, Hedges LV, Higgins JPT, et al. Introduction to meta-analysis. Hoboken: Wiley, 2009.
2.
3.
4.
5.
6.
7.
8.
9.	Feng H, Du S, Zhu G, et al. Leveraging large language models for automated Chinese essay scoring. Cham: Springer, 2024.
10.
11.	Huang H, Tang T, Zhang D, et al. Not all languages are created equal in LLMs: improving multilingual capability by cross-lingual-thought prompting. 2023.
12.	Tian S, Qianzi C, Ning L, et al. The distribution pattern of symptoms among different variants of COVID-19. 2023.
13.	Safary Official. Safary - streamlining automation for all systematic reviews and analysis. 2024.
14.
15.
16.
17.	Zhai X, Nyaaba M, Ma W. Can generative AI and ChatGPT outperform humans on cognitive-demanding problem-solving tasks in science? Sci Educ (Dordr), 2024.
18.	Christopher JR, Martin R, Julia B, et al. Using a large language model (ChatGPT) to assess risk of bias in randomized controlled trials of medical interventions: protocol for a pilot study of interrater agreement with human reviewers. 2023.
19.
20.
21.	Ho N, Schmid L, Yun SY. Large language models are reasoning teachers. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023.
22.	Ranganath K, Piyush K, Omesh T. Enhancing trust in large language models with uncertainty-aware fine-tuning. The Thirteenth International Conference on Learning Representations. 2024.
23.	Zhang J, Muhamed A, Anantharaman A, et al. ReAugKD: retrieval-augmented knowledge distillation for pre-trained language models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023.
24.	Zhang S, Liang Y, Wang S, et al. Towards understanding and improving knowledge distillation for neural machine translation. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023.
25.	Ortega L. Understanding second language acquisition. 2014.
26.
27.
28.	Clark K, Khandelwal U, Levy O, et al. What does bert look at? An analysis of BERT’s attention. 2019.

1. Borenstein M, Hedges LV, Higgins JPT, et al. Introduction to meta-analysis. Hoboken: Wiley, 2009.
2.
3.
4.
5.
6.
7.
8.
9. Feng H, Du S, Zhu G, et al. Leveraging large language models for automated Chinese essay scoring. Cham: Springer, 2024.
10.
11. Huang H, Tang T, Zhang D, et al. Not all languages are created equal in LLMs: improving multilingual capability by cross-lingual-thought prompting. 2023.
12. Tian S, Qianzi C, Ning L, et al. The distribution pattern of symptoms among different variants of COVID-19. 2023.
13. Safary Official. Safary - streamlining automation for all systematic reviews and analysis. 2024.
14.
15.
16.
17. Zhai X, Nyaaba M, Ma W. Can generative AI and ChatGPT outperform humans on cognitive-demanding problem-solving tasks in science? Sci Educ (Dordr), 2024.
18. Christopher JR, Martin R, Julia B, et al. Using a large language model (ChatGPT) to assess risk of bias in randomized controlled trials of medical interventions: protocol for a pilot study of interrater agreement with human reviewers. 2023.
19.
20.
21. Ho N, Schmid L, Yun SY. Large language models are reasoning teachers. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023.
22. Ranganath K, Piyush K, Omesh T. Enhancing trust in large language models with uncertainty-aware fine-tuning. The Thirteenth International Conference on Learning Representations. 2024.
23. Zhang J, Muhamed A, Anantharaman A, et al. ReAugKD: retrieval-augmented knowledge distillation for pre-trained language models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023.
24. Zhang S, Liang Y, Wang S, et al. Towards understanding and improving knowledge distillation for neural machine translation. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023.
25. Ortega L. Understanding second language acquisition. 2014.
26.
27.
28. Clark K, Khandelwal U, Levy O, et al. What does bert look at? An analysis of BERT’s attention. 2019.

Chinese Journal of Evidence-Based Medicine

Evaluation of the accuracy of the large language model for risk of bias assessment in analytical studies

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content