The QUADAS-2, QUIPS, and PROBAST tools are not specific for prognostic accuracy studies and the use of these tools to assess the risk of bias in prognostic accuracy studies is prone to bias. Therefore, QUAPAS, a risk of bias assessment tool for prognostic accuracy studies, has recently been developed. The tool combines QUADAS-2, QUIPS, and PROBAST, and consists of 5 domains, 18 signaling questions, 5 risk of bias questions, and 4 applicability questions. This paper will introduce the content and usage of QUAPAS to provide inspiration and references for domestic researchers.
ObjectivesTo comprehensively evaluate the methodological quality and applicability of the results of systematic reviews on acupuncture treatment for primary depression.MethodsWeb of Science, EMbase, PubMed, The Cochrane Library, CNKI, CBM, WanFang Data and VIP databases were electronically searched to collect systematic reviews/meta-analyses on acupuncture treatment for primary depression from inception to December 5th, 2018. Two researchers independently screened and extracted data by using tools of AMSTAR 2 to evaluate the methodological quality, using ROBIS to assess risk of bias, and using CASP-S.R to evaluate the applicability of the results.ResultsA total of 18 systematic reviews/meta-analyses were included, and all focused on acupuncture intervention, including 2 primary outcome indicators. According to AMSTAR 2 evaluation results, there were 4 high quality studies, 12 medium quality studies and 2 low quality studies; ROBIS results found 10 high bias risk studies, 7 low bias risk studies and 1 unclear; CASP-S.R showed only 4 design studies applicable to local individuals, and there were no studies on the relationship between design benefits, hazards and costs.ConclusionsThe quality of systematic reviews/meta-analyses for acupuncture treatment of primary depression is moderate, however with a certain bias. Most studies may not directly benefit local individuals. All studies have no relationship with cost hazards. It is expected for further reviewers to strictly follow systematic evaluation method to improve research quality and reduce bias, while the applicability of the systematic review to individuals from different regions should be considered as well as the relationship between the benefit and cost hazard. In addition, more valid RCTs are required to provide higher quality evidence and explore correlated and comprehensive mechanism.
Evidence synthesis is the process of systematically gathering, analyzing, and integrating available research evidence. The quality of evidence synthesis depends on the quality of the original studies included. Validity assessment, also known as risk of bias assessment, is an essential method for assessing the quality of these original studies. Currently, there are numerous validity assessment tools available, but some of them lack a rigorous development process and evaluation. The application of inappropriate validity assessment tools to assessing the quality of the original studies during the evidence synthesis process may compromise the accuracy of study conclusions and mislead the clinical practice. To address this dilemma, the LATITUDES Network, a one-stop resource website for validity assessment tools, was established in September 2023, led by academics at the University of Bristol, U.K. This Network is dedicated to collecting, sorting and promoting validity assessment tools to improve the accuracy of original study validity assessments and increase the robustness and reliability of the results of evidence synthesis. This study introduces the background of the establishment of the LATITUDES Network, the included validity assessment tools, and the training resources for the use of validity assessment tools, in order to provide a reference for domestic scholars to learn more about the LATITUDES Network, to better use the appropriate validity assessment tools to conduct study quality assessments, and to provide references for the development of validity assessment tools.
This study aims to introduce how to use the PROBAST (prediction model risk of bias assessment tool) to evaluate risk of bias and applicability of the study of diagnostic or prognostic predictive models, including the introduction of the background, the scope of application and use of the tool. This tool mainly involves the four areas of participants, predictors, outcomes and analyses. The risk of bias in the research is evaluated through the four areas, while the applicability is evaluated in the first three. PROBAST provides a standardized approach to evaluate the critical appraisal of the study of diagnostic or prognostic predictive models, which screens qualified literature for data analysis and helps to establish a scientific basis for clinical decision-making.
RoB2 (revised version 2019), an authoritative tool for assessing the risk of bias in randomized controlled trials, has been updated and improved based on the original version. This article elaborated and interpreted the background and main content of RoB2 (revised version 2019), as well as the operation process of the new software. Compared with the previous version of RoB2 (revised version 2018), RoB2 (revised version 2019) has the advantages of rich content, complete details, accurate questions, and simple operation, etc. Additionally, it is more user-friendly for researchers and beginners. The risk bias assessment of randomized controlled trials is more comprehensive and accurate, and it is an authoritative, trustworthy, and popular tool for evaluating the risk of bias in randomized controlled studies in medical practice.
With the rapid development of artificial intelligence (AI) and machine learning technologies, the development of AI-based prediction models has become increasingly prevalent in the medical field. However, the PROBAST tool, which is used to evaluate prediction models, has shown growing limitations when assessing models built on AI technologies. Therefore, Moons and colleagues updated and expanded PROBAST to develop the PROBAST+AI tool. This tool is suitable for evaluating prediction model studies based on both artificial intelligence methods and regression methods. It covers four domains: participants and data sources, predictors, outcomes, and analysis, allowing for systematic assessment of quality in model development, risk of bias in model evaluation, and applicability. This article interprets the content and evaluation process of the PROBAST+AI tool, aiming to provide references and guidance for domestic researchers using this tool.
Selective non-reporting and publication bias of study results threaten the validity of systematic reviews and meta-analyses, thus affect clinical decision making. There are no rigorous methods to evaluate the risk of bias in network meta-analyses currently. This paper introduces the main contents of ROB-MEN (risk of bias due to missing evidence in network meta-analysis), including tables of the tool, operation process and signal questions. The pairwise comparisons table and the ROB-MEN table are the tool’s core. The ROB-MEN tool can be applied to very large and complex networks including lots of interventions to avoid time-consuming and labor-intensive process, and it has the advantages of clear logic, complete details and good applicability. It is the first tool used to evaluate the risk of bias due to missing evidence in network meta-analysis and is useful to researchers, thus being worth popularizing and applying.
The COSMIN community updated the COSMIN-RoB checklist on reliability and measurement error in 2021. The updated checklist can be applied to the assessment of all types of outcome measurement studies, including clinician-reported outcome measures (ClinPOMs), performance-basd outcome measurement instruments (PerFOMs), and laboratory values. In order to help readers better understand and apply the updated COSMIN-RoB checklist and provide methodological references for conducting systematic reviews of ClinPOMs, PerFOMs and laboratory values, this paper aimed to interpret the updated COSMIN-RoB checklist on reliability and measurement error studies.
Objective To systematically review the accuracy and consistency of large language models (LLM) in assessing risk of bias in analytical studies. Methods The cohort and case-control studies related to COVID-19 based on the team's published systematic review of clinical characteristics of COVID-19 were included. Two researchers independently screened the studies, extracted data, and assessed risk of bias of the included studies with the LLM-based BiasBee model (version Non-RCT) used for automated evaluation. Kappa statistics and score differences were used to analyze the agreement between LLM and human evaluations, with subgroup analysis for Chinese and English studies. Results A total of 210 studies were included. Meta-analysis showed that LLM scores were generally higher than those of human evaluators, particularly in representativeness of exposed cohorts (△=0.764) and selection of external controls (△=0.109). Kappa analysis indicated slight agreement in items such as exposure assessment (κ=0.059) and adequacy of follow-up (κ=0.093), while showing significant discrepancies in more subjective items, such as control selection (κ=−0.112) and non-response rate (κ=−0.115). Subgroup analysis revealed higher scoring consistency for LLM in English-language studies compared to that of Chinese-language studies. Conclusion LLM demonstrate potential in risk of bias assessment; however, notable differences remain in more subjective tasks. Future research should focus on optimizing prompt engineering and model fine-tuning to enhance LLM accuracy and consistency in complex tasks.
The risk of bias assessment tool 2.0 (RoB 2.0) for analyzing cluster randomized trials and crossover trials (revised version 2021) has been updated. The current paper briefly delineates the history of the RoB 2.0 tool and includes an explanation and interpretation of the updated contents and software operation process for use with cluster randomized trials and crossover trials. Compared with the previous versions, the updated RoB 2.0 tool (revised version 2021) has the advantage of precise language and is easily understood. Thus, the updated RoB 2.0 tool merits popularization and further general application.