An Introduction of Principles and Methods of Applying GRADE to Network Meta-analysis_Chinese Journal of Evidence-Based Medicine

Authors：

 YANGNan ¹ , XIAOShu-jun ² , ZHOU ² , YUYang ² , YAOLiang ^3,4,5 , CHENYao-long ^3,4,5 , TIANJin-hui ^3,4,5 , LILun ^1,3,4,5 ,  YANGKe-hu ^3,4,5

1. The First Clinical Medicine College of Lanzhou University, Lanzhou 730000, China;
2. School of Basic Medical Sciences, Lanzhou University, Lanzhou 730000, China;
3. Key Laboratory of Evidence-Based Medicine and Knowledge Translation of Gansu Province, Lanzhou 730000, China;
4. Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou 730000, China;
5. Chinese GRADE Center, Lanzhou University, Lanzhou 730000, China;

Corresponding author：

YANGKe-hu, Email: yangkh@lzu.edu.cn

Keywords：

GRADE; Network meta-analysis; Evidence quality

DOI：

10.7507/1672-2531.20160092

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

The biggest advantages of network meta-analysis (NMA) are to compare the effectiveness of different interventions about one conditions using a quantitative way, pool the results of direct comparison and indirect comparison, and rank the effectiveness based on outcomes, so as to select the best decision for patients. In the paper we introduce the methods of applying GRADE system to NMAs based on the papers published by GRADE working group and other relative studies. The steps of using GRADE to NMAs are mainly based on four aspects: firstly, presenting direct and indirect effect estimates and 95% CI; secondly, rating of quality of direct and indirect estimates; thirdly, presenting the results of NMAs; and the last step is to rating the quality of NMA effect estimates. The methods of rating the quality of direct comparison are the same to use GRADE in traditional meta-analysis. The rating of the quality of the indirect estimates is based on the ratings of the two pairwise estimates that contributes to the indirect estimate of the comparison of interest. The lower confidence rating of the two direct comparisons constitutes the confidence rating of the indirect comparison. When both direct and indirect evidence are available, we suggest using the higher of the two quality ratings as the quality rating for NMA estimate. The four steps of rating the quality of NMA from GRADE working group have promoted the theoretical system of NMA. But the process requires the evaluators to be familiar with GRADE system, and conduct pilot test to make sure the evaluators had understood the items of GRADE system correctly. In addition, we also need to concern that the non-transitivity among different groups and the inconsistency between direct comparison and indirect comparison.

Citation： YANGNan, XIAOShu-jun, ZHOU, YUYang, YAOLiang, CHENYao-long, TIANJin-hui, LILun, YANGKe-hu. An Introduction of Principles and Methods of Applying GRADE to Network Meta-analysis. Chinese Journal of Evidence-Based Medicine, 2016, 16(5): 598-603. doi: 10.7507/1672-2531.20160092 Copy

1.	Jansen JP, Fleurence R, Devine B, et al. Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. Value Health, 2011, 14(4): 417-428.
2.	Jansen JP, Trikalinos T, Cappelleri JC, et al. Indirecttreatment comparison/network meta-analysis study questionnaire to assess relevanceand credibility to inform health care decision making: an ISPOR-AMCP-NPC Good PracticeTask Force report. Value Health, 2014, 17(2): 157-173.
3.	陈耀龙, 姚亮, 杜亮, 等. GRADE 在系统评价中应用的必要性及注意事项. 中国循证医学杂志, 2013, 13(12): 1401-1404.
4.	陈耀龙, 姚亮, 杜亮, 等. GRADE在诊断准确性试验系统评价中应用的原理、方法、挑战及发展趋势. 中国循证医学杂志, 2014, 14(11): 1402-1406.
5.	姚亮, 陈耀龙, 杜亮, 等. GRADE在诊断试验系统评价中应用的实例解读. 中国循证医学杂志, 2014, 2014, 14(11): 1407-1412.
6.	Puhan MA, Schünemann HJ, et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ, 2014.
7.	Salanti G, Del Giovane C, Chaimani A, et al. Evaluating the quality of evidence from a network meta-analysis. PLoS One, 2014, 9(7): e99682.
8.	Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines: 8. Rating the quality of evidence——indirectness. J Clin Epidemiol, 2011, 64(12): 1303-1310.
9.	Balshem H, Helfand M, Schünemann HJ, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol, 2011, 64(4): 401-406.
10.	Guyatt GH, Oxman AD, Schünemann HJ, et al. GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol, 2011, 64(4): 380-382.
11.	Guyatt GH, Oxman AD, Vist G, et al. GRADE guidelines: 4. Rating the quality of evidence——study limitations (risk of bias). J Clin Epidemiol, 2011, 64(4): 407-415.
12.	Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines: 7. Rating the quality of evidence——inconsistency. J Clin Epidemiol, 2011, 64(12): 1294-1302.
13.	Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines 6. Rating the quality of evidence——imprecision. J Clin Epidemiol, 2011, 64(12): 1283-1293.
14.	Guyatt GH, Oxman AD, Montori V, et al. GRADE guidelines: 5. Rating the quality of evidence——publication bias. J Clin Epidemiol, 2011, 64(12): 1277-1282.
15.	van Enst WA1, Ochodo E, Scholten RJ, et al. Investigation of publication bias in meta-analyses of diagnostic test accuracy: a meta-epidemiological study. BMC Med Res Methodol, 2014, 14(1): 70.
16.	Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics, 1994: 1088-1101.
17.	Egger M, Smith GD, Schneider M, et al. Bias in meta-analysis detected by a simple, graphical test. BMJ, 1997, 315(109): 629-634.
18.	Iorio A, Spencer FA, Falavigna M, et al. Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients. BMJ, 2015, 350: h870.
19.	Murad MH, Drake MT, Mullan RJ, et al. Clinical review. Comparative effectiveness of drug treatments to prevent fragility fractures: a systematic review and network meta-analysis. J Clin Endocrinol Metab, 2012, 97(6): 1871-1880.
20.	Altman D G, Song F, Sakarovitch C, et al. Indirect comparisons of competing interventions. Gray Publishing, 2005.
21.	Dias S, Welton NJ, Caldwell DM, Ades AE. Checking consistency in mixed treatmentcomparison meta-analysis. Stat Med, 2010, 29(7-8): 932-944.
22.	Kessels A, ter Riet G, Puhan M, et al. A simple regression model for network meta-analysis. OA Epidemiology, 2013, 1(1): 7.
23.	Higgins JPT, Jackson D, Barrett JK, et al. Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies. Res Synth Methods, 2012, 3(2): 98-110.
24.	张超, 鄢金柱, 孙凤, 等. 网状 Meta 分析一致性的鉴别与处理方法. 中国循证医学杂志, 2014, 14(7): 884-888.
25.	Avenell A, Mak JCS, O'Connell D. Vitamin D and vitamin D analogues for preventing fractures in post-menopausal women and older men. Cochrane Database Syst Rev, 2014, 4: CD000227.
26.	Sutton A, Ades AE, Cooper N, et al. Use of indirect and mixed treatment comparisons for technology assessment. Pharmacoeconomics, 2008, 26(9): 753-767.
27.	Madan J, Stevenson MD, Cooper KL, et al. Consistency between direct and indirect trial evidence: is direct evidence always more reliable? Value Health, 2011, 14(6): 953-960.
28.	Sturtz S, Bender R. Unsolved issues of mixed treatment comparison meta-analysis: network size and inconsistency. Res Synth Methods, 2012, 3(4): 300-311.
29.	Song F, Clark A, Bachmann M, Maas J. Simulation evaluation of statistical properties of methods for indirect and mixed treatment comparisons. BMC Med Res Methodol, 2012, 12(1): 138.
30.	Mills E, Bansback N, Ghement I, et al. Multiple treatment comparison meta-analyses: a step forward into complexity. Clin Epidemiol, 2011, 3: 193-202.
31.	Dias S, Welton NJ, Ades AE. Study designs to detect sponsorship and other biases in systematic reviews. J Clin Epidemiol, 2010, 63(6): 587-588.
32.	Piepho HP. Network-meta analysis made easy: Detection of inconsistency using factorial analysis-of-variance models. BMC Med Res Methodol, 2014, 14(1): 61.
33.	Cipriani A, Higgins JPT, Geddes JR, et al. Conceptual and technical challenges in network meta-analysis. Ann Intern Med, 2013, 159(2): 130-137.
34.	Hoaglin DC, Hawkins N, Jansen JP, et al. Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2. Value Health, 2011, 14(4): 429-437.
35.	Hutton B, Salanti G, Chaimani A, et al. The qualityof reporting methods and results in network meta-analyses: an overview of reviews andsuggestions for improvement. PLoS One, 2014, 9(3): e92508.
36.	Stettler C, Wandel S, Allemann S, et al. Outcomes associated with drug-eluting and bare-metal stents: a collaborative network meta-analysis. Lancet, 2007, 370(9591): 937-948.
37.	Palmerini T, Biondi-Zoccai G, Riva DD, et al. Risk of stroke with percutaneous coronary intervention compared with on-pump and off-pump coronary artery bypass graft surgery: Evidence from a comprehensive network meta-analysis. Am Heart J, 2013, 165(6): 910-917.
38.	Pandor A, Gomersall T, Stevens JW, et al. Remote monitoring after recent hospital discharge in patients with heart failure: a systematic review and network meta-analysis. Heart, 2013, 99(23): 1717-1726.
39.	Windecker S, Stortecky S, Stefanini GG, et al. Revascularisation versus medical treatment in patients with stable coronary artery disease: network meta-analysis. BMJ, 2014, 348: g3859.
40.	Mustafa RA, Santesso N, Brozek J, et al. The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses. J Clin Epidemiol, 2013, 66(7): 736-742.
41.	田金徽, 陈耀龙, 杨克虎, 等. SR/MA研究进展与挑战. 兰州大学学报(医学版). 2016, 42(1): 42-47.

1. Jansen JP, Fleurence R, Devine B, et al. Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. Value Health, 2011, 14(4): 417-428.
2. Jansen JP, Trikalinos T, Cappelleri JC, et al. Indirecttreatment comparison/network meta-analysis study questionnaire to assess relevanceand credibility to inform health care decision making: an ISPOR-AMCP-NPC Good PracticeTask Force report. Value Health, 2014, 17(2): 157-173.
3. 陈耀龙, 姚亮, 杜亮, 等. GRADE 在系统评价中应用的必要性及注意事项. 中国循证医学杂志, 2013, 13(12): 1401-1404.
4. 陈耀龙, 姚亮, 杜亮, 等. GRADE在诊断准确性试验系统评价中应用的原理、方法、挑战及发展趋势. 中国循证医学杂志, 2014, 14(11): 1402-1406.
5. 姚亮, 陈耀龙, 杜亮, 等. GRADE在诊断试验系统评价中应用的实例解读. 中国循证医学杂志, 2014, 2014, 14(11): 1407-1412.
6. Puhan MA, Schünemann HJ, et al. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ, 2014.
7. Salanti G, Del Giovane C, Chaimani A, et al. Evaluating the quality of evidence from a network meta-analysis. PLoS One, 2014, 9(7): e99682.
8. Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines: 8. Rating the quality of evidence——indirectness. J Clin Epidemiol, 2011, 64(12): 1303-1310.
9. Balshem H, Helfand M, Schünemann HJ, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol, 2011, 64(4): 401-406.
10. Guyatt GH, Oxman AD, Schünemann HJ, et al. GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol, 2011, 64(4): 380-382.
11. Guyatt GH, Oxman AD, Vist G, et al. GRADE guidelines: 4. Rating the quality of evidence——study limitations (risk of bias). J Clin Epidemiol, 2011, 64(4): 407-415.
12. Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines: 7. Rating the quality of evidence——inconsistency. J Clin Epidemiol, 2011, 64(12): 1294-1302.
13. Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines 6. Rating the quality of evidence——imprecision. J Clin Epidemiol, 2011, 64(12): 1283-1293.
14. Guyatt GH, Oxman AD, Montori V, et al. GRADE guidelines: 5. Rating the quality of evidence——publication bias. J Clin Epidemiol, 2011, 64(12): 1277-1282.
15. van Enst WA1, Ochodo E, Scholten RJ, et al. Investigation of publication bias in meta-analyses of diagnostic test accuracy: a meta-epidemiological study. BMC Med Res Methodol, 2014, 14(1): 70.
16. Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics, 1994: 1088-1101.
17. Egger M, Smith GD, Schneider M, et al. Bias in meta-analysis detected by a simple, graphical test. BMJ, 1997, 315(109): 629-634.
18. Iorio A, Spencer FA, Falavigna M, et al. Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients. BMJ, 2015, 350: h870.
19. Murad MH, Drake MT, Mullan RJ, et al. Clinical review. Comparative effectiveness of drug treatments to prevent fragility fractures: a systematic review and network meta-analysis. J Clin Endocrinol Metab, 2012, 97(6): 1871-1880.
20. Altman D G, Song F, Sakarovitch C, et al. Indirect comparisons of competing interventions. Gray Publishing, 2005.
21. Dias S, Welton NJ, Caldwell DM, Ades AE. Checking consistency in mixed treatmentcomparison meta-analysis. Stat Med, 2010, 29(7-8): 932-944.
22. Kessels A, ter Riet G, Puhan M, et al. A simple regression model for network meta-analysis. OA Epidemiology, 2013, 1(1): 7.
23. Higgins JPT, Jackson D, Barrett JK, et al. Consistency and inconsistency in network meta-analysis: concepts and models for multi-arm studies. Res Synth Methods, 2012, 3(2): 98-110.
24. 张超, 鄢金柱, 孙凤, 等. 网状 Meta 分析一致性的鉴别与处理方法. 中国循证医学杂志, 2014, 14(7): 884-888.
25. Avenell A, Mak JCS, O'Connell D. Vitamin D and vitamin D analogues for preventing fractures in post-menopausal women and older men. Cochrane Database Syst Rev, 2014, 4: CD000227.
26. Sutton A, Ades AE, Cooper N, et al. Use of indirect and mixed treatment comparisons for technology assessment. Pharmacoeconomics, 2008, 26(9): 753-767.
27. Madan J, Stevenson MD, Cooper KL, et al. Consistency between direct and indirect trial evidence: is direct evidence always more reliable? Value Health, 2011, 14(6): 953-960.
28. Sturtz S, Bender R. Unsolved issues of mixed treatment comparison meta-analysis: network size and inconsistency. Res Synth Methods, 2012, 3(4): 300-311.
29. Song F, Clark A, Bachmann M, Maas J. Simulation evaluation of statistical properties of methods for indirect and mixed treatment comparisons. BMC Med Res Methodol, 2012, 12(1): 138.
30. Mills E, Bansback N, Ghement I, et al. Multiple treatment comparison meta-analyses: a step forward into complexity. Clin Epidemiol, 2011, 3: 193-202.
31. Dias S, Welton NJ, Ades AE. Study designs to detect sponsorship and other biases in systematic reviews. J Clin Epidemiol, 2010, 63(6): 587-588.
32. Piepho HP. Network-meta analysis made easy: Detection of inconsistency using factorial analysis-of-variance models. BMC Med Res Methodol, 2014, 14(1): 61.
33. Cipriani A, Higgins JPT, Geddes JR, et al. Conceptual and technical challenges in network meta-analysis. Ann Intern Med, 2013, 159(2): 130-137.
34. Hoaglin DC, Hawkins N, Jansen JP, et al. Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2. Value Health, 2011, 14(4): 429-437.
35. Hutton B, Salanti G, Chaimani A, et al. The qualityof reporting methods and results in network meta-analyses: an overview of reviews andsuggestions for improvement. PLoS One, 2014, 9(3): e92508.
36. Stettler C, Wandel S, Allemann S, et al. Outcomes associated with drug-eluting and bare-metal stents: a collaborative network meta-analysis. Lancet, 2007, 370(9591): 937-948.
37. Palmerini T, Biondi-Zoccai G, Riva DD, et al. Risk of stroke with percutaneous coronary intervention compared with on-pump and off-pump coronary artery bypass graft surgery: Evidence from a comprehensive network meta-analysis. Am Heart J, 2013, 165(6): 910-917.
38. Pandor A, Gomersall T, Stevens JW, et al. Remote monitoring after recent hospital discharge in patients with heart failure: a systematic review and network meta-analysis. Heart, 2013, 99(23): 1717-1726.
39. Windecker S, Stortecky S, Stefanini GG, et al. Revascularisation versus medical treatment in patients with stable coronary artery disease: network meta-analysis. BMJ, 2014, 348: g3859.
40. Mustafa RA, Santesso N, Brozek J, et al. The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses. J Clin Epidemiol, 2013, 66(7): 736-742.
41. 田金徽, 陈耀龙, 杨克虎, 等. SR/MA研究进展与挑战. 兰州大学学报(医学版). 2016, 42(1): 42-47.

Previous Article
Using SYRCLE Tools to Evaluate the Methodological Quality of Animal Experiments of Stroke in China
Next Article
Application of the Trial Sequential Analysis Software for Meta-analysis

Chinese Journal of Evidence-Based Medicine

An Introduction of Principles and Methods of Applying GRADE to Network Meta-analysis

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content