Evaluating the performance of neural networks in propensity score estimation_Chinese Journal of Evidence-Based Medicine

Authors：

GOU Zhongping ¹ , XIANG Liangcheng ² , LIANG Xiufang ¹ , ZHANG Wei ³ , CHEN Peiran ² ,  FENG Ping ¹

1. Institute of Drug Clinical Trials, West China Hospital, Sichuan University, Chengdu 610041, P.R.China;
2. West China Second University Hospital, Sichuan University, Chengdu 610041, P.R.China;
3. Cheng Fei Hospital, Chengdu 610041, P.R.China;

Corresponding author：

FENG Ping, Email: fengping@wchscu.cn

Keywords：

Propensity score; Logistic regression; Neural networks; Simulation

DOI：

10.7507/1672-2531.201910108

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Objectives To explore the value of neural networks (NN) in estimating propensity score, and to compare the performance of propensity score methods based on both logistic regression (LR) and NN.Methods Data sets including ten binary or continuous covariates, binary treatment variable and continuous outcome variable were simulated by SAS 9.2 software, and 5 scenarios differing by non-linear and/or non-additive associations between treatment assignment and covariates were set up. The sample sizes 500, 1000, 2000, 5000 and 10000 were considered. Propensity scores were estimated using either LR or NN model using only partial covariates associated with the outcome (methods LR₁, NN₁), or all covariates associated with either outcome or treatment (methods LR₂, NN₂). The average treatment effect (ATE) estimates, standard error (SE), bias, and mean square error (MSE) of ATE among the different models were compared.Results The 95% confidence intervals of the average treatment effect were narrower in NN than that in LR models. SE, bias and MSE increased with the increasing complexity of non-linear and/or non-additive associations between the treatment and covariates, and smaller SE, bias, and MSE were observed in LR₁ than LR₂, and in NN₁ than NN₂. NN generally produced less bias than LR under most scenarios when variables associated with the outcome were introduced. SE and MSE decreased with the increasing sample size for both LR and NN models.Conclusions NN for estimating propensity scores may be less biased and produce more precise estimates for ATE than LR in a meaningful manner when the complex association between treatment and covariates exists.

Citation： GOU Zhongping, XIANG Liangcheng, LIANG Xiufang, ZHANG Wei, CHEN Peiran, FENG Ping. Evaluating the performance of neural networks in propensity score estimation. Chinese Journal of Evidence-Based Medicine, 2020, 20(9): 1075-1082. doi: 10.7507/1672-2531.201910108 Copy

1.	Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika, 1983, 70(1): 41-55.
2.	Austin PC. The performance of different propensity score methods for estimating marginal odds ratios. Stat Med, 2007, 26(16): 3078-3094.
3.	Austin PC. The performance of different propensity-score methods for estimating differences in proportions (risk differences or absolute risk reductions) in observational studies. Stat Med, 2010, 29(20): 2137-2148.
4.	Austin PC. A critical appraisal of propensity-score matching in the medical literature between 1996 and 2003. Stat Med, 2008, 27(12): 2037-2049.
5.	王永吉, 蔡宏伟, 夏结来, 等. 倾向指数第一讲倾向指数的基本概念和研究步骤. 中华流行病学杂志, 2010, 31(3): 347-348.
6.	Abdia Y, Kulasekera KB, Datta S, <italic>et al</italic>. Propensity scores based methods for estimating average treatment effect and average treatment effect among treated: A comparative study. Biom J, 2017, 59(5): 967-985.
7.	Pham CT, Gibb CL, Mittinty MN, <italic>et al</italic>. A comparison of propensity score-based approaches to health service evaluation: a case study of a preoperative physician-led clinic for high-risk surgical patients. J Eval Clin Pract, 2016, 22(5): 761-770.
8.	Wu M, Zhao Y, Wang R, <italic>et al</italic>. Epidemiology of Functional abdominal bloating and its impact on health related quality of life: male-female stratified propensity score analysis in a population based survey in mainland China. PLoS One, 2014, 9(7): e102320.
9.	Kim CY, Collier CD, Liu RW, <italic>et al</italic>. Are limb-sparing surgical resections comparable to amputation for patients with pelvic chondrosarcoma? A case-control, propensity score-matched analysis of the national cancer database. Clin Orthop Relat Res, 2019, 477(3): 596-605.
10.	Spertus JV, Normand ST. Bayesian propensity scores for high-dimensional causal inference: a comparison of drug-eluting to bare-metal coronary stents. Biom J, 2018, 60(4): 721-733.
11.	Brookhart MA, Schneeweiss S, Rothman KJ, <italic>et al</italic>. Variable selection for propensity score models. Am J Epidemiol, 2006, 163(12): 1149-1156.
12.	Westreich D, Lessler J, Funk MJ. Propensity score estimation: neural networks, support vector machines, decision trees (CART), and meta-classifiers as alternatives to logistic regression. J Clin Epidemiol, 2010, 63(8): 826-833.
13.	Thoemmes FJ, Kim ES. A systematic review of propensity score methods in the social sciences. Multivariate Behav Res, 2011, 46(1): 90-118.
14.	Weitzen S, Lapane KL, Toledano AY, <italic>et al</italic>. Principles for modeling propensity scores in medical research: a systematic literature review. Pharmacoepidemiol Drug Saf, 2004, 13(12): 841-853.
15.	吴美京, 吴骋, 王睿, 等. 倾向性评分法中评分值的估计方法及比较. 中国卫生统计, 2013, 30(3): 440-444.
16.	Ellis AR, Dusetzina SB, Hansen RA, <italic>et al</italic>. Confounding control in a non-experimental study of STAR*D data: logistic regression balanced covariates better than boosted CART. Ann Epidemiol, 2013, 23(4): 204-209.
17.	Wyss R, Schneeweiss S, van der Laan M, <italic>et al</italic>. Using super learner prediction modeling to improve high-dimensional propensity score estimation. Epidemiology, 2018, 29(1): 96-106.
18.	McCaffrey DF, Griffin BA, Almirall D, <italic>et al</italic>. A tutorial on propensity score estimation for multiple treatments using generalized boosted models. Stat Med, 2013, 32(19): 3388-3414.
19.	McCandless LC, Gustafson P, Austin PC. Bayesian propensity score analysis for observational data. Stat Med, 2009, 28(1): 94-112.
20.	Linden A, Yarnold PR. Using classification tree analysis to generate propensity score weights. J Eval Clin Pract, 2017, 23(4): 703-712.
21.	McCaffrey DF, Ridgeway G, Morral AR. Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychol Methods, 2004, 9(4): 403-425.
22.	Pirracchio R, Petersen ML, van der Laan M. Improving propensity score estimators' robustness to model misspecification using super learner. Am J Epidemiol, 2015, 181(2): 108-119.
23.	Setoguchi S, Schneeweiss S, Brookhart MA, <italic>et al</italic>. Evaluating uses of data mining techniques in propensity score estimation: a simulation study. Pharmacoepidemiol Drug Saf, 2008, 17(6): 546-555.
24.	Leacy FP, Stuart EA. On the joint use of propensity and prognostic scores in estimation of the average treatment effect on the treated: a simulation study. Stat Med, 2014, 33(20): 3488-3508.
25.	Wyss R, Ellis AR, Brookhart MA, <italic>et al</italic>. The role of prediction modeling in propensity score estimation: an evaluation of logistic regression, bCART, and the covariate-balancing propensity score. Am J Epidemiol, 2014, 180(6): 645-655.
26.	杨文姣,肖俊玲,丁国武. ARIMA模型和BP神经网络模型在甘肃省结核病发病率预测中的应用. 中华疾病控制杂志, 2019, 23(6): 728-732.
27.	王雅文, 沈忠周, 严宝湖, 等. ARIMA模型和ARIMA-GRNN模型在AIDS发病预测中的应用. 中华疾病控制杂志, 2018, 22(12): 1287-1290.
28.	潘丹, 贾龙飞, 曾安, 等. 生成式对抗网络在医学图像处理中的应用. 生物医学工程学杂志, 2018, 35(6): 970-976.
29.	梁蒙蒙, 周涛, 张飞飞, 等. 卷积神经网络及其在医学图像分析中的应用研究. 生物医学工程学杂志, 2018, 35(6): 977-985.
30.	左东奇, 韩霖, 陈科, 等. 基于卷积神经网络提取超声图像甲状腺结节钙化点的研究. 生物医学工程学杂志, 2018, 35(5): 679-687.
31.	Feng P, Zhou XH, Zou QM, <italic>et al</italic>. Generalized propensity score for estimating the average treatment effect of multiple treatments. Stat Med, 2012, 31(7): 681-697.
32.	Ray WA, Murray KT, Hall K, <italic>et al</italic>. Azithromycin and the risk of cardiovascular death. N Engl J Med, 2012, 366(20): 1881-1890.
33.	Meghea CI, Raffo JE, Zhu Q, <italic>et al</italic>. Medicaid home visitation and maternal and infant healthcare ultilization. Am J Prev Med, 2013, 56(4): 441-447.
34.	Lee BK, Lessler J, Stuart EA. Improving propensity score weighting using machine learning. Stat Med, 2010, 29(3): 337-346.
35.	Tsiatis AA, Davidian M. Comment: Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Stat Sci, 2007, 22(4): 569-573.
36.	Wyss R, Ellis AR, Lunt M, <italic>et al</italic>. Model misspecification when excluding instrumental variables from PS models in settings where instruments modify the effects of covariates on treatment. Epidemiol Methods, 2014, 3(1): 83-96.
37.	周洁, 张晟, 何书, 等. 使用R和Stata软件实现倾向性评分匹配. 中国卫生统计, 2018, 35(4): 628-632+636.
38.	Lefebvre G, Delaney JA, Platt RW. Impact of mis-specification of the treatment model on estimates from a marginal structural model. Stat Med, 2008, 27(18): 3629-3642.

1. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika, 1983, 70(1): 41-55.
2. Austin PC. The performance of different propensity score methods for estimating marginal odds ratios. Stat Med, 2007, 26(16): 3078-3094.
3. Austin PC. The performance of different propensity-score methods for estimating differences in proportions (risk differences or absolute risk reductions) in observational studies. Stat Med, 2010, 29(20): 2137-2148.
4. Austin PC. A critical appraisal of propensity-score matching in the medical literature between 1996 and 2003. Stat Med, 2008, 27(12): 2037-2049.
5. 王永吉, 蔡宏伟, 夏结来, 等. 倾向指数第一讲倾向指数的基本概念和研究步骤. 中华流行病学杂志, 2010, 31(3): 347-348.
6. Abdia Y, Kulasekera KB, Datta S, <italic>et al</italic>. Propensity scores based methods for estimating average treatment effect and average treatment effect among treated: A comparative study. Biom J, 2017, 59(5): 967-985.
7. Pham CT, Gibb CL, Mittinty MN, <italic>et al</italic>. A comparison of propensity score-based approaches to health service evaluation: a case study of a preoperative physician-led clinic for high-risk surgical patients. J Eval Clin Pract, 2016, 22(5): 761-770.
8. Wu M, Zhao Y, Wang R, <italic>et al</italic>. Epidemiology of Functional abdominal bloating and its impact on health related quality of life: male-female stratified propensity score analysis in a population based survey in mainland China. PLoS One, 2014, 9(7): e102320.
9. Kim CY, Collier CD, Liu RW, <italic>et al</italic>. Are limb-sparing surgical resections comparable to amputation for patients with pelvic chondrosarcoma? A case-control, propensity score-matched analysis of the national cancer database. Clin Orthop Relat Res, 2019, 477(3): 596-605.
10. Spertus JV, Normand ST. Bayesian propensity scores for high-dimensional causal inference: a comparison of drug-eluting to bare-metal coronary stents. Biom J, 2018, 60(4): 721-733.
11. Brookhart MA, Schneeweiss S, Rothman KJ, <italic>et al</italic>. Variable selection for propensity score models. Am J Epidemiol, 2006, 163(12): 1149-1156.
12. Westreich D, Lessler J, Funk MJ. Propensity score estimation: neural networks, support vector machines, decision trees (CART), and meta-classifiers as alternatives to logistic regression. J Clin Epidemiol, 2010, 63(8): 826-833.
13. Thoemmes FJ, Kim ES. A systematic review of propensity score methods in the social sciences. Multivariate Behav Res, 2011, 46(1): 90-118.
14. Weitzen S, Lapane KL, Toledano AY, <italic>et al</italic>. Principles for modeling propensity scores in medical research: a systematic literature review. Pharmacoepidemiol Drug Saf, 2004, 13(12): 841-853.
15. 吴美京, 吴骋, 王睿, 等. 倾向性评分法中评分值的估计方法及比较. 中国卫生统计, 2013, 30(3): 440-444.
16. Ellis AR, Dusetzina SB, Hansen RA, <italic>et al</italic>. Confounding control in a non-experimental study of STAR*D data: logistic regression balanced covariates better than boosted CART. Ann Epidemiol, 2013, 23(4): 204-209.
17. Wyss R, Schneeweiss S, van der Laan M, <italic>et al</italic>. Using super learner prediction modeling to improve high-dimensional propensity score estimation. Epidemiology, 2018, 29(1): 96-106.
18. McCaffrey DF, Griffin BA, Almirall D, <italic>et al</italic>. A tutorial on propensity score estimation for multiple treatments using generalized boosted models. Stat Med, 2013, 32(19): 3388-3414.
19. McCandless LC, Gustafson P, Austin PC. Bayesian propensity score analysis for observational data. Stat Med, 2009, 28(1): 94-112.
20. Linden A, Yarnold PR. Using classification tree analysis to generate propensity score weights. J Eval Clin Pract, 2017, 23(4): 703-712.
21. McCaffrey DF, Ridgeway G, Morral AR. Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychol Methods, 2004, 9(4): 403-425.
22. Pirracchio R, Petersen ML, van der Laan M. Improving propensity score estimators' robustness to model misspecification using super learner. Am J Epidemiol, 2015, 181(2): 108-119.
23. Setoguchi S, Schneeweiss S, Brookhart MA, <italic>et al</italic>. Evaluating uses of data mining techniques in propensity score estimation: a simulation study. Pharmacoepidemiol Drug Saf, 2008, 17(6): 546-555.
24. Leacy FP, Stuart EA. On the joint use of propensity and prognostic scores in estimation of the average treatment effect on the treated: a simulation study. Stat Med, 2014, 33(20): 3488-3508.
25. Wyss R, Ellis AR, Brookhart MA, <italic>et al</italic>. The role of prediction modeling in propensity score estimation: an evaluation of logistic regression, bCART, and the covariate-balancing propensity score. Am J Epidemiol, 2014, 180(6): 645-655.
26. 杨文姣,肖俊玲,丁国武. ARIMA模型和BP神经网络模型在甘肃省结核病发病率预测中的应用. 中华疾病控制杂志, 2019, 23(6): 728-732.
27. 王雅文, 沈忠周, 严宝湖, 等. ARIMA模型和ARIMA-GRNN模型在AIDS发病预测中的应用. 中华疾病控制杂志, 2018, 22(12): 1287-1290.
28. 潘丹, 贾龙飞, 曾安, 等. 生成式对抗网络在医学图像处理中的应用. 生物医学工程学杂志, 2018, 35(6): 970-976.
29. 梁蒙蒙, 周涛, 张飞飞, 等. 卷积神经网络及其在医学图像分析中的应用研究. 生物医学工程学杂志, 2018, 35(6): 977-985.
30. 左东奇, 韩霖, 陈科, 等. 基于卷积神经网络提取超声图像甲状腺结节钙化点的研究. 生物医学工程学杂志, 2018, 35(5): 679-687.
31. Feng P, Zhou XH, Zou QM, <italic>et al</italic>. Generalized propensity score for estimating the average treatment effect of multiple treatments. Stat Med, 2012, 31(7): 681-697.
32. Ray WA, Murray KT, Hall K, <italic>et al</italic>. Azithromycin and the risk of cardiovascular death. N Engl J Med, 2012, 366(20): 1881-1890.
33. Meghea CI, Raffo JE, Zhu Q, <italic>et al</italic>. Medicaid home visitation and maternal and infant healthcare ultilization. Am J Prev Med, 2013, 56(4): 441-447.
34. Lee BK, Lessler J, Stuart EA. Improving propensity score weighting using machine learning. Stat Med, 2010, 29(3): 337-346.
35. Tsiatis AA, Davidian M. Comment: Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Stat Sci, 2007, 22(4): 569-573.
36. Wyss R, Ellis AR, Lunt M, <italic>et al</italic>. Model misspecification when excluding instrumental variables from PS models in settings where instruments modify the effects of covariates on treatment. Epidemiol Methods, 2014, 3(1): 83-96.
37. 周洁, 张晟, 何书, 等. 使用R和Stata软件实现倾向性评分匹配. 中国卫生统计, 2018, 35(4): 628-632+636.
38. Lefebvre G, Delaney JA, Platt RW. Impact of mis-specification of the treatment model on estimates from a marginal structural model. Stat Med, 2008, 27(18): 3629-3642.

Chinese Journal of Evidence-Based Medicine

Evaluating the performance of neural networks in propensity score estimation

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content