Medical image segmentation method based on self-attention and multi-view attention_Journal of Biomedical Engineering

Authors：

 PAN Dan ¹ , LYU Jin ¹ , ZENG An ²

1. School of Electronics and Information Engineering, Guangdong University of Technology and Education, Guangzhou 510450, P. R.China;
2. School of Computers, Guangdong University of Technology, Guangzhou 510006, P. R.China;

Corresponding author：

PAN Dan, Email: pandan@gpnu.edu.cn

Keywords：

Medical image segmentation; Convolutional neural network; Self-attention; Multi-view attention; Transformer

DOI：

10.7507/1001-5515.202408007

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Most current medical image segmentation models are primarily built upon the U-shaped network (U-Net) architecture, which has certain limitations in capturing both global contextual information and fine-grained details. To address this issue, this paper proposes a novel U-shaped network model, termed the Multi-View U-Net (MUNet), which integrates self-attention and multi-view attention mechanisms. Specifically, a newly designed multi-view attention module is introduced to aggregate semantic features from different perspectives, thereby enhancing the representation of fine details in images. Additionally, the MUNet model leverages a self-attention encoding block to extract global image features, and by fusing global and local features, it improves segmentation performance. Experimental results demonstrate that the proposed model achieves superior segmentation performance in coronary artery image segmentation tasks, significantly outperforming existing models. By incorporating self-attention and multi-view attention mechanisms, this study provides a novel and efficient modeling approach for medical image segmentation, contributing to the advancement of intelligent medical image analysis.

1.	Cacciatore S, Spadafora L, Bernardi M, et al. Management of coronary artery disease in older adults: recent advances and gaps in evidence. J Clin Med, 2023, 12(16): 5233.
2.	Min J K, Shaw L J, Berman D S. The present state of coronary computed tomography angiography: a process in evolution. J Am Coll Cardiol, 2010, 55(10): 957-965.
3.	Dai J, Li Y, He K, et al. R-FCN: object detection via region-based fully convolutional networks. Advances in Neural Information Processing Systems, 2016, 29: 5-10.
4.	Siddique N, Paheding S, Elkin C P, et al. U-Net and its variants for medical image segmentation: a review of theory and applications. IEEE Access, 2021, 9: 82031-82057.
5.	Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention (MICCAI 2015) , 2015: 234-241.
6.	Chen L C, Papandreou G, Kokkinos I, et al. Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFS. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(4): 834-848.
7.	Lin G, Milan A, Shen C, et al. RefineNet: multi-path refinement networks for high-resolution semantic segmentation//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1925-1934.
8.	Guo C, Szemenyei M, Yi Y, et al. SA-UNet: spatial attention U-Net for retinal vessel segmentation//International Conference on Pattern Recognition (ICPR) , 2021: 1236-1242.
9.	Jin Q, Meng Z, Sun C, et al. RA-UNet: a hybrid deep attention-aware network to extract liver and tumor in CT scans. Frontiers in Bioengineering and Biotechnology, 2020, 8: 605132.
10.	Baheti B, Innani S, Gajre S, et al. Eff-UNet: a novel architecture for semantic segmentation in unstructured environment//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle: IEEE, 2020: 1473-1481.
11.	Han K, Xiao A, Wu E, et al. Transformer in transformer. Advances in Neural Information Processing Systems, 2021, 34: 15908-15919.
12.	Chen J, Lu Y, Yu Q, et al. TransUNet: transformers make strong encoders for medical image segmentation. arXiv preprint, 2021, arXiv: 2102. 04306.
13.	Cao H, Wang Y, Chen J, et al. Swin-Unet: Unet-like pure transformer for medical image segmentation. Computer Vision-ECCV 2022 Workshops (ECCV 2022), Springer, 2023, 13803: 205-218.
14.	Huang X, Deng Z, Li D, et al. MISSFormer: an effective medical image segmentation transformer. arXiv preprint, 2021, arXiv: 2109. 07162.
15.	Zhao H, Jia J, Koltun V. Exploring self-attention for image recognition//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 10076-10085.
16.	Zhang H, Goodfellow I, Metaxas D, et al. Self-attention generative adversarial networks//International Conference on Machine Learning, 2019: 7354-7363.
17.	Wang X, Girshick R, Gupta A, et al. Non-local neural networks//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 7794-7803.
18.	Devlin J, Chang M W, Lee K, et al. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint, 2018, arXiv, 1810. 04805.
19.	Gaál G, Maga B, Lukács A. Attention U-Net based adversarial architectures for chest x-ray lung segmentation. arXiv preprint, 2020, arXiv: 2003. 10304.
20.	Yin H, Vahdat A, Alvarez J M, et al. A-vit: Adaptive tokens for efficient vision transformer// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 10809-10818.
21.	Lin D, Li Y, Cheng Y, et al. Multi-view 3D object retrieval leveraging the aggregation of view and instance attentive features. Knowledge-Based Systems, 2022, 247: 108754.
22.	Song R, Zhang W, Zhao Y, et al. Unsupervised multi-view CNN for salient view selection and 3D interest point detection. International Journal of Computer Vision, 2022, 130(5): 1210-1227.
23.	Sun K, Zhang J, Liu J, et al. DRCNN: dynamic routing convolutional neural network for multi-view 3d object recognition. IEEE Transactions on Image Processing (TIP), 2021, 30: 868-877.
24.	Zheng D, Zheng X, Yang L T, et al. MFFN: multi-view feature fusion network for camouflaged object detection//2023 IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa: IEEE, 2023: 6221-6231.
25.	Zhu Q, Wang Y, Chu X, et al. Multi-view coupled self-attention network for pulmonary nodules classification//Proceedings of the Asian Conference on Computer Vision. 2022, 1384, 6: 37-51.
26.	Kingma D P, Ba J. Adam: a method for stochastic optimization. arXiv preprint, 2014, arXiv: 1412. 6980.
27.	Zeng A, Wu C, Lin G, et al. ImageCAS: a large-scale dataset and benchmark for coronary artery segmentation based on computed tomography angiography images. Computerized Medical Imaging and Graphics, 2023, 109: 102287.
28.	Hatamizadeh A, Tang Y, Nath V, et al. UNETR: transformers for 3D medical image segmentation//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022: 574-584.
29.	Lee H H, Bao S, Huo Y, et al. 3D UX-Net: a large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation. arXiv preprint, 2022, arXiv: 2209. 15076.
30.	Zhao C, Xiang S, Wang Y, et al. Context-aware network fusing transformer and V-Net for semi-supervised segmentation of 3D left atrium. Expert Systems with Applications, 2023, 214: 119105.
31.	Dong C, Xu S, Dai D, et al. A novel multi-attention, multi-scale 3D deep network for coronary artery segmentation. Medical Image Analysis, 2023, 85: 102745.

1. Cacciatore S, Spadafora L, Bernardi M, et al. Management of coronary artery disease in older adults: recent advances and gaps in evidence. J Clin Med, 2023, 12(16): 5233.
2. Min J K, Shaw L J, Berman D S. The present state of coronary computed tomography angiography: a process in evolution. J Am Coll Cardiol, 2010, 55(10): 957-965.
3. Dai J, Li Y, He K, et al. R-FCN: object detection via region-based fully convolutional networks. Advances in Neural Information Processing Systems, 2016, 29: 5-10.
4. Siddique N, Paheding S, Elkin C P, et al. U-Net and its variants for medical image segmentation: a review of theory and applications. IEEE Access, 2021, 9: 82031-82057.
5. Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention (MICCAI 2015) , 2015: 234-241.
6. Chen L C, Papandreou G, Kokkinos I, et al. Deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFS. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(4): 834-848.
7. Lin G, Milan A, Shen C, et al. RefineNet: multi-path refinement networks for high-resolution semantic segmentation//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1925-1934.
8. Guo C, Szemenyei M, Yi Y, et al. SA-UNet: spatial attention U-Net for retinal vessel segmentation//International Conference on Pattern Recognition (ICPR) , 2021: 1236-1242.
9. Jin Q, Meng Z, Sun C, et al. RA-UNet: a hybrid deep attention-aware network to extract liver and tumor in CT scans. Frontiers in Bioengineering and Biotechnology, 2020, 8: 605132.
10. Baheti B, Innani S, Gajre S, et al. Eff-UNet: a novel architecture for semantic segmentation in unstructured environment//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle: IEEE, 2020: 1473-1481.
11. Han K, Xiao A, Wu E, et al. Transformer in transformer. Advances in Neural Information Processing Systems, 2021, 34: 15908-15919.
12. Chen J, Lu Y, Yu Q, et al. TransUNet: transformers make strong encoders for medical image segmentation. arXiv preprint, 2021, arXiv: 2102. 04306.
13. Cao H, Wang Y, Chen J, et al. Swin-Unet: Unet-like pure transformer for medical image segmentation. Computer Vision-ECCV 2022 Workshops (ECCV 2022), Springer, 2023, 13803: 205-218.
14. Huang X, Deng Z, Li D, et al. MISSFormer: an effective medical image segmentation transformer. arXiv preprint, 2021, arXiv: 2109. 07162.
15. Zhao H, Jia J, Koltun V. Exploring self-attention for image recognition//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 10076-10085.
16. Zhang H, Goodfellow I, Metaxas D, et al. Self-attention generative adversarial networks//International Conference on Machine Learning, 2019: 7354-7363.
17. Wang X, Girshick R, Gupta A, et al. Non-local neural networks//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 7794-7803.
18. Devlin J, Chang M W, Lee K, et al. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint, 2018, arXiv, 1810. 04805.
19. Gaál G, Maga B, Lukács A. Attention U-Net based adversarial architectures for chest x-ray lung segmentation. arXiv preprint, 2020, arXiv: 2003. 10304.
20. Yin H, Vahdat A, Alvarez J M, et al. A-vit: Adaptive tokens for efficient vision transformer// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 10809-10818.
21. Lin D, Li Y, Cheng Y, et al. Multi-view 3D object retrieval leveraging the aggregation of view and instance attentive features. Knowledge-Based Systems, 2022, 247: 108754.
22. Song R, Zhang W, Zhao Y, et al. Unsupervised multi-view CNN for salient view selection and 3D interest point detection. International Journal of Computer Vision, 2022, 130(5): 1210-1227.
23. Sun K, Zhang J, Liu J, et al. DRCNN: dynamic routing convolutional neural network for multi-view 3d object recognition. IEEE Transactions on Image Processing (TIP), 2021, 30: 868-877.
24. Zheng D, Zheng X, Yang L T, et al. MFFN: multi-view feature fusion network for camouflaged object detection//2023 IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa: IEEE, 2023: 6221-6231.
25. Zhu Q, Wang Y, Chu X, et al. Multi-view coupled self-attention network for pulmonary nodules classification//Proceedings of the Asian Conference on Computer Vision. 2022, 1384, 6: 37-51.
26. Kingma D P, Ba J. Adam: a method for stochastic optimization. arXiv preprint, 2014, arXiv: 1412. 6980.
27. Zeng A, Wu C, Lin G, et al. ImageCAS: a large-scale dataset and benchmark for coronary artery segmentation based on computed tomography angiography images. Computerized Medical Imaging and Graphics, 2023, 109: 102287.
28. Hatamizadeh A, Tang Y, Nath V, et al. UNETR: transformers for 3D medical image segmentation//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022: 574-584.
29. Lee H H, Bao S, Huo Y, et al. 3D UX-Net: a large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation. arXiv preprint, 2022, arXiv: 2209. 15076.
30. Zhao C, Xiang S, Wang Y, et al. Context-aware network fusing transformer and V-Net for semi-supervised segmentation of 3D left atrium. Expert Systems with Applications, 2023, 214: 119105.
31. Dong C, Xu S, Dai D, et al. A novel multi-attention, multi-scale 3D deep network for coronary artery segmentation. Medical Image Analysis, 2023, 85: 102745.

Journal of Biomedical Engineering

Latest ArticlesMedical image segmentation method based on self-attention and multi-view attention

Abstract Full text Figures/Tables Video References Cited by

Format

Content