Multi-scale feature extraction and classification of motor imagery electroencephalography based on time series data enhancement_Journal of Biomedical Engineering

Authors：

 LI Hongli ¹ , LIU Haoyu ¹ , CHEN Hongyu ¹ , ZHANG Ronghua ²

1. School of Control Science and Engineering, Tiangong University, Tianjin 300387, P. R. China;
2. School of Artificial Intelligence, Tiangong University, Tianjin 300387, P. R. China;

Corresponding author：

LI Hongli, Email: lihongliln@163.com

Keywords：

Brain-computer interface; Motor imagery; Attention mechanism; Time series data augmentation; Deep learning

DOI：

10.7507/1001-5515.202205069

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

The brain-computer interface (BCI) based on motor imagery electroencephalography (MI-EEG) enables direct information interaction between the human brain and external devices. In this paper, a multi-scale EEG feature extraction convolutional neural network model based on time series data enhancement is proposed for decoding MI-EEG signals. First, an EEG signals augmentation method was proposed that could increase the information content of training samples without changing the length of the time series, while retaining its original features completely. Then, multiple holistic and detailed features of the EEG data were adaptively extracted by multi-scale convolution module, and the features were fused and filtered by parallel residual module and channel attention. Finally, classification results were output by a fully connected network. The application experimental results on the BCI Competition IV 2a and 2b datasets showed that the proposed model achieved an average classification accuracy of 91.87% and 87.85% for the motor imagery task, respectively, which had high accuracy and strong robustness compared with existing baseline models. The proposed model does not require complex signals pre-processing operations and has the advantage of multi-scale feature extraction, which has high practical application value.

Citation： LI Hongli, LIU Haoyu, CHEN Hongyu, ZHANG Ronghua. Multi-scale feature extraction and classification of motor imagery electroencephalography based on time series data enhancement. Journal of Biomedical Engineering, 2023, 40(3): 418-425. doi: 10.7507/1001-5515.202205069 Copy

1.	Värbu K, Muhammad N, Muhammad Y. Past, present, and future of EEG-based BCI applications. Sensors, 2022, 22(9): 3331.
2.	Abiri R, Borhani S, Sellers E W, et al. A comprehensive review of EEG-based brain-computer interface paradigms. J Neural Eng, 2019, 16(1): 011001.
3.	胡莹, 刘燕, 程晨晨, 等. 基于自适应时频共空间模式结合卷积神经网络的多任务运动想象脑电分类. 生物医学工程学杂志, 2022, 39(6): 1065-1073, 1081.
4.	Kai K A, Zhang Y C, Zhang H, et al. Filter bank common spatial pattern (FBCSP) in brain-computer interface// IEEE International Joint Conference on Neural Networks (IJSCNN). Hong Kong: IEEE, 2008: 2390-2397.
5.	Khademi Z, Ebrahimi F, Kordy H M. A review of critical challenges in MI-BCI: from conventional to deep learning methods. J Neurosci Methods, 2022, 383: 109736.
6.	Han Y, Wang B, Luo J, et al. A classification method for EEG motor imagery signals based on parallel convolutional neural network. Biomed Signal Process Control, 2022, 71: 103190.
7.	Amin S U, Alsulaiman M, Muhammad G, et al. Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion. Future Gener Comput Syst, 2019, 101: 542-554.
8.	Dai G, Zhou J, Huang J, et al. HS-CNN: a CNN with hybrid convolution scale for EEG motor imagery classification. J Neural Eng, 2020, 17(1): 016025.
9.	Altuwaijri G A, Muhammad G, Altaheri H, et al. A multi-branch convolutional neural network with squeeze-and-excitation attention blocks for EEG-based motor imagery signals classification. Diagnostics, 2022, 12(4): 995.
10.	Li D, Xu J, Wang J, et al. A multi-scale fusion convolutional neural network based on attention mechanism for the visualization analysis of EEG signals decoding. IEEE Trans Neural Syst Rehabil Eng, 2020, 28(12): 2615-2626.
11.	Wu H, Niu Y, Li F, et al. A parallel multiscale filter bank convolutional neural networks for motor imagery EEG classification. Front Neurosci, 2019, 13: 1275.
12.	Fan C, Yang H, Hou Z, et al. Bilinear neural network with 3-D attention for brain decoding of motor imagery movements from the human EEG. Cogn Neurodyn, 2021, 15: 181-189.
13.	何群, 邵丹丹, 王煜文, 等. 基于多特征卷积神经网路的运动想象脑电信号分析及意图识别. 仪器仪表学报, 2022, 41(1): 138-146.
14.	Wang H, Yu H, Wang H. EEG_GENet: A feature-level graph embedding method for motor imagery classification based on EEG signals. Biocybernet Biomed Eng, 2022, 42(3): 1023-1040.
15.	Yang J, Liu L, Yu H, et al. Multi-hierarchical fusion to capture the latent invariance for calibration-free brain-computer interfaces. Front Neurosci, 2022, 16: 304.
16.	Jia Z, Lin Y, Wang J, et al. MMCNN: a multi-branch multi-scale convolutional neural network for motor imagery classification// Hutter F, Kersting K, Lijffijt J, et al. Machine Learning and Knowledge Discovery in Databases (ECML PKDD). Ghent: Springer, 2020: 736-751.
17.	Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions// 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston: IEEE, 2015: 1-9.
18.	Szegedy C, Ioffe S, Vanhoucke V, et al. Inception-v4, Inception-ResNet and the impact of residual connections on learning// Thirty-First AAAI Conference on Artificial Intelligence. San Francisco: AAAI, 2017: 4278-4284.
19.	Tang X, Li W, Li X, et al. Motor imagery EEG recognition based on conditional optimization empirical mode decomposition and multi-scale convolutional neural network. Expert Syst Appl, 2020, 149: 113285.
20.	He K, Zhang X, Ren S, et al. Deep residual learning for image recognition// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE, 2016: 770-778.
21.	Hu J, Shen L, Albanie S, et al. Squeeze-and-Excitation networks. IEEE Trans Pattern Anal Mach Intell, 2020, 42(8): 2011-2023.
22.	Majidov I, Whangbo T. Efficient classification of motor imagery electroencephalography signals using deep learning methods. Sensors, 2019, 19(7): 1736.
23.	Xu M, Yao J, Zhang Z, et al. Learning EEG topographical representation for classification via convolutional neural network. Pattern Recogn, 2020, 105(4): 107390.
24.	Raza H, Cecotti H, Li Y, et al. Adaptive learning with covariate shift-detection for motor imagery-based brain-computer interface. Soft Comput, 2016, 20(8): 3085-3096.
25.	Zhang Y, Nam C S, Zhou G, et al. Temporally constrained sparse group spatial patterns for motor imagery BCI. IEEE Trans Cybern, 2019, 49(9): 3322-3332.
26.	Gaur P, Pachori R B, Wang H, et al. A multi-class EEG-based BCI classification using multivariate empirical mode decomposition based filtering and Riemannian geometry. Expert Syst Appl, 2018, 95: 201-211.
27.	Zhang C, Kim Y K, Eskandarian A. EEG-inception: an accurate and robust end-to-end neural network for EEG-based motor imagery classification. J Neural Eng, 2021, 18(4): 046014.
28.	Lawhern V J, Solon A J, Waytowich N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J Neural Eng, 2018, 15(5): 056013.
29.	Li M, Zhu W, Liu H, et al. Adaptive feature extraction of motor imagery EEG with optimal wavelet packets and SE-isomap. Appl Sci, 2017, 7(4): 390.
30.	Zheng Q, Zhu F, Heng P. Robust support matrix machine for single trial EEG classification. IEEE Trans Neural Syst Rehabil Eng, 2018, 26(3): 551-562.

1. Värbu K, Muhammad N, Muhammad Y. Past, present, and future of EEG-based BCI applications. Sensors, 2022, 22(9): 3331.
2. Abiri R, Borhani S, Sellers E W, et al. A comprehensive review of EEG-based brain-computer interface paradigms. J Neural Eng, 2019, 16(1): 011001.
3. 胡莹, 刘燕, 程晨晨, 等. 基于自适应时频共空间模式结合卷积神经网络的多任务运动想象脑电分类. 生物医学工程学杂志, 2022, 39(6): 1065-1073, 1081.
4. Kai K A, Zhang Y C, Zhang H, et al. Filter bank common spatial pattern (FBCSP) in brain-computer interface// IEEE International Joint Conference on Neural Networks (IJSCNN). Hong Kong: IEEE, 2008: 2390-2397.
5. Khademi Z, Ebrahimi F, Kordy H M. A review of critical challenges in MI-BCI: from conventional to deep learning methods. J Neurosci Methods, 2022, 383: 109736.
6. Han Y, Wang B, Luo J, et al. A classification method for EEG motor imagery signals based on parallel convolutional neural network. Biomed Signal Process Control, 2022, 71: 103190.
7. Amin S U, Alsulaiman M, Muhammad G, et al. Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion. Future Gener Comput Syst, 2019, 101: 542-554.
8. Dai G, Zhou J, Huang J, et al. HS-CNN: a CNN with hybrid convolution scale for EEG motor imagery classification. J Neural Eng, 2020, 17(1): 016025.
9. Altuwaijri G A, Muhammad G, Altaheri H, et al. A multi-branch convolutional neural network with squeeze-and-excitation attention blocks for EEG-based motor imagery signals classification. Diagnostics, 2022, 12(4): 995.
10. Li D, Xu J, Wang J, et al. A multi-scale fusion convolutional neural network based on attention mechanism for the visualization analysis of EEG signals decoding. IEEE Trans Neural Syst Rehabil Eng, 2020, 28(12): 2615-2626.
11. Wu H, Niu Y, Li F, et al. A parallel multiscale filter bank convolutional neural networks for motor imagery EEG classification. Front Neurosci, 2019, 13: 1275.
12. Fan C, Yang H, Hou Z, et al. Bilinear neural network with 3-D attention for brain decoding of motor imagery movements from the human EEG. Cogn Neurodyn, 2021, 15: 181-189.
13. 何群, 邵丹丹, 王煜文, 等. 基于多特征卷积神经网路的运动想象脑电信号分析及意图识别. 仪器仪表学报, 2022, 41(1): 138-146.
14. Wang H, Yu H, Wang H. EEG_GENet: A feature-level graph embedding method for motor imagery classification based on EEG signals. Biocybernet Biomed Eng, 2022, 42(3): 1023-1040.
15. Yang J, Liu L, Yu H, et al. Multi-hierarchical fusion to capture the latent invariance for calibration-free brain-computer interfaces. Front Neurosci, 2022, 16: 304.
16. Jia Z, Lin Y, Wang J, et al. MMCNN: a multi-branch multi-scale convolutional neural network for motor imagery classification// Hutter F, Kersting K, Lijffijt J, et al. Machine Learning and Knowledge Discovery in Databases (ECML PKDD). Ghent: Springer, 2020: 736-751.
17. Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions// 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston: IEEE, 2015: 1-9.
18. Szegedy C, Ioffe S, Vanhoucke V, et al. Inception-v4, Inception-ResNet and the impact of residual connections on learning// Thirty-First AAAI Conference on Artificial Intelligence. San Francisco: AAAI, 2017: 4278-4284.
19. Tang X, Li W, Li X, et al. Motor imagery EEG recognition based on conditional optimization empirical mode decomposition and multi-scale convolutional neural network. Expert Syst Appl, 2020, 149: 113285.
20. He K, Zhang X, Ren S, et al. Deep residual learning for image recognition// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE, 2016: 770-778.
21. Hu J, Shen L, Albanie S, et al. Squeeze-and-Excitation networks. IEEE Trans Pattern Anal Mach Intell, 2020, 42(8): 2011-2023.
22. Majidov I, Whangbo T. Efficient classification of motor imagery electroencephalography signals using deep learning methods. Sensors, 2019, 19(7): 1736.
23. Xu M, Yao J, Zhang Z, et al. Learning EEG topographical representation for classification via convolutional neural network. Pattern Recogn, 2020, 105(4): 107390.
24. Raza H, Cecotti H, Li Y, et al. Adaptive learning with covariate shift-detection for motor imagery-based brain-computer interface. Soft Comput, 2016, 20(8): 3085-3096.
25. Zhang Y, Nam C S, Zhou G, et al. Temporally constrained sparse group spatial patterns for motor imagery BCI. IEEE Trans Cybern, 2019, 49(9): 3322-3332.
26. Gaur P, Pachori R B, Wang H, et al. A multi-class EEG-based BCI classification using multivariate empirical mode decomposition based filtering and Riemannian geometry. Expert Syst Appl, 2018, 95: 201-211.
27. Zhang C, Kim Y K, Eskandarian A. EEG-inception: an accurate and robust end-to-end neural network for EEG-based motor imagery classification. J Neural Eng, 2021, 18(4): 046014.
28. Lawhern V J, Solon A J, Waytowich N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J Neural Eng, 2018, 15(5): 056013.
29. Li M, Zhu W, Liu H, et al. Adaptive feature extraction of motor imagery EEG with optimal wavelet packets and SE-isomap. Appl Sci, 2017, 7(4): 390.
30. Zheng Q, Zhu F, Heng P. Robust support matrix machine for single trial EEG classification. IEEE Trans Neural Syst Rehabil Eng, 2018, 26(3): 551-562.

Journal of Biomedical Engineering

Multi-scale feature extraction and classification of motor imagery electroencephalography based on time series data enhancement

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content