MB-ConvLSTM: a novel hybrid deep learning model for accurate sign language recognition

What the paper says

Purpose The purpose of this paper is to introduce a novel deep learning model for translating sign language in the Web-based e learning platforms. The increased utilization of Web-based e-learning systems highlighted the need for an accurate, rapid and highly generalized system for recognizing sign language to facilitate effective communication for deaf/mute diverse learning communities. Design/methodology/approach In this paper, a novel lightweight hybrid deep learning model has been introduced, called a multi-branch convolutional LSTM fusion network (MB-ConvLSTM), that integrates multiscale convolutional branches with long short-term memory (LSTM) units. The model uses modern deep learning methods and optimizes computing efficiency to integrate smoothly with Web-based e-learning systems, therefore, improving accessibility while promoting diversity in virtual classrooms. Findings The presented deep model has a unique property of capturing both spatial and temporal features and provides a highly accurate recognition for sign language in different environmental conditions and positions of variant hand shapes. The sign image will pass through multiple preprocessing stages, and the features will be extracted using two methods (i.e. linear discriminant analysis and gray-level co-occurrence matrix) before the final classification phase. The high generalization and the computational efficiency of the proposed system have been proved by assessing it in recognizing three public data sets of sign language from variant cultures, including American, Arabic and Malaysian. The proposed work outperforms the existing state-of-the-art models in terms of precision (100%), recall (100%) and F1-measure (100%) for American and Malaysian sign language and (99.7%) for Arabic sign language; moreover, the recognition time also decreased, and the lowest time was equal to 732 ms for Malaysian sign language, which makes it ideal for real-world applications. Research limitations/implications The lightweight architecture of the proposed model ensures scalability for deployment on cloud-based Web services, whereas its low latency supports live interactions in multimodal learning environments. The proposed model surpasses current state-of-the-art models in accuracy (100%), precision (100%), recall (100%) and F1-measure (100%) for American and Malaysian data sets, and (99.7%) for Arabic sign language. Furthermore, the inference time was reduced, with the minimum inference time recorded at 732 ms from recognizing the Malaysian sign language, making it suitable for practical applications. Originality/value A novel lightweight hybrid deep learning model has been introduced, called MB-ConvLSTM, that integrates multiscale convolutional branches with LSTM units. The presented deep model is able to capture both spatial and temporal features and provides highly accurate recognition results in different environmental conditions and positions of variant hand shapes. The model uses modern deep learning methods and optimizes computing efficiency to integrate smoothly with Web-based e-learning systems, therefore, improving accessibility while promoting diversity in virtual classrooms.