PenSLR: Persian end-to-end Sign Language Recognition Using Ensembling

Amirparsa Salmankhah, Amirreza Rajabi, Negin Kheirmand, Ali Fadaeimanesh, Amirreza Tarabkhah, Amirreza Kazemzadeh, Hamed Farbeh·June 24, 2024

Summary

The paper presents PenSLR, a state-of-the-art system for Persian Sign Language Recognition (PSL) using an IMU and flexible sensors in a glove. It combines a deep learning framework with CTC loss and introduces Star Alignment, an ensemble technique, to enhance performance. The authors contribute a new PSL dataset with 16 signs and over 3,000 samples, achieving impressive word accuracy (94.58% and 96.70% in subject-independent and dependent setups). Star Alignment significantly improves sentence-level accuracy. The study addresses communication barriers for the deaf community and contributes to the development of wearable-based SLR systems, focusing on overcoming privacy concerns and handling spatial and temporal data. Future work includes expanding the dataset and incorporating non-manual cues for enhanced recognition.

Key findings

7

Tables

2

Introduction
Background
Persian Sign Language (PSL) significance
Current limitations in PSL recognition systems
Objective
Development of PenSLR system
Addressing communication barriers for the deaf community
Key Contributions
New PSL dataset
Star Alignment ensemble technique
Method
Data Collection
Sensor setup: IMU and flexible sensors in a glove
Data collection process
Ethical considerations: Privacy concerns
Data Preprocessing
Data cleaning and filtering
Spatial and temporal data processing
Glove sensor data fusion
Deep Learning Framework
Model architecture
Combination of deep learning layers
CTC Loss function
Model training and evaluation
Star Alignment
Ensemble technique description
Performance enhancement through alignment
Sentence-level accuracy improvement
Experiments and Results
Dataset Description
New PSL dataset: 16 signs, 3,000+ samples
Data splits: Subject-independent and dependent setups
Performance Metrics
Word accuracy: 94.58% and 96.70%
Sentence-level accuracy with Star Alignment
Comparison with State-of-the-Art
Advantages over existing PSL systems
Future Work
Dataset expansion
Incorporating non-manual cues
Privacy-preserving wearable SLR systems
Conclusion
Significance of PenSLR for the deaf community
Potential impact on wearable technology for SLR
References
Cited works in the field of PSL recognition and wearable technology
Basic info
papers
human-computer interaction
artificial intelligence
Advanced features