An Air-Written Real-Time Multilingual Numeral String Recognition System Using Deep Convolutional Neural Networks

Abstract

Air-writing is a modern practice for free-space writing of characters or words with hand or finger movements. In the new era, hand gestures are commonly used for Human-Computer Interaction (HCI) or controlling machines in several applications. Writing with a finger or holding any object in hand and 3-D movement in the air is helpful for several applications. However, it is hard to build simulated environments due to the complex structure of the hand and regulate the joints. Palm detection is another way to reduce the complexity of hand detection. Deep learning-based techniques enhance object detection tasks with excellent performance in this task. Convolutional Neural Networks (CNN) are the powerful frameworks used in detecting palms facing several challenges in this task. Therefore, we propose a novel approach using hand landmark detection to identify palms. Our proposed CNN-based air-writing recognition model is intended to detect and recognize numbers in 2 different languages. We present the CNN-based 2-D model to detect air-writing in real-time video. The proposed models were implemented on our developed datasets for two languages, Devanagari and English, with 99.55% accuracy. Our contribution is focused on the air-writing interface using a fingertip to write the numbers instead of traditional input devices. In addition, different air-writing gestures are introduced to control the writing activity. Therefore, our proposed model integrates detection, recognition, and control of air-writing activity. Hence, we achieved precise detection and management of palm movements and accurate recognition of air-written numbers. The results are promising and valuable for real-time multilingual numeral string recognition.