Air Handwriting Recognition from Non-Positional Wearable Sensors: A Multi-Stage Sequential Modeling Approach

Abstract

Air handwriting enables text input without physical contact, but existing approaches rely mainly on computer vision and IMUs, which perform position-aware trajectory reconstruction. My senior thesis design project compares alternative sensing paradigms for air handwriting recognition using non-positional, wearable sensing modalities, specifically EMGs and piezoresistive bend and stretch sensors to infer handwriting patterns. This project presents a three-stage machine learning pipeline developed to interpret noisy time-series signals, consisting of spatial modeling for classifying directional primitives, sequential modeling for individual letter recognition, and continuous sequence decoding for real-time multi-letter inference. The intention is for the device to be as minimally invasive as possible and avoid restriction of finger movement, while proving the feasibility of bend and stretch sensors or EMGs in creating successful dynamic handwriting detection at finger scale. The project evaluates the efficacy of various model architectures used in conjunction with one another to track the user’s air handwriting. Experimental results demonstrated that despite the limitations of these low-cost nonpositional sensors, appropriate modeling and design can achieve robust, meaningful results. These findings highlight the capacity of different algorithmic approaches in compensating for limited sensing fidelity, and the separability of patterns in small-scale finger and wrist scale movements. Lastly, the project discusses future directions in which the system can be designed to be more robust to more sources of variability.