Automatic Generation of HamNoSys Notation from Static Sign Language

This paper presents a novel system for automatically converting static hand signs to the Hamburg Notation System (HamNoSys) symbols. HamNoSys notation is a standardized notation for sign language transcription; it relies on manual annotation that is laborious and requires expertise. Although machine learning (ML) has been applied to the sign language to text interpretation, the sign language to HamNoSys notation conversion is underexplored. Our methodology utilizes MediaPipe for hand tracking and employs a modular approach consisting of five modules: hand shape, thumb position, finger crossing, thumb in between, and bent finger identification. Geometric features (joint angles and distances) are extracted for training XGBoost and Artificial Neural Network classifiers. Experimental findings indicate that the XGBoost model provides superior classification performance on hand shapes and thumb positions, outperforming ANN, while both models perform similarly for crossing fingers and thumb in between detection. A rule-based strategy using angle thresholds is applied for finger bending states. The system merges all generated outputs to produce a complete HamNoSys notation, validated by finding similarities between the avatar produced via the JASigning tool and the input static hand sign image. This automation cuts down on manual annotation and can be used in sign language education, translation, and assistive technologies. The system's promising performance lays the groundwork for sign language transcription that can be scaled up and run more efficiently.