Deep learning based interactive drawing system using gesture and ASL with multimodal feedback

The Air Canvas system introduces a groundbreaking approach to virtual interaction by enabling users to draw and communicate through hand gestures, utilizing computer vision technology. This innovative platform replaces traditional writing and drawing tools, allowing users to “paint in the air” with their fingers, eliminating the need for physical surfaces or input devices. By leveraging real-time hand landmark detection via MediaPipe, the application accurately tracks complex finger movements to support freehand drawing, shape creation, and text input using American Sign Language (ASL). The system combines an intuitive gesture-based interface with a dynamic virtual drawing environment. Users can interact with a virtual toolbar to change brush size, color, stroke thickness, and toggle between drawing and erasing modes, all through predefined hand gestures. It accommodates both single-finger drawing and multi-finger inputs, enhancing user control and creativity. A key feature of this system is its built-in ASL recognition module, which uses a Support Vector Machine (SVM) classifier trained on hand landmark data to identify static ASL letters. This provides an accessible text input method for users with hearing or speech impairments.