Multimodal Command System for HumanComputer Interaction

Recent advances in computing have increased the demand for interaction paradigms that enable intuitive and efficient communication between users and digital systems. Traditional interfaces such as keyboards and mice often limit accessibility and natural interaction, particularly in complex or hands-free environments. This paper proposes a multimodal human–computer interaction system that integrates hand gesture recognition and voice command processing to enable seamless desktop control and interaction with external applications. The system employs MediaPipe-based hand landmark extraction and speech-to-text processing, coordinated through an AI agent using the Model Context Protocol (MCP) for task orchestration and service integration. Experimental evaluation demonstrates that the proposed framework achieves high recognition accuracy with low latency, supporting real-time interaction across local and cloud-based services. The results indicate that multimodal fusion combined with agent-based automation enhances usability, responsiveness, and accessibility, positioning the system as a scalable solution for next-generation human–computer interaction.