HARMONY: A Framework for Multimodal LLM-Powered AI Agents in Smart Homes via the Model Context Protocol

Multimodal Large Language Models (MLLMs) represent a significant leap in AI capabilities by enabling simultaneous interpretation and generation of visual, audio, and textual inputs to produce more comprehensive, context-aware interactions. Smart home development has the ability to deploy ambient intelligence to enable devices to anticipate and respond to users’ behaviors via consolidated sensing and control infrastructures. The existing MLLM systems are challenged by fragmented ecosystems, restricted inputs, and reactive intelligence. To address these challenges, this paper introduces a novel framework Home Autonomous Resilient Multimodal Operations Nexus for You (HARMONY) that combines multimodal generative Artificial Intelligence (AI) with standardized communication protocols for smart home ecosystems. HARMONY leverages a central MLLM to process multimodal inputs and coordinate specialized AI agents within the Samsung Smart Things platform. These agents query device states and execute contextually appropriate actions related to comfort, security, and culinary assistance via the Model Context Protocol (MCP). This architecture enables intuitive multimodal control and proactive user intent recognition. The study demonstrates a significant evolution in Large Language Models (LLM) orchestration capabilities from basic single-task models to context-reflective agents enhancing personalization, operational efficiency, and decision-making while highlighting the trade-offs between execution speed, semantic reasoning, and agentic flexibility.