Artistry meets algorithm: Exploring the multimodality of artificial intelligence in interaction design prototyping

Artificial intelligence (AI)-driven services for design and prototyping are increasingly used in professional design environments. Although existing studies have often focused on isolated aspects of AI integration into design workflows, a comprehensive understanding of how this technology can transform the prototyping process remains limited. In this study, we examine this relatively unexplored space through the crafting of the Artistry Design Studio. This studio reimagines the design process through multimodal interfaces and conversational agents. By integrating generative AI and voice-activated agents, we enable a fluid dialogue between designers and AI. We contribute with (1) the Artistry Design Studio, a theoretically and empirically informed design embodiment inspired by generative AI, conversational agents, and multimodal interactions; (2) a series of design explorations analyzed through the concept-driven interaction design methodology and manifested into two design concepts; and (3) reflections on the complexities of AI-driven prototyping, from issues of design agency to opportunities with human-centered AI. Together, these contributions illustrate new possibilities for how multimodal and generative AI can support creative collaboration and prototyping in interaction design.