Agents in Concert: A Case-Study of Bringing AI to the Stage in Practice

Recent years have seen a surge in musical performances accompanied by generative agents. Artificial voices, timbres synthesized by neural networks, and agents that mirror or respond to human performers are rapidly taking the stage. In parallel, practitioners in human-computer interaction (HCI) and music technology have called for practice-based research that identifies the most salient affordances of these developments by examining their use in the real-world contexts of music making. To advance practice-based research on human-AI music creation, we present a longitudinal account of two months of codesign with top local jazz musicians, spanning early explorations, the identification of emerging goals, and rehearsals. Our work culminates in a public concert for a live audience of 97, featuring three pieces co-improvised with AI agents. Drawing on systems including VampNet, Somax2, and the jam_bot, each piece was tailored to the stylistic strengths of the performers and the unique strengths and limitations of each system. Through this extensive iterative process, we uncovered a wide range of design interventions, from augmenting GenAI systems with a guitar pedal to situate it in a loop-based creative practice, to enabling musicians to anticipate AI response by visually forecasting its predictions. Where musicians tended to rein in the wilder qualities of the generative systems, some audience members expected a human-AI performance to allow as much agency and spontaneity as possible. In post-concert reflection, musicians also expressed the desire to practice more which in turn could enable them to let the agency of the systems shine. They also encouraged future musicians to lean more into the uncertainty. Together, we see a unique practice emerging through this musician-AI live improv medium.