A Design Space for Live Music Agents

Live music provides a uniquely rich setting for studying creativity and interaction due to its spontaneous nature. The pursuit of live music agents–intelligent systems supporting real-time music performance and interaction–has captivated researchers across HCI, AI, and computer music for decades, and recent advancements in AI suggest unprecedented opportunities to evolve their design. However, the interdisciplinary nature of music has led to fragmented development across research communities, hindering effective communication and collaborative progress. In this work, we bring together perspectives from these diverse fields to map the current landscape of live music agents. Based on our analysis of 184 systems across both academic literature and video, we develop a comprehensive design space that categorizes dimensions spanning usage contexts, interactions, technologies, and ecosystems. By highlighting trends and gaps in live music agents, our design space offers researchers, designers, and musicians a structured lens to understand existing systems and shape future directions in real-time human-AI music co-creation. We release our annotated systems as a living artifact at https://live-music-agents.github.io.

💡 Research Summary

The paper addresses the fragmented state of research on live music agents—intelligent systems that interact with musicians in real time—by constructing a comprehensive design space that unifies perspectives from Human‑Computer Interaction (HCI), Artificial Intelligence (AI), and Computer Music. The authors surveyed 184 systems, drawing from 153 peer‑reviewed papers across major conferences (e.g., CHI, NIPS, ISMIR) and 31 publicly available videos that showcase live performances or demos. This dual‑source approach captures both scholarly contributions and real‑world practice, ensuring a holistic view of the field.

The design space is organized around four high‑level aspects: Usage Context, Interaction, Technology, and Ecosystem. Each aspect is broken down into multiple dimensions, yielding a total of 31 dimensions and 165 specific design codes. Usage Context dimensions include performance scale (solo, ensemble, large venue), collaborator type (human, AI, hybrid), and environment (studio, stage, online). Interaction dimensions cover input modalities (MIDI, audio, gesture, speech, text), control scope (global vs. local), responsiveness (reactive, predictive, proactive), and feedback channels (visual, auditory, haptic). Technology dimensions enumerate model families (rule‑based, deep neural networks, large language models), learning paradigms (offline, online, reinforcement), latency constraints, and audio synthesis approaches (sample‑based, neural synthesis). Ecosystem dimensions address deployment form (plugin, standalone app, cloud service), licensing (open‑source, commercial), business models, and policy/ethical considerations (copyright, data privacy, fairness).

Analysis of the coded systems reveals several key trends and gaps. The majority of existing agents function as reactive accompanists—providing automatic harmony, rhythm, or accompaniment—while truly proactive creative partners that initiate musical ideas are rare. Input diversity is expanding: although MIDI and audio remain dominant, there is a noticeable rise in speech, text, and gesture‑based controls, reflecting advances in multimodal AI. However, real‑time audio generation still suffers from latency and quality limitations, even with state‑of‑the‑art neural synthesis models. From an HCI standpoint, many systems lack transparent control mechanisms, explainability, and user‑centred customization, which hampers musician trust and seamless integration into creative workflows. The ecosystem analysis shows a fragmented landscape of licensing and policy: commercial products coexist with academic prototypes, leading to reproducibility challenges and unclear ethical guidelines.

The authors demonstrate the utility of the design space through several use cases. By mapping existing systems onto the space, they identify unexplored combinations—e.g., a large‑language‑model‑driven real‑time score generator coupled with gesture input and cloud‑based collaborative editing—that represent promising research directions. They also provide a design checklist derived from the space, enabling practitioners to systematically consider relevant dimensions during early concept development.

Importantly, the design space and the annotated dataset are released as a living artifact (https://live-music-agents.github.io), inviting the community to contribute updates, extend dimensions, and refine codes. This open‑access approach aims to foster interdisciplinary dialogue, accelerate responsible innovation, and ultimately support the emergence of highly general, proactive, and ethically grounded live music agents that can act as true creative partners in real‑time musical performance.