Whispers of the Butterfly: A Research-through-Design Exploration of In-Situ Conversational AI Guidance in Large-Scale Outdoor MR Exhibitions

Large-scale outdoor mixed reality (MR) art exhibitions distribute curated virtual works across open public spaces, but interpretation rarely scales without turning exploration into a scripted tour. Through Research-through-Design, we created Dream-Butterfly, an in-situ conversational AI docent embodied as a small non-human companion that visitors summon for multilingual, exhibition-grounded explanations. We deployed Dream-Butterfly in a large-scale outdoor MR exhibition at a public university campus in southern China, and conducted an in-the-wild between-subject study (N=24) comparing a primarily human-led tour with an AI-led tour while keeping staff for safety in both conditions. Combining questionnaires and semi-structured interviews, we characterize how shifting the primary explanation channel reshapes explanation access, perceived responsiveness, immersion, and workload, and how visitors negotiate responsibility handoffs among staff, the AI guide, and themselves. We distill transferable design implications for configuring mixed human-AI guiding roles and embodying conversational agents in mobile, safety-constrained outdoor MR exhibitions.

💡 Research Summary

The paper investigates how to scale interpretive support in large‑scale outdoor mixed‑reality (MR) art exhibitions, where artworks are distributed across open public spaces and visitors roam freely with head‑mounted displays. Traditional human docents can provide deep, adaptive explanations but are difficult to scale outdoors, especially when multilingual audiences and safety concerns are involved. The authors therefore propose a mixed‑role configuration: human staff remain responsible for safety, device assistance, and route coordination, while an embodied conversational AI guide, “Dream‑Butterfly,” serves as the primary source of artwork interpretation.

Dream‑Butterfly is a small, non‑humanoid companion that follows the visitor in an idle, non‑intrusive mode and can be summoned by a hand gesture. Once summoned, it lands on the visitor’s hand, signals readiness, and engages in a walk‑and‑talk dialogue. The system pipeline consists of (1) on‑device speech‑to‑text, (2) a retrieval component that fetches curator‑ or artist‑provided textual material anchored to each artwork, (3) a context‑constrained large language model (LLM) that generates a response grounded in the retrieved material, and (4) speech synthesis plus optional subtitles in Mandarin, Cantonese, or English. The embodiment was deliberately lightweight to respect the limited compute budget of the HMD, and its non‑human form was chosen to avoid uncanny expectations while still providing a visible social presence.

A between‑subjects field study was conducted on a university campus in southern China, where a 30‑plus artwork MR exhibition covered roughly 26,000 m². Twenty‑four participants were randomly assigned to either an AI‑led condition (Dream‑Butterfly is the main explanation channel) or a human‑led condition (a human docent provides explanations). In both conditions, human staff were present for safety. Participants explored the exhibition freely, summoning the AI or interacting with the human guide as needed. After the experience, participants completed Likert‑scale questionnaires measuring explanation access, perceived responsiveness, immersion, and cognitive/physical workload, and took part in semi‑structured interviews probing responsibility hand‑offs and trust.

Results show that the AI‑led group accessed explanations more frequently (≈32 % higher) and rated the immediacy of responses significantly higher than the human‑led group. Immersion scores were modestly higher, but participants reported a reduced awareness of the boundary between virtual and physical worlds, raising safety awareness concerns. Cognitive workload was higher for the AI condition, especially during multilingual switches where speech‑recognition errors increased. Qualitative interviews revealed that visitors treated the AI as a “supplementary tool” and still expected human staff to intervene for safety or technical problems; they also appreciated the non‑human avatar for its low social risk and felt it did not demand excessive authority.

From these findings the authors derive five design principles: (1) prioritize on‑demand, interruptible explanations over scripted tours; (2) ground AI responses in curated material and have the system abstain or ask clarifying questions when evidence is insufficient; (3) keep the embodiment computationally lightweight; (4) employ low‑risk, non‑humanoid avatars to avoid uncanny expectations; and (5) provide explicit UI cues that delineate the AI’s interpretive role from the human staff’s safety role.

The paper contributes (1) the Dream‑Butterfly system itself, (2) an in‑the‑wild comparative evaluation of AI‑first versus human‑first guiding configurations in a safety‑constrained outdoor MR setting, and (3) transferable design implications for mixed human‑AI guiding roles, responsibility hand‑offs, and embodiment choices. The work demonstrates that conversational AI can substantially improve access to interpretive content in large outdoor MR exhibitions, but it also highlights the need for careful integration of safety protocols, multilingual robustness, and clear role boundaries to ensure a safe, engaging, and cognitively manageable visitor experience. Future work is suggested on multi‑agent collaborations, real‑time error recovery, and broader cross‑cultural deployments.