Make the Unhearable Visible: Exploring Visualization for Musical Instrument Practice

We explore the potential of visualization to support musicians in instrument practice through real-time feedback and reflection on their playing. Musicians often struggle to observe the patterns in their playing and interpret them with respect to their goals. Our premise is that these patterns can be made visible with interactive visualization: we can make the unhearable visible. However, understanding the design of such visualizations is challenging: the diversity of needs, including different instruments, skills, musical attributes, and genres, means that any single use case is unlikely to illustrate the broad potential and opportunities. To address this challenge, we conducted a design exploration study where we created and iterated on 33 designs, each focusing on a subset of needs, for example, only one musical skill. Our designs are grounded in our own experience as musicians and the ideas and feedback of 18 musicians with various musical backgrounds and we evaluated them with 13 music learners and teachers. This paper presents the results of our exploration, focusing on a few example designs as instances of possible instrument practice visualizations. From our work, we draw design considerations that contribute to future research and products for visual instrument education.

💡 Research Summary

The paper investigates how interactive visualizations can make the “unhearable” aspects of musical instrument practice visible, thereby providing richer feedback for learners and teachers. The authors first categorize unhearable patterns into two groups: (1) limitations of auditory perception—subtle timing deviations, overlapping hand‑and‑foot drum sounds, and fine pitch errors that are difficult to discern by ear; and (2) limitations of memory and cognition—long‑term trends, variation over bars, and distributional information that require mental integration across time. Recognizing that these patterns are naturally suited to visual analysis, the study embarks on a five‑year design space exploration.

Using their own experience as musicians and feedback from 18 external musicians, the authors generated 33 prototype visualizations. Each prototype focuses on a single musical skill (rhythm, pitch, chords, dynamics, improvisation) and employs familiar visual encodings such as piano rolls, line charts, bar/area charts, pie charts, and heatmaps. The scope is deliberately limited to MIDI‑capable instruments (digital piano keyboards, electronic drum kits, and electric guitars with a special pickup) to ensure reliable, low‑latency data capture. Designs emphasize immediacy (feedback during or immediately after an exercise), familiarity (standard chart types that musicians already recognize), and isolation of skills so that each visualization can be evaluated against a specific practice goal.

A qualitative evaluation with 13 participants (students and teachers) revealed several key findings. Rhythm visualizations that map timing deviations to color or positional offsets allowed learners to spot millisecond‑level errors instantly. Pitch visualizations using histograms highlighted systematic intonation issues, aiding finger placement correction. Drum visualizations that separate hand and foot tracks into distinct lines made complex polyrhythms and hidden coordination errors apparent. Guitar visualizations that display string‑wise note density via pie charts exposed uneven usage patterns. For improvisation, heatmaps of note duration and pitch distribution conveyed both variability and repetition, guiding learners toward more expressive playing. Teachers reported that the visual summaries enabled them to set concrete, data‑driven practice targets (e.g., “maintain timing error below 10 ms for the next eight bars”).

From these experiences the authors derive five design considerations: (a) present only the minimal necessary information to avoid visual overload; (b) provide a baseline or reference for comparison; (c) tailor encodings to instrument‑specific affordances; (d) adjust feedback granularity according to learner expertise; and (e) incorporate interactive navigation (zoom, pan, filtering) to explore longer recordings. Limitations include reliance on MIDI (excluding acoustic instruments without suitable sensors), the need to further reduce latency for truly real‑time feedback, and potential differences in how novices versus experts interpret visual cues.

The paper’s contributions are threefold: (1) a broad design exploration that showcases 33 distinct visualizations for music practice, illustrating how subtle auditory information can be rendered visible; (2) empirical evidence from user studies that such visual feedback is perceived as valuable and can improve self‑assessment; and (3) a set of design considerations to guide future research and commercial product development in visual music education. The authors suggest future work on audio‑based pitch extraction, multimodal feedback (LEDs, AR overlays), and longitudinal studies to measure learning gains over extended periods. Overall, the study demonstrates that making the unhearable visible is not only feasible but also beneficial for enhancing practice efficiency and pedagogical interaction.