Augmenting Music Sheets with Harmonic Fingerprints

A ugmenting Music Sheets with Harmonic Fingerprints Matthias Miller Department of Computer Science University of K onstanz, Germany matthias.miller@uni.kn Alexandra Bonnici Faculty of Engineering University of Malta, Malta alexandra.bonnici@um.edu.mt Mennatallah El- Assady Department of Computer Science University of K onstanz, Germany menna.el- assady@uni.kn Figure 1: Augmenting piano sheet music with harmonic ngerprint glyphs facilitates the identication of recurring har- monic patterns and the comparison of musical parts to understand dierences in the note distribution. Here, an e xcerpt from Chopin’s ‘ Grande V alse Brillante ’ is augmente d with the ngerprints sho wing a recurring pattern in the rst four glyphs. ABSTRA CT Common Music Notation ( CMN) is the well-established foundation for the written communication of musical information, such as rhythm or harmony . CMN suers from the complexity of its visual encoding and the need for e xtensive training to acquire pr ociency and legibility . While alternative notations using additional visual variables (e.g., color to improv e pitch identication) have been pro- posed, the community does not r eadily accept notation systems that vary widely from the CMN. Therefore, to support student musicians in understanding harmonic relationships, instead of replacing the CMN, we present a visualization technique that augments digital sheet music with a harmonic ngerprint glyph . Our design exploits the cir cle of fths, a fundamental concept in music the ory , as visual metaphor . By attaching such glyphs to each bar of a composition w e provide additional information about the salient harmonic features available in a musical piece. W e conducted a user study to analyze the performance of experts and non-experts in an identication and comparison task of recurring patterns. The evaluation sho ws that the harmonic ngerprint supports these tasks without the ne ed for close-reading, as when compared to a not-annotated music sheet. CCS CONCEPTS • Human-centered computing → Visualization design . Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and the full citation on the rst page. Copyrights for components of this work owned by others than the author(s) must be honor ed. Abstracting with credit is permitted. T o copy otherwise, or republish, to post on servers or to redistribute to lists, r equires prior specic permission and /or a fee. Request permissions from permissions@acm.org. DocEng ’19, September 23–26, 2019, Berlin, Germany © 2019 Copyright held by the owner/author(s). Publication rights licensed to A CM. ACM ISBN 978-1-4503-6887-2/19/09. . . $15.00 https://doi.org/10.1145/3342558.3345395 KEY W ORDS Visualization, Harmony , Analysis, Sheet Music A CM Reference Format: Matthias Miller, Alexandra Bonnici, and Mennatallah El-A ssady. 2019. Aug- menting Music Sheets with Harmonic Fingerprints. In ACM Symposium on Document Engineering 2019 (DocEng ’19), Septemb er 23–26, 2019, Berlin, Ger- many . ACM, New Y ork, NY, USA, 10 pages. https://doi.org/10.1145/3342558. 3345395 1 IN TRODUCTION Isidore of Se ville recounts how until the 7th-century music was only conserved as an auditor y memory as there was yet no means of notating music [ 1 ]. By the 9th centur y , neumes came into use as visual aids to indicate the relative pitch and melody direction in Gregorian chant [ 28 ]. O v er several centuries this neumatic notation was adapted to reect the evolution of musical instruments as well as dierent music composition styles, to cr eate the Conventional Music Notation (CMN) used today . Despite the widespread use of the CMN, the notation faces criticism on tw o counts namely that the notation is not adequate to r epresent musical notation, and that it does not adequately represent all information of a musical score. The CMN is thought to be inadequate as a musical notation b ecause it is mainly based on the use of a ve-line sta such that only nine vertical spaces are available to represent the 12 tones of the chromatic scale. Thus the CMN intr oduces sharp and at signs to represent all 12 tones, unnecessarily complicating the visualization of the note pitches. Likewise , the notation of the note durations does not intuitively represent the score ’s rhythmic properties. These criticisms gave rise to alternative music notation systems to facilitate the reading of music, esp ecially for novice learners. Among these alternative systems, we may nd notation systems such as Klavarskribo [ 14 ], Hummingbird [ 32 ], Dodeka [ 21 ] and DocEng ’19, September 23–26, 2019, Berlin, Germany Miller , Bonnici, and El-Assady . Pizzicato [ 20 ] among others. These alternative notation systems may include the introduction of dierent sta systems to capture equally spaced chromatic inter vals [ 14 ], the use of dierent symbols to represent note pitches and durations [ 16 , 32 ], piano-roll style music representations [21], and numeric-based notations [20]. Howev er , these alternative notations have not b een adopted by the music community due to the lack of accessibility of music scores written in the dierent notation and because the number of musicians who can read the CMN by far outnumbers those who can read alternative notations. Thus, the problem of reading and quickly understanding the CMN remains a struggle for novices [17]. T echnology , notably , screen applications have changed the way musicians are sharing and using musical scores [ 24 ]. Music scores are no longer restricted to print media. Advances in optical music recognition as well as the automated conversion of musical scores into dierent le formats led to the emergence of alternative nota- tions. Instead of providing a complete overhaul of the CMN, often additional score annotations are employ ed as additional teaching aids [ 7 ]. Furthermore, the digital representation of music as web or tablet-based apps makes it possible to have both conventional representations and new notations available for the same do cument such that the student may switch between notation systems easily . In general, she et music encodes musical features such as rhythm, dynamics, harmony , and other instructions as visual symb ols which specify how a p erformer can reproduce a comp osition [ 18 ]. The no- tation, howev er , omits the explicit representation of the relationship that exists between the individual notes and their location within the overarching musical structure [ 11 ]. While understanding these complex relationships is of immediate interest for the musician [ 6 ], such an understanding typically requires analyzing the score and thus, a certain le vel of musical e xpertise [ 7 , 17 ]. The understanding of the harmonic structure of the scor e allows for faster learning of new musical pieces and enhances musical memory and should ideally be more accessible to novices. Annotating the score with a visual r epresentation of the hidden harmonic patterns would, there- fore, help to convey this musical knowledge leading to a deeper understanding of the underlying musical structure [17]. In this paper , we contribute a harmonic ngerprint glyph vi- sualization to encode and augment the distribution of notes in pre-dened windows within musical pieces. This avoids the intro- duction of new music notations that dierm from the conventional music notation. W e use the circle of fths [ 10 ] as a visual metaphor to design the harmonic ngerprint. By attaching this glyph, we add valuable information about the harmonic content to digital sheet music, giving readers the opportunity to eciently identify and compare harmonic relationships acr oss a do cument. T o test the performance of the ngerprint, we conducted a qualitative user study , collecting comparative feedback from both domain experts and non-experts. W e concentrate on the following three research questions to emphasize the b enets and drawbacks of our approach: Q1 Does the proposed ngerprint annotation method support the identication of recurring and long musical patterns? Q2 Does the ngerprint annotation inuence the harmony anal- ysis of the score? Q3 Does the reader’s musical expertise level inuence the eec- tiveness of the ngerprint for performing harmony analysis? 2 RELA TED WORK Understanding harmonic relationships and tonal progressions of a musical composition is a fundamental aspect of music analysis [ 22 ]. T o rev eal these connections of harmony , musicians need to analyze the music to identify the chords use and identify any patterns us- ing knowledge of music theor y . Performing such a task requires knowledge and expertise, which ar e not necessarily within reach of beginner musicians [ 17 ]. Harmony visualization tools make har- monic analysis more accessible by providing visual aids which highlight non-explicit information of the score. Visualization algo- rithms typically follow Shneiderman’s Visual Information Seeking Mantra , that is, follow the three rules: o verview rst; zoom and lter; then details-on-demand [25]. 2.1 Music Harmony Visualization There are dierent ways to visualize harmonic relationships in musical works. Abstract music visualizations enable obtaining an overview even of full musical pieces, but r eaders are not able to look at the details anymore. In contrast, pro viding original details of music score facilitate the understanding of the exact harmonic and rhythmic relationships on a lower level, but the reader can not readily recognize the ov erall structure. Jänicke et al . explain that combining close and distant reading allows users to nd regions of interest while still providing all details instead of only presenting an abstract representation. In this way , readers are directed by distant reading while investigating the underlying details afterward [ 13 ]. W e transfer Jänicke et al . ’s close and distant reading concept from the text domain to music visualization by showing its importance for the visualization harmony in music [13]. 2.1.1 Distant Reading. Analyzing particular attributes of structural data and providing them in a visual way is a typical approach to rev eal unknown pat- terns. For example, Keim and Oelke present a method to apply visual, literary analysis for dierent types of features that are char- acteristic of text [ 15 ]. The proposed visual literature ngerprinting method has pr oven to be an ee ctiv e approach to show meaningful dierences between several te xt sections while including various text features such as av erage sentence length for creating the n- gerprint visualization. Keim and Oelke do not include close reading, so users who want to further investigate the details in interesting areas of the visualization are not supported. One approach to visualize the harmonic content of a musical piece is to create a representation which displays the ov erall form of the music. For example, W attenb erg intr oduces an arc diagram visualization to re veal sequential melodic patterns. Sequential r epe- titions of melo dy notes are highlighted by arcs connected the notes. The arc’s radius and width indicativ e of the distance and similar- ity between recurring melodic sequences [ 31 ]. Similar approaches include the r epresentation of the score using similarity matrix visu- alizations [ 33 ], or the use of the T onnetz grid to create an Isochord visualization of the score [ 2 ]. Using a similar approach, Schroer uses circos graphs which are used to reveal patterns in genomic data. Using these graphs, Schroer represents the harmonic relationship between all the twelve tones of the chromatic scale by linking the chord root with other notes that are played simultaneously [23]. Augmenting Music Sheets with Harmonic Fingerprints DocEng ’19, September 23–26, 2019, Berlin, Germany Such diagram representations of the score emphasize the com- monalities that exist between whole sections of the musical piece. Therefore, these visualizations can highlight similarities in struc- ture that exist between dierent musical scores that are typical for a genre. However , these approaches do not retain the relation to the original score notation. While it is easy to visualize the overall structure of the scor e, the spatial position of these highlighted struc- tures gets lost. Enabling music readers to retriev e the detailed CMN on demand is essential to support understanding of the underlying harmonies. This r equires to combine distant reading with close reading to exploit the advantages of both methods. 2.1.2 Close Reading. Local visualizations of the score retain the spatial information of the highlighted structure concerning the full score. Smith and Williams exploit three-dimensional space and color to visualize typical musi- cal information base d on MIDI les [ 26 ]. They propose to map tone data described by pitch, volume, and timbr e to colored spheres in their visualization. The aim of Smith and Williams is to visually present music to listeners who are not familiar with reading music notation. Rather than providing a static visualization, their model uses a dynamic visual model which decreases the color intensity of the notes with the progress of time, thus allowing the listener to distinguish b etw een dierent tones played at dierent points in time. While such an approach is visually pleasing, it do es not allow the listener to understand the underlying harmonic sections or recurring patterns easily . Algorithms such as that described by Snydal and Hearst create melodic landscapes and harmonic palettes from transcriptions of jazz improvisations [ 27 ], while Miyazaki et al . use cylinders whose height, diameter and color represent pitch, volume and duration information of the notes in the score. In the latter algorithm, the visualization can be represented both at an overview level and also at a detailed level according to user preference [ 19 ]. Ciuha et al . describe an approach to visualize concurrent tones, using color to highlight the harmony [ 4 ]. In this visualization, the horizontal axis represents the temporal progress of a single musical pie ce while the vertical axis encodes the note pitches. Depending on the tem- poral segment size the visualization is blurred based on the tonal unambiguousness. In this way , this method conveys the harmonic journey of a piece while using an appealing repr esentation to en- code the anity of notes by applying a color wheel on the circle of fths to reect the dissonance of intervals. Sapp proposes a visual method which emphasizes the key strength at each section [ 22 ]. This model uses color to encode the dominant notes or key at several hierarchical levels. At the top of the hier- archy , the model represents the most signicant key of the entire composition. The lower levels of the hierarchy repr esent detailed tonal progressions, r esulting in a triangular repr esentation of the keys used in the music. Thus, this model is especially useful to reveal the tonal variations in the piece. 2.2 Annotating the CMN Score The visualization strategies discussed so far are detached from the CMN which makes it dicult to keep the connection b etw een the original music sheets and the visualization. One strategy to resolve this pr oblem is to display the original notation format and annotate the score. An example of such an approach commonly used in music notation is the placement of chord symbols to indicate the accompanying harmonies which the musician nee ds to perform. Howev er , these chord symbols are a rather general description of the harmony progressions and cannot reect the complexity of specic musical parts in detail. Thus, while scores can be annotated using a similar analogy , an annotated visualization w ould let the reader focus on the harmonic aspects of the score. Addressing these concerns, Cuthbert and Ariza describe music21 , a tool which provides a number of features for visual scor e descrip- tions. Among these, the harmonic analysis of the score is shown as chords written in closed-form. The metric analysis is represented as asterisk signs beneath the notes, while a plot of the pitch class against the bar number illustrates the pitch usage over the progres- sion of the score. The VisualHarmony tool described by Malandrino et al . over- lays visual information about the tonal information directly onto the score CMN. The tool aims to support music composition by highlighting score parts that do not comply with classical music theory rules [ 17 ]. The tool facilitates the identication of chord tonality and the respective scale degrees. This is used to display melodic errors supporting the music composition task. In Visual- Harmony , the visual information is shown as colored rectangular boxes enclosing the individual chords. Thus, the system preserves the spatial lo cation of the harmonic patterns. However , the overlap- ping visualization was found to be too distracting, particularly for directly playing the music and thus, De Prisco et al . suggest that the visual information should be displayed above the sta [8]. From this overview of the related work, we may note that visu- alization algorithms which represent the melodic patterns in the music, do so on a global level, without pr eserving the spatial lo- cation of these patterns. While visualizations such that described by [ 17 ] do preserve the spatial location, these focus on analyzing the harmony of the music. While this is important, the harmonic analysis does not capture the melodic patterns that can also be present in the score . Thus, an annotation system which captures both the harmonic patterns and the melodic patterns would be of benet to the community . 3 HARMON Y FINGERPRIN T DESIGN This section describes the propose d harmonic ngerprint visual- ization. This visualization approach will capture the harmonic and melodic content of each bar of the musical score. It exploits the cir- cle of fths as a visual metaphor for its design. While highlighting the harmonic information contained within a bar , the glyph also represents tonal information in a way with which musicians may already be familiar . 3.1 Circle of Fifths The circle of fths is a practical concept to explain the geometric structure of the chord r elationships between all twelve chr omatic pitch classes and is often used by musicians and in music peda- gogy [ 29 ] to visualize the relationships between pitch classes. In 1728, Heinichen augmented the circle of fths to introduce the relationship between major and minor keys, repr esenting the circle DocEng ’19, September 23–26, 2019, Berlin, Germany Miller , Bonnici, and El-Assady . (a) The musical cir cle originally published by Heinichen in 1728 [10] (b) Separation of major and minor chords and their corresponding accidentals Figure 2: The circle of fths shows the relationships between the dierent 24 available keys in W estern music. of fths with 24 segments as shown in Figur e 2(a). Figur e 2(b) fa- cilitates the understanding of the parallel minor and major chords while additionally pro viding the number of accidentals for each key . The distance of the pitch classes in the circle is a measure for the tonal similarity of the ke ys. For e xample, if C Major is the tonic, then F Major or G Major are the most similar chords in terms of auditorial perception. 3.2 Design Rationale In the glyph’s design, we take into consideration multiple musical characteristics which determine the annotated visualization and the used annotation type: 3.2.1 The Region of Interest. In the design of the harmonic ngerprint, we capture the harmonic variations at a detailed level, while retaining a general overview of the piece. For this reason, we cho ose single bars as the region of interest for which we automatically create the glyph. Since the tonal center remains the same within a bar , particularly for music with dance-like structure [ 29 ]. Representing harmonic content per bar provides a reasonable compromise between detail and a general overview of the score. 3.2.2 Representation of Harmonic Content. T o represent the harmonic ngerprint of the bar , we use the root note of the chord within the bar , if it is identiable. The root note is a common way of describing a chord in music theory , and thus, the glyph uses a description which is familiar to musicians. Other alternatives from music theory would include describing the chord using Roman numerals or gured bass [ 30 ], but we think that this would be less meaningful to beginners or non-musical experts. The ngerprint allows for distant reading of harmonic relationships in single bars summarizing all notes into a radial chromatic histogram. 3.2.3 Representation of Melodic Content. The glyph captures the note pitches which also form the melodic content of the bar . Since the role of the glyph is to pro vide a general overview of the harmonic material, we represent this through a radial normalized histogram of the pitch class of notes within the bar . Notes with the same pitch name but in dierent octaves are grouped in the same histogram bin allowing to identify harmonic relationships across a piece. In this manner , bars which hav e similar melodic patterns but written in dierent octaves, or for which there is rhythmic variance, will obtain the same histogram. At an overview level, this is desirable as it captures the broad similarities between bars while keeping close-reading possible through keeping the CMN unchanged. 3.2.4 Capturing Harmonic Relationships between Notes. The glyph visualization provide an overview of the harmonic re- lationship b etw een notes in the bar . This overview should allow musicians to understand at a glance, the pr esence of consonances or dissonances between notes in the respective bar . Neighboring segments in the ngerprint visualization are the most similar notes in music harmony theor y , whereas segments that are place d on the other side of the circle are the most dissimilar relationships. In combination, the reader can eciently identify the dominant notes for each bar . The circle of fths metaphor enables us to illustrate the harmonic relationship using a musical concept that is familiar to the musician. 3.3 Creating the Glyph W e illustrate in Figur e 3(d) the specic glyph shape using the 6th bar from Chopin’s waltz “Grande V alse Brillante” as an example. A histogram of the pitch classes, displayed in Figure 3(c), captur es the information about the notes forming the melodic and harmonic content. This representation is quite sparse and as a r esult, would clutter the annotation. W e alter the histogram by rearranging the pitch classes radially such that each of the tw elve pitch classes from the chromatic scale is displayed by subtending angles of 30 ° (360 ° / 12 = 30 ° ) as shown in Figur e 3(a). This allo ws us to represent the his- togram’s content more compactly . The number of note occurrences within the bar is encoded by the radius of the segment, r esulting in a visualization that is similar to a non-stacked Nightingale Rose Chart [ 3 ]. Moreover , rearranging the pitch classes into the circle of fths format highlights the harmonic relations within the prede- ned music sheet window . Hence, the harmonic ngerprint can be considered as a statistical overview of the number of notes from a single bar . The r eading complexity of the glyph remains unchanged due to its independence of original music document complexity which enables high scalability . T o further improve the r eadability of the visual ngerprint, we use the color scale from Ciuha et al . , which we show in Figure 3(a). This color scheme reects the distance of pitch classes in the color space [ 4 ]. The visual double encoding of the pitch classes in color and position simplies the comparison of dierent ngerprints. Lastly , to further emphasize the harmonic content of the bar , we display the root note of the chord formed by the notes in the bar at the center of the glyph as illustrated in Figur e 3(a). The r esulting shape structure of the glyph is similar to that shown in Figure 3(d). 3.4 Adding Harmonic Fingerprints to the Score For the scope of this work, w e assume that sheet music is readily available in MusicXML le format [ 9 ]. MusicXML is an XML-based Augmenting Music Sheets with Harmonic Fingerprints DocEng ’19, September 23–26, 2019, Berlin, Germany (a) A radial r epresentation of how the colors ar e mappe d to the pitch classes according to the circle of fths. (b) Bar 6 bar taken from Chopin’s “Grande Valse Brillante” using the color encoding from (a) on the notes. (c) W e calculate a colored histogram based on the chromatic pitch classes of all the notes that are present within a bar . (d) The nal ngerprint encodes the notes distribution of a bar by seg- ment area size and color . Figure 3: W e use the circle of fths as a visual metaphor to encode the distribution of notes in a bar ( b) by mapping all notes to their corr esponding pitch class (c). The amount of notes of each class is then encode d by the size of the area segments as shown in (d). The pitch classes are encoded both using color and the segments position inside the ngerprint (a). digital sheet music inter change format designed to provide a univer- sal format for CMN. The use of MusicXML is becoming more wide- spread and music writing software such as MuseScore 1 , Sibelius 2 among many others, supp ort MusicXML representations of new works. Likewise, digital libraries such as the Mutopia Project 3 pro- vide MusicXML sources for classical works. T o obtain the statistical data required for the glyph the Mu- sicXML le is pre-processed using the music21 Python libary [ 5 ]. Through this library , we compute the number of occurrences of each pitch class, identify the chord and the chord’s root note for each bar in the score. T o display the annotated sheet music with the harmonic nger- print, we use OpenSheetMusicDisplay 4 , an open-source application to display MusicXML les in a browser . Similar to the obser vations made by De Prisco et al . , we place the glyphs directly above the bars such that the viewer can intuitively detect the corresponding ngerprint [ 8 ]. T o ensure that the annota- tions do not overlap with the CMN symbols, we increase the space between each system to provide enough space for the ngerprint annotations. W e introduce this additional space thr ough increasing the standard distance between systems in the OpenShe etMusicDis- play application by an amount proportional to the diameter of the glyphs. The allocation of this extra space ensures that both the harmonic ngerprint and the sheet music are legible. 4 EV ALU A TION T o evaluate the p erformance of the ngerprint annotations, we conducted a user study to analyze music theme pattern identi- cation tasks on dierent sheet music to investigate the users’ performance with and without the ngerprint annotations. W e specically focuse d in revealing dierences between domain ex- perts and non-experts to assess the intuitiveness of our appr oach as w ell as identifying tasks that can be supported by the ngerprint visualization introduced in the previous section. 1 https://musescore.org/en 2 https://www.avid.com/sibelius-ultimate 3 http://www.mutopiapr oject.org/ 4 https://github.com/opensheetmusicdisplay/opensheetmusicdisplay 4.1 User Study Methodology The assessment of the user study is base d on the overall satisfaction reported by the users and their performance in nding the patterns of the provided music sheet sections. T o perform the evaluation, we conduct a within-subject evalu- ation, giving each participant two scores, one with and the other without annotations. W e instructed the participants to mark any patterns that they can identify on the printed music score, without explicitly telling them to perform harmony analysis. T o reduce the potential for bias in the evaluation, we selecte d the scores such that they are of equal length and complexity . Moreover , we prepared an annotated and a non-annotated version of both scores randomizing the order with which we presented the scores. 4.1.1 Dataset and Controls. W e applied the ngerprint annotations algorithm to the Frédéric François Chopin’s “ Grande V alse Brillante ” [ 12 ], for which we pro- cessed a ground truth musical harmony analysis beforehand. In total, this waltz has 311 bars of music arranged into seven distinct themes which can b e distinguishe d from the melodic and harmonic variations between the themes. Figure 4 shows the rst 68 bars of this waltz highlighting the themes. By comparing the annotations obtained from the proposed ngerprint annotation system, it is possible to determine whether the ngerprint annotation system is indeed representing these thematic divisions. Since the music score is suciently long, we use selected parts in the evaluation as discussed her eunder . W e extracted two suitable music-sheet sections which we denote as MS1 and MS2 respectively , with MS1 consisting of the rst 68 bars of the waltz while MS2 con- sists of 64 bars, from bar 69 up to bar 132. Bar 68 has a natural break in the music dividing MS1 and MS2 into two dierent musical parts. Both MS1 and MS2 consist of recurring patterns: MS1 consists of two major themes as shown in Figure 4, while MS2 consists of a very similar structure but has three dierent themes instead of two. The full length of the themes varies between 7 or 15 consecu- tive bars, with the latter containing four-bar sub-patterns. These themes were used as ground truth to estimate the performance of the participants. DocEng ’19, September 23–26, 2019, Berlin, Germany Miller , Bonnici, and El-Assady . Figure 4: The rst 68 bars of the ‘Grande V alse Brillante’ (forming MS1 of user evaluation). This consists of two major themes which are enclosed here by the blue and green rectangles. Bars 12 and 45, enclosed in yellow and red, divide the two themes into two parts, with the second part of each theme consisting of some rhythmic variations to the rst occurrence of the theme. 4.1.2 Participants. In total, we evaluated the ngerprint annotation approach with eight participants fr om diverse backgr ounds who completed all phases of the user study . W e sele cted the participants by dividing them into two equal-sized groups. The rst group consists of four domain non-experts (N1–N4) who have little to no music background (mean score of 1.5) and have never performed a harmony analysis (mean score of 1.0). However , these non-experts are e xperts with a post-graduate degree in the eld of data analysis and visualization with a mean age of 34 ± 9. The second group is represented by people who are either play- ing instruments daily or have fundamental to solid knowledge about the harmonic relationships in sheet music (mean score of 3.75). Within the scope of this paper , we refer to the second group as our domain experts (E1–E4). These four participants have an intermediate musical knowledge (mean score of 3.25). Three e x- pert participants practice music as their hobby on a daily bases, while the last is singing in a choir but is aware of the harmonic relationship of multiple voices. 4.1.3 Study Design. W e conducte d the study in thr ee phases. In the preliminary phase, we collected general demographic information from the partici- pants, namely their age, gender , level of musical background and familiarity with harmony analysis. T o collect the information re- lated to musical background, we asked participants to rate their knowledge on a ve-point Likert scale, from novice (1) to expert (5). Likewise, we asked the participants to rate their familiarity with harmony analysis from no knowledge (1) to very familiar (5). In the second phase of the study , we aske d the participants to perform the pattern identication task by adding handwritten anno- tations to highlight patterns on the musical score. The participants carried out the task twice, once with a scor e containing ngerprints and once with the CMN only . W e recorded the amount of time taken by the participants in performing the tasks, advising them that the analysis should not exceed 30 minutes. W e deliberately decide d to exclude a sp ecic training task to investigate the accessibility to the music document on dierent knowledge levels without further explanation of the musical scores or the harmonic ngerprint. The nal phase of the study consiste d of a short inter vie w to elicit user feedback helping us to answer the research questions stated at the end of the introduction section. W e asked participants to receive valuable feedback to gauge their opinion on the usefulness of the harmonic ngerprint and their strategy to fulll the given pattern identication task. 4.2 Use Cases Before presenting the qualitative feedback from the user study interviews, we describ e two use cases that are supported by our harmonic ngerprint annotations. 4.2.1 Identify Octave-Invariant Harmony Relationships. Identifying harmony requires to understand how multiple notes represent a tonal center . This chord depends only on the notes’ pitch classes of the chromatic scale and is independent of the oc- tavation. Consequently , the absolute tone pitch do es not change the harmony if the pitch class remains unchanged. T o identify such similarities, a reader has to identify the line for each note and apply the accidentals of the current key to e xtract the underlying chord. This is a tedious task if the number of notes increases. For e xample, if the note C is available thr ee times simultaneously (e.g., C2, C4, C5) in addition to other notes, then the chord identication task becomes more complex. The ngerprint annotation simply merges all pitch class occurrences making it easier for analysts to focus on the composition of pitch classes to de cide which harmonic aspects are most dominant in a bar and dene the chord. 4.2.2 Musical Theme Extraction. Depending on the composition, the complexity of the CMN makes it dicult to nd recurring musical themes in a score . By using a unique color scale to represent each pitch class, the annotate d Augmenting Music Sheets with Harmonic Fingerprints DocEng ’19, September 23–26, 2019, Berlin, Germany ngerprint enables readers to eciently skim a document for sim- ilarity . Since color is easier to dierentiate then black and white, readers can visually lter for similar harmony patterns. Moreover , the layout of the CMN symbols, such as stem direction, note dura- tion, or note divisions have no visible inuence on the ngerprint and provide a trustw orthy foundation to identify harmonic dier- ences and commonalities. Figure 4 displays how music themes can be identied by comparing the ngerprint visualizations across a score. Due to distant reading, even small dierences of var ying pitch classes can be detecte d. By keeping the CMN, close-reading is still possible, if required to compare rhythmic details within the bars. In addition, the ngerprint enables to identify the most frequent note in each bar at a glance. 4.3 User Feedback Figure 4 shows the annotation results for the bars 5–68 of Chopin’s “Grande V alse Brillante” ar e shown superimposed on the ground truth thematic analysis of this waltz. The second occurrence of theme 1 has similar ngerprints as the rst occurrence. There ar e some rhythmic dierences in the melody line of the second occur- rence which are not reected in the ngerprint visualization. The ngerprints for the second theme are considerably dier ent from that of the rst theme, highlighting the shift in the key . Re curring themes have very similar ngerprints, as expected. T o understand the eect of the harmonic ngerprint glyph for music readers that are on dier ent music expertise levels, we report the qualitative results and elicited feedback on the usefulness of our visualization from the participants. Base d on this feedback and the users’ performance in nding the themes we will addr ess the research questions Q1, Q2, and Q3 from the introduction. W e compared the dierences between the participants’ annotations and the ground truth of the harmonic patterns that are included in the datasets. W e highlight the participants’ statements by using italic and double quotation marks in the following paragraphs. 4.3.1 Usefulness of the Fingerprint. Except for N3, all participants state d that they would like to use the ngerprint annotations in the future in pattern identication tasks having dierent r easons. For example , N1 “ used the ngerprint as an anchor and to keep the overview , [but] got lost in the CMN b e cause of the notes’ optical similarity . ” N2 explained that by using the glyph, she could “ compare images ” to identify the pattern because it is easier since the complexity of the CMN makes it har d to identify dierences. First, N3 performed the pattern identication task with- out the ngerprint and was confused about the ngerprint because the “ notes where easier to compare ” and the ngerprint seemed to not match with the CMN. For N4, it was “ much easier with the ngerprint [...] to see patterns based on the color and location [...] than tr ying to scan and compare the notes. ” E1 argue d that “ to practice a piano piece, it might be helpful to spot parts which are equal or have small dierences. ” E3 found it “ an interesting way of putting a visual asp ect to the combination of notes that ar e indicated within the music score. ” E4 stated that it is “ easier to nd the patterns with the visualization, ” but was unsure whether she might have missed some important information because she did not look further into the CMN. E4 could “ recognize how the piece dev elops ” and “ obtain an overview over the pie ce easier . ” 4.3.2 Paern Identification Strategy . Depending on whether the ngerprint was annotate d to the sheet, the dierent user groups applied dierent strategies to identify repeating patterns. All non-experts, except N3, preferred the an- notated condition since they found it easier to nd matchings. N1 “identied dierences and similarities by comparing rst the pres- ence of the glyph’s segments and afterward its size without looking at the CMN anymore if the glyphs matched” . N4 stated: “ I starte d with the rst icon [...] to nd another that matche d until the next. ” N2 and N4 only highlighte d the patterns by adding handwritten annotations by grouping the ngerprints instead of using the CMN. N1 “ identied dierences and similarities by comparing rst the pres- ence of the glyph’s segments and after ward its size without looking at the CMN anymore if the glyphs matche d. ” N1 and N3 incrementally summarized subpatterns without identifying a theme in full. In the rst round, E1 was provided with the non-annotate d doc- ument where he textually added the chor ds symbol to the rst 12 bars. E1 continue d by highlighting the melodies throughout the document without looking at the harmony any further . E1 found the ngerprint helpful “ to spot bars which are the same ” and was “ gathering information by switching focus between glyphs and [the] music she et. ” At rst, E2 “ skimmed through the whole piece to gain an overall picture of the piece [ and then] moved on to analyzing bar by bar [to] compare phrases to each other to nd where repetitions oc- curred. ” Figure 5 displays E3’s handwritten annotations: E3 divided Theme 1 in MS1 as two subpatterns, excluding bars 9, 12, and 17, and completely identied Theme 2 . T o nd similarities, E3 “ tried to merge the harmonic ngerprint glyph with the melodic patterns to identify the se quences present ” and “ followed the labeling of the chords as well as the note patterns [...] at a dierent pitch level. ” T o identify recurring “ melodic patterns ” in MS2, E3 applied the same strategy as for MS1 with the glyphs by “ marking the tonal and real se quences [...] through the identication of the patterns where the note inter vals are dierent [...], but the pitch is moving in the same direction. ” 4.3.3 Challenges. The participants faced various challenges in the execution of the assigned task. For example, N1 stated: “ I had to go back to the bars that I wanted to compare mor e often because I couldn’t remember all the details as good as with the ngerprint. ” N1 argued that he “ found dierences in the lower system, while the upper system did not change within a bar sequence. ” N1 mainly annotated shorter patterns and could not identify a complete theme. Similarly , N3 did not nd complete harmonic theme patterns without the ngerprint but often highlighted subpatterns of two subsequent bars. N1 also gave the feedback that “ the ngerprint is easier to remember how they lo ok compared to the original music sheet, especially if patterns are farther away . ” N2 found it dicult to “ identify patterns that have optics jumps [be cause of ] new rows continuing on the ne xt page. ” N3 was the only participant, who considered the ngerprint to be confusing since it indicated similarity whereas the music sheet did not reect this. Hence, N3 found the glyph visualization misleading in nding exact patterns. N3, who was provided with MS2 without the ngerprint, indicated that it is “ dicult to see exactly on which line the notes are placed and if they are similar to the others. ” In the second round, N4 DocEng ’19, September 23–26, 2019, Berlin, Germany Miller , Bonnici, and El-Assady . Figure 5: W e told the participants to add handwritten annotations to the printed sheet music. The gure shows how E3 high- lighted dierent patterns. E3 was provided with MS1 including the ngerprint visualization. She highlighted the rst half of Theme 1 in red and its last two bars using blue color . She identie d Theme 2 completely by marking it in green. had to nd the patterns without the glyphs. Due to their absence, N4 perceived it to be “ more tedious to look more closely at the single notes, ” especially if the patterns broke across multiple systems. For E1, the most challenging aspect of the analysis task was in “ reading the sheet and to decide which overall chord it is [be cause] it is time-consuming. ” One of the most challenging aspects for E2 and E4 was “ to analyze the piece and nd similarities [...] without having heard the [...] music b eing played, as this would make such repetitions e xtremely evident. ” E3 “ was not quite sure what the colors within the harmonic ngerprint glyph represented. ” Due to the bar- based distribution of the ngerprints, E4 declared to be “ too focused on single bars at the b eginning ” requiring more time to nd larger patterns in the sheet. 5 DISCUSSION AND LESSONS LEARNED The qualitative user study that we conducted to analyze the per- formance of our ngerprint annotation revealed both benets and drawbacks compared to the CMN only . Except for N3, all non- experts could identify many parts or full patterns of the themes without the need for looking at the CMN details. Thus, the n- gerprints can facilitate the access to she et music for p eople that have little to no knowledge about music theory or harmony rules because it encodes salient harmonical characteristics (Q1). In this way , novices can b ecome more motivated to look at she et music to understand the relationship between the notes in a bar and the annotated visualization. N2 and N4 found the visualization to b e visually compelling which increased their interest to look through the music sheet (Q2). Novices and music learners often be come overwhelmed by the complexity of the CMN and give up early due to the steep learning curve in understanding the CMN. 5.1 Educational A sp ects Since the circle of fths is the foundation of harmonic r elationships in W estern music, the ngerprint is a suitable method to teach learners about the harmonic structure in sheet music. It provides a quick overview of bars and facilitates the dierentiation of bars that appear in dierent sections of a music she et. Mainly , the ngerprint allows one to eciently identify those bars which have the same visual melodic pattern, but which are written at a dierent pitch. Consequently , these bars can be distinguished from bars that are only transposed by octaves without the need for close reading of the single notes in CMN. T o check rhythmic dierences, users are still able to look at the original notation, since we did not change the CMN in the processing step of the ngerprint annotation algorithm. Participants that performed the pattern search task without the ngerprint primarily focused not only on the pitch class but on the rhythmic characteristics. Since the ngerprint only encodes the harmonic characteristics, it serves as a natural lter of the rhythmic aspects which are, besides the note pitches, the most dominant optical features in sheet music. 5.2 Sta Separation An interesting nding was that non-expert users who rst worked through the task having the ngerprints attached to the sheet, tended to combine the upper and lower stav es in the second con- dition, e ven when the ngerprint was not given any more (Q2). Due to typical structures in the dier ent staves, T wo participants of each group identied the patterns without combining the staves. Therefore they did not dierentiate between these parts, e ven when combined the resulting harmony is dierent. They analyzed the upper and the lower stave separately to nd harmonic similarities when the ngerprint was not pr esent by starting with patterns with a length of single bars and extended them until they identi- ed dierences. The remaining study participants r ecognized the ngerprint as a summar y of the bars and seldom looked further into the details of the CMN (Q2). W e call these contrasting strate- gies top-down (with ngerprints), and bottom-up (no ngerprints) analysis approaches. The top-down approach was mainly applied Augmenting Music Sheets with Harmonic Fingerprints DocEng ’19, September 23–26, 2019, Berlin, Germany by N1–N4 by including the ngerprint when adding handwritten annotations to the printed music sheet. 5.3 Harmonic Invariance In the third theme of MS2, all notes from bars 119–124 are trans- posed by an o ctav e higher in bars 127–132. T o identify this similar- ity , the reader has to identify the single notes to understand that harmony is similar . Both N2 and N3 were provided with MS2 with- out the ngerprint and could not identify this harmonic similarity . In comparison, N1 and N4 used the harmonic ngerprint to see the similarities without knowing the underlying relationships. With having the domain knowledge, E2 and E3 were able to detect this harmonic similarity even without being pro vided with the visual annotation. E4 marke d this harmonic similarity by highlighting the ngerprints whereas E1 highlighted the pattern in the CMN. Hence, we assume that when readers want to nd harmonic patterns of all simultaneous notes, the ngerprint is supportive of extracting such similarities more eciently . Another drawback that we e xtracted from the feedback of E1–E3 is that they tend to ignore the ngerprint in favor of the CMN because of its familiarity (Q3). For this user group, the ngerprint representations are less intuitiv e than the CMN. Since we did not explain the design decisions of the visualization to the participants, we elicite d that users who do not tr y to understand the detailed harmonic relationship trust the ngerprint without questioning. One music student replied that she was not sure ab out the color coding in the glyph. Nonetheless, she still found it a promising way to put a visual asp ect of the combination of notes that are indicated within the music score. 5.4 Limitations and Future W ork The ngerprint visualization that we designed to rev eal harmonic relationships in sheet music has proved to be helpful to readers in some scenarios including harmonic pattern identication tasks. Nevertheless, we identied some issues in the current design. First, only one color map can b e sele cted to enco de the pitch class that is dicult or impossible to read for pe ople who are vi- sually impaired. Secondly , we currently combine all notes in a bar to extract the root of the dominant chord. Ther efore, the nger- prints certainly show the exact notes histogram for each bar , which is e qually considered in calculating the tonal center for the bars. Consequently , the root note that is displayed at the center of the ngerprint is sometimes not correct. For example, due to the tonal relationship of keys, the r oot note of the other chord mode of the relative key , or even a single note of a melody se quence that is not even part of the actual chord is selected as the root by music21 . In the future, we plan to improve the accuracy by weighting the notes by their duration and excluding underrepresented notes from the calculation of the root note. Similarly , we want to additionally reect the dominance of notes by their duration in the ngerprint. For example, if a note is played only once but has a long duration, then it is currently underrepresented by the harmonic ngerprint and should be weighted according to its tone length. Both musicians and music analysts are not only interested in the distribution of notes in a bar but want to know if the tonal center is a minor or major chord. W e will include this information in the ngerprint to save readers the time in extracting this information from the glyph visualization which is currently required. W e think that it can be an interesting experiment to see how dierences in music styles are r eected in the harmonic ngerprint that can visualize the salient characteristics of music. Based on the qualitative user feedback, we elicited valuable feed- back regarding the information that is encoded in the ngerprint visualization. Music readers who have no understanding about the underlying music theor y require an introduction to the ngerprint because one may assume that it encodes all musical information. Afterward, they can better estimate whether the visualization is suitable for a given task. This has led to confusion for some of our non-expert paricipants who did not trust the visualization after identifying presumed dierences due to the harmonic invariance between the ngerprint and the CMN. When conducting a quanti- tative evaluation we recommend to explain the visualization to the participants in advance to enhance the performance in harmony analysis tasks by improving its understanding. It will be useful to investigate the suitability for dierent tasks to elicit in which situation the ngerprint supports the reader and when it is better to only show the CMN. Eventually , contrar y to our expectations, understanding the meaning of the ngerprint visualization is not readily intuitive even for domain experts. This can be a result of the experts’ fa- miliarity with the CMN, but also on their denition of a pattern, which was not specie d in the preliminary phase of the user study . W e aim to conduct another survey which will include a detaile d explanation of the ngerprint to the users, to see whether the per- formance increases in dierent music analysis tasks, if the r eader is aware of the encoded information. 6 CONCLUSION W e introduce d a visualization method to encode salient harmonic characteristics as a ngerprint. For the design of the harmonic ngerprint glyphs , 5 we exploit the circle of fths, which is the foundation for harmony relationships in W estern music, as a visual metaphor to augment digital music sheet with additional harmonic information. T o analyze the usefulness of our approach, we con- ducted a qualitative user study with four domain experts and four participants without a musical background. The evaluation rev ealed a potential for identication of recurring harmony progressions and the understanding of the underlying harmonic structures in sheet music, even if the optical appearance suggests dissimilarity . More- over , our method is suitable to support distant and close reading in sheet music exposing harmonic relationships on a rather abstract level, while keeping the original music notation unchanged. As a consequence, readers can view sheet music on dierent levels of detail. In the future, we aim to enhance our approach by integrating user feedback and by setting up a visual analysis system for music scores. Thus, we further aim at combining close and distant r eading into an interactive visual analysis system. Consequently , this web application will enable fast access to suitable music analysis tools for any sheet music available in MusicXML. 5 Our system is available under https://musicvis.dbvis.de/app/ngerprint DocEng ’19, September 23–26, 2019, Berlin, Germany Miller , Bonnici, and El-Assady . REFERENCES [1] Stephen A. Barney , W . J. Lewis, J. A. Beach, and Oliver Berghof. 2006. The Etymologies of Isidore of Seville . Cambridge University Press. https://doi.org/10. 1017/CBO9780511482113 [2] T ony Bergstr om, Karrie Karahalios, and John C. Hart. 2007. Isochords: Visualizing Structure in Music. In Proc. of Graphics Interface 2007 (GI ’07) . ACM, 297–304. https://doi.org/10.1145/1268517.1268565 [3] Lee Brasseur . 2005. Florence Nightingale ' s Visual Rhetoric in the Rose Diagrams. T echnical Communication Quarterly 14, 2 (2005), 161–182. https://doi.org/10. 1207/s15427625tcq1402_3 [4] Peter Ciuha, Bojan Klemenc, and Franc Solina. 2010. Visualization of Concur- rent Tones in Music with Colours . In Proc. of the International Conference on Multimedia . ACM Press. https://doi.org/10.1145/1873951.1874320 [5] Michael Scott Cuthbert and Christopher Ariza. 2010. music21: A Toolkit for Computer- Aided Musicology and Symbolic Music Data. In Proc. of the Interna- tional Society of Music Information Retrieval Conference . [6] W . Bas De Haas, José Pedro Magalhães, Frans Wiering, and Remco C. V eltkamp. 2013. Automatic functional harmonic analysis. Computer Music Journal 37, 4 (2013), 37–53. https://doi.org/10.1162/COMJ_a_00209 [7] Roberto De Prisco, Delna Malandrino, Donato Pirozzi, Gianluca Zaccagnino, and Rocco Zaccagnino. 2017. Understanding the structure of musical compositions: Is visualization an eective approach? Information Visualization 16, 2 (2017), 139–152. https://doi.org/10.1177/1473871616655468 [8] Roberto De Prisco, Delna Malandrino, Donato Pirozzi, Gianluca Zaccagnino, and Rocco Zaccagnino. 2018. Evaluation Study of Visualisations for Harmonic Anal- ysis of 4-part Music. In 22nd International Conference Information Visualisation (IV) . IEEE, 484–489. [9] Michael Good. 2001. MusicXML: An internet-friendly format for sheet music. In XML Conference and Expo . 03–04. [10] Johann David Heinichen. 1969. Der General-Bass in der Composition (Dresden, 1728; facs. ed., Hildesheim . Ge org Olms V erlag. [11] Samuel Hunt, T om Mitchell, and Chris Nash. 2017. How can music visualisation techniques reveal dierent perspectives on musical structure? . In International Conference on T echnologies for Music Notation and Representation . [12] IMSLP. 2019. Grande valse brillante , Op.18 (Chopin, Frédéric) . Retrieved July 15, 2019, from https://imslp .org/wiki/Grande_valse_brillante, _Op.18_(Chopin, _Frédéric). [13] Stefan Jänicke, Greta Franzini, Muhammad Faisal Cheema, and Gerik Scheuer- mann. 2015. On Close and Distant Reading in Digital Humanities: A Survey and Future Challenges. In Eurographics Conference on Visualization (Euro Vis) - ST ARs , R. Borgo, F. Ganovelli, and I. Viola (Eds.). The Eurographics Association. https://doi.org/10.2312/eurovisstar .20151113 [14] Wilke Jan Kaasjager . 2001. Klavarskribo. https://doi.org/10.1093/gmo/ 9781561592630.article.15102 [15] Daniel A. K eim and Daniela Oelke. 2007. Literature ngerprinting: A new method for visual literar y analysis. In IEEE Conference on Visual A nalytics Science and T echnology . IEEE, 115–122. https://doi.org/10.1109/V AST .2007.4389004 [16] Yi- Ting Kuo and Ming-Chuen Chuang. 2013. A proposal of a color music notation system on a single melody for music beginners. International Journal of Music Education 31, 4 (2013), 394–412. https://doi.org/10.1177/0255761413489082 [17] D. Malandrino, D . Pirozzi, and R. Zaccagnino. 2018. Visualization and music har- mony: Design, implementation, and evaluation. In 22nd International Conference Information Visualisation . 498–503. https://doi.org/10.1109/iV .2018.00092 [18] Matthias Miller, Johannes Häußler , Matthias Kraus, Daniel Keim, and Mennatallah El- Assady . 2018. Analyzing Visual Mappings of T raditional and Alternative Music Notation. IEEE VIS W orkshop on Visualization for the Digital Humanities (2018). [19] Reiko Miyazaki, Issei Fujishiro , and Rumi Hiraga. 2003. Exploring MIDI Datasets. In A CM SIGGRAPH 2003 Sketches &Amp; Applications (SIGGRAPH ’03) . ACM, 1. https://doi.org/10.1145/965400.965453 [20] Arpege Music. 2019. Pizzicato music notation. Retriev ed April 9, 2019, from http://www.arpegemusic.com/alternative- music- notation.htm. [21] Jacques-Daniel Rochat. 2019. Dodeka music notation. Retrieved April 9, 2019, from https://www .dodekamusic.com/how- to- learn- music- faster/. [22] Craig Stuart Sapp. 2005. Visual Hierarchical K ey Analysis. Computers in Enter- tainment 3, 4 (Oct. 2005), 1–19. https://doi.org/10.1145/1095534.1095544 [23] Kelly Schroer . 2019. Visualizing Harmony. Retrieved April 9, 2019, from http://kellyschroer .com/harmony.html. [24] Véronique Sébastien, Didier Sébastien, and Noël Conruyt. 2013. Annotating works for music education: Propositions for a musical forms and structures ontology and a musical performance ontology . In Pr oc. of the International Society of Music Information Retrieval . 451–456. [25] B. Shneiderman. 1996. The eyes have it: a task by data type taxonomy for infor- mation visualizations. In Proceedings 1996 IEE E Symposium on Visual Languages . IEEE Comput. Soc. Press, 336–343. https://doi.org/10.1109/VL.1996.545307 [26] S.M. Smith and G.N. Williams. 1997. A visualization of music. In Proc. of Visual- ization ’97 . IEEE, 499–503. [27] Jon Snydal and Marti Hearst. 2005. Impro Viz: visual explorations of jazz impro- visations. In CHI ’05 extended abstracts on Human factors in computing systems . ACM Pr ess, 1805. https://doi.org/10.1145/1056808.1057027 [28] Hope Strayer . 2013. From Neumes to Notes: The Evolution of Music Notation. Musical Oerings 4, 1 (2013), 1–14. https://doi.org/10.15385/jmo.2013.4.1.1 [29] E. T aylor. 1989. The AB guide to music theory - part 1 . Associated Board of the Royal Schools of Music. Retrieved April 9, 2019, from https://books.google.com. mt/books?id=h7hZSAAACAAJ. [30] E. T aylor. 1989. The AB guide to music theory - part 2 . Associated Board of the Royal Schools of Music. Retrieved April 9, 2019, from https://books.google.com. mt/books?id=h7hZSAAACAAJ. [31] M. W attenberg. 2002. Arc diagrams: visualizing structure in strings. In IEEE Symposium on Information Visualization . IEEE Comput. So c, 110–116. https: //doi.org/10.1109/INFVIS.2002.1173155 [32] Blake W est. 2019. Hummingbird. Retrieved April 9, 2019, from http://www. hummingbirdnotation.com/. [33] Jacek W olkowicz, Stephen Brooks, and Vlado Keselj. 2009. Midivis: Visualizing Music Structure via Similarity Matrices. In Proc. of the 2009 International Computer Music Conference .