Automatic Organisation and Quality Analysis of User-Generated Content with Audio Fingerprinting

The increase of the quantity of user-generated content experienced in social media has boosted the importance of analysing and organising the content by its quality. Here, we propose a method that uses audio fingerprinting to organise and infer the quality of user-generated audio content. The proposed method detects the overlapping segments between different audio clips to organise and cluster the data according to events, and to infer the audio quality of the samples. A test setup with concert recordings manually crawled from YouTube is used to validate the presented method. The results show that the proposed method achieves better results than previous methods.

💡 Research Summary

This paper presents a novel method for automatically organizing and assessing the quality of user-generated audio content, particularly focusing on the abundant and redundant recordings of live events like concerts found on social media platforms. The core challenge addressed is managing a large collection of unsynchronized, variable-quality audio clips from the same event to group them logically and identify the best versions.

The proposed method operates through a three-stage pipeline: Synchronization, Clustering, and Quality Inference, all leveraging audio fingerprinting technology.

Synchronization: Instead of using audio fingerprinting for traditional music identification (like Shazam), the authors repurpose it for temporal alignment. They employ a landmark-based audio fingerprinting algorithm (based on Wang’s work and implemented by Cotton and Ellis). This algorithm identifies robust spectral peaks in the audio to create a compact “fingerprint.” By comparing fingerprints between all pairs of audio samples in a database, the system detects overlapping segments and calculates the precise time offset between them. This step is crucial as user recordings are never perfectly aligned in time.

Clustering: The matching information from the synchronization phase is used to group recordings of the same song or event into clusters. This is modeled as a graph problem: each audio sample is a vertex, and an edge exists between two vertices if their fingerprints match. Connected components in this graph form the clusters. However, fingerprint matching can produce false positives, leading to incorrect merging of clusters from different songs. To mitigate this, the authors introduce a two-layer filtering strategy:

1. Landmark-level filtering: If multiple matches with different offsets are found between the same pair of samples, only the match with the highest number of matching landmarks is retained.
2. Sample-level filtering: For the list of samples matched to a query sample, their “percentage of matching landmarks” is analyzed. False positives tend to have a significantly lower percentage. The method filters out samples that fall below the average percentage and are located after a point where the percentage drops sharply (using a slope threshold, t_d = -0.07). This prevents erroneous edges from being added to the graph.

Quality Inference: Once clusters are formed, the system ranks the quality of samples within each cluster. Previous work (Kennedy and Naaman) used a sample’s degree (number of connections) in the graph as a proxy for quality. This paper proposes a more nuanced metric: a sample’s quality score is defined as the sum of all matching landmarks it has with every other sample in the database. The rationale is that higher-quality recordings (with less noise and clearer audio) possess more identifiable and stable acoustic features, leading to more landmark matches across the board. This metric aims to consistently rank professionally produced recordings higher than amateur user-generated ones.

Evaluation: The method was validated using a dataset of concert recordings manually crawled from YouTube. The results demonstrated that the proposed filtering approach effectively removed false positives at both landmark and sample levels. Furthermore, the new quality inference metric (sum of matching landmarks) outperformed the simpler neighbor-counting method, successfully assigning higher scores to known high-quality, professionally edited recordings compared to user-generated clips.

In conclusion, this work successfully adapts audio fingerprinting for content organization and quality assessment. Its key contributions are the introduction of a robust filtering mechanism to improve clustering accuracy and a more informative quality scoring metric based on the aggregate strength of fingerprint matches. This approach offers a practical solution for managing large archives of user-generated audio, enabling automatic event-based grouping and the selection of the best available versions for end-users.