On Evaluation of Unsupervised Feature Selection for Pattern Classification

Unsupervised feature selection aims to identify a compact subset of features that captures the intrinsic structure of data without supervised label. Most existing studies evaluate the performance of methods using the single-label dataset that can be instantiated by selecting a label from multi-label data while maintaining the original features. Because the chosen label can vary arbitrarily depending on the experimental setting, the superiority among compared methods can be changed with regard to which label happens to be selected. Thus, evaluating unsupervised feature selection methods based solely on single-label accuracy is unreasonable for assessing their true discriminative ability. This study revisits this evaluation paradigm by adopting a multi-label classification framework. Experiments on 21 multi-label datasets using several representative methods demonstrate that performance rankings differ markedly from those reported under single-label settings, suggesting the possibility of multi-label evaluation settings for fair and reliable comparison of unsupervised feature selection methods.

💡 Research Summary

The paper challenges the prevailing evaluation paradigm in unsupervised feature selection (UFS), which traditionally relies on single‑label classification accuracy to compare methods. The authors argue that many real‑world datasets are inherently multi‑label, and converting such data into a single‑label problem by arbitrarily selecting one label discards valuable information and introduces a random bias: the apparent superiority of a method may simply reflect the luck of the chosen label rather than its true ability to capture the underlying data structure.

To address this issue, the study proposes a comprehensive multi‑label evaluation framework. Twenty‑one publicly available multi‑label datasets spanning text, biology, image, and signal domains are used. For each dataset, several representative UFS algorithms—including graph‑based methods (Laplacian Score, MCFS), information‑theoretic approaches (UDFS, NDFS), and evolutionary techniques (RUFS, EMUFS)—select a fixed number of top‑k features in a completely unsupervised manner. The selected feature subsets are then fed into a multi‑label k‑Nearest Neighbor classifier (ML‑kNN with k = 10). Performance is measured using four widely accepted multi‑label metrics: Hamming Loss, Ranking Loss, One‑Error (all lower‑is‑better) and Multi‑Label Accuracy (higher‑is‑better). Each experiment is repeated ten times under an 80/20 hold‑out split, and average results are reported.

The experimental findings are striking. While prior single‑label studies often claim that newer sparse‑representation or information‑theoretic methods (e.g., FSDK, RUSLP, CN‑AFS) outperform classic graph‑based techniques like MCFS, the multi‑label results tell a different story. EMUFS (Entropy Maximization UFS) achieves the highest average rank (2.76) across datasets, but MCFS follows closely (average rank 3.05) and, in several cases, surpasses the newer methods on both accuracy and loss‑based measures. For instance, on the GpositiveGO and Scene datasets, MCFS attains Multi‑Label Accuracy comparable to or higher than FSDK and CN‑AFS, while also delivering lower Hamming and Ranking losses. Conversely, on datasets with very high label cardinality (e.g., PlantGO, VirusGO), EMUFS retains an edge, suggesting that entropy‑based selection better preserves complex label dependencies.

A deeper inspection of the loss metrics reveals that performance rankings are not stable across all measures. Some methods excel in minimizing Hamming Loss but lag in Ranking Loss, indicating that they capture individual label correctness well but struggle to order relevant versus irrelevant labels correctly. This variability underscores the importance of using a suite of metrics rather than a single accuracy figure.

The authors conclude that the evaluation protocol itself can dramatically reshape perceived method superiority. By incorporating all labels simultaneously, multi‑label evaluation provides a more faithful assessment of how well a feature subset represents the intrinsic structure of the data, including inter‑label correlations that single‑label setups ignore. Consequently, the paper advocates for the adoption of multi‑label evaluation as a new standard in UFS research, urging future algorithm design to explicitly consider label dependencies and encouraging the community to report results across multiple multi‑label metrics. This shift promises more reliable benchmarking and better alignment between research outcomes and real‑world multi‑label applications such as image tagging, text categorization, and biomedical annotation.