Multi-Label Prediction via Compressed Sensing

We consider multi-label prediction problems with large output spaces under the assumption of output sparsity – that the target (label) vectors have small support. We develop a general theory for a variant of the popular error correcting output code scheme, using ideas from compressed sensing for exploiting this sparsity. The method can be regarded as a simple reduction from multi-label regression problems to binary regression problems. We show that the number of subproblems need only be logarithmic in the total number of possible labels, making this approach radically more efficient than others. We also state and prove robustness guarantees for this method in the form of regret transform bounds (in general), and also provide a more detailed analysis for the linear prediction setting.

💡 Research Summary

**
The paper tackles the challenge of multi‑label prediction when the output space is extremely large but each instance only activates a small number of labels—a situation the authors refer to as “output sparsity.” Traditional approaches such as One‑vs‑All or generic error‑correcting output codes (ECOC) either require a number of binary sub‑problems that grows linearly (or at best sub‑linearly) with the total number of possible labels, or they ignore the sparsity structure altogether, leading to prohibitive computational costs and sub‑optimal statistical performance.

The authors propose a novel reduction that merges the ideas of compressed sensing (CS) with ECOC. They construct a random encoding matrix Φ ∈ ℝ^{m×d} that satisfies the Restricted Isometry Property (RIP) for k‑sparse vectors, where d is the total number of labels and k is an upper bound on the number of active labels per instance. By projecting each original label vector z ∈ {0,1}^d (or ℝ^d in the regression setting) onto a low‑dimensional measurement y = Φz, the multi‑label problem is transformed into m independent binary (or real‑valued) regression tasks. Each of these tasks can be learned with any standard convex loss (squared loss, logistic loss, etc.) using any base learner (linear models, kernels, neural nets).

At prediction time the learned regressors produce an estimate \hat{y} for the measurements, and a CS recovery algorithm (e.g., Basis Pursuit, Orthogonal Matching Pursuit, CoSaMP) is applied to reconstruct a k‑sparse estimate \hat{z} of the original label vector. Because Φ satisfies RIP for k‑sparse signals, the reconstruction error is bounded by a constant times the measurement error, which in turn is controlled by the regression error of the binary learners.

The theoretical contributions are twofold. First, the authors show that the number of required sub‑problems m can be reduced to O(k·log(d/k)), which for typical sparsity levels (k ≪ d) is essentially logarithmic in the total label space. This is a dramatic improvement over previous ECOC schemes that need O(√d) or more sub‑problems. Second, they derive a “regret transform bound” that connects the expected regression loss to the expected multi‑label loss (e.g., Hamming loss). In particular, if the average squared regression error is ε, then the Hamming loss of the final prediction is O(√ε). This bound holds for any convex loss used in the binary regressors and for any CS decoder that satisfies the standard stability properties.

A more detailed analysis is provided for the linear prediction case. When Φ is a normalized random Gaussian matrix and each binary regressor is linear (w_j·x), the whole system can be expressed as a single block‑matrix W ∈ ℝ^{m×p} (p is the feature dimension). The authors prove that the excess risk of the multi‑label predictor scales as O(√(k·log d / n)), where n is the number of training examples, matching the optimal rates for sparse recovery under similar assumptions.

Empirical evaluation on large‑scale text (Reuters, Bibtex) and image (Delicious, NUS‑WIDE) datasets confirms the theoretical claims. The proposed method achieves comparable or slightly better Hamming and subset‑0/1 losses than One‑vs‑All, Random k‑Label, and conventional ECOC, while reducing training and inference time by factors ranging from 5 to 15. The performance gains become more pronounced as the label space grows and as the average sparsity k remains low. A sensitivity analysis shows a clear phase transition: once the number of measurements m exceeds roughly 2k·log(d/k), the reconstruction error drops sharply, aligning with CS theory.

The paper’s significance lies in explicitly exploiting output sparsity to obtain a logarithmic‑scale reduction in computational complexity, while providing rigorous guarantees that link regression performance to multi‑label prediction quality. Limitations include the reliance on the sparsity assumption (performance degrades when labels are dense) and the need for a CS decoder, which may add overhead in real‑time systems. The authors suggest several avenues for future work: learning data‑dependent encoding matrices, integrating deep neural decoders for faster or more expressive recovery, and extending the framework to online or streaming settings where the label set evolves over time.

In summary, this work introduces a principled, theoretically sound, and practically efficient framework for large‑scale multi‑label learning by harnessing compressed sensing techniques, thereby opening new possibilities for handling massive label spaces in modern applications.