A Minimum Description Length Approach to Multitask Feature Selection

Many regression problems involve not one but several response variables (y’s). Often the responses are suspected to share a common underlying structure, in which case it may be advantageous to share information across them; this is known as multitask learning. As a special case, we can use multiple responses to better identify shared predictive features – a project we might call multitask feature selection. This thesis is organized as follows. Section 1 introduces feature selection for regression, focusing on ell_0 regularization methods and their interpretation within a Minimum Description Length (MDL) framework. Section 2 proposes a novel extension of MDL feature selection to the multitask setting. The approach, called the “Multiple Inclusion Criterion” (MIC), is designed to borrow information across regression tasks by more easily selecting features that are associated with multiple responses. We show in experiments on synthetic and real biological data sets that MIC can reduce prediction error in settings where features are at least partially shared across responses. Section 3 surveys hypothesis testing by regression with a single response, focusing on the parallel between the standard Bonferroni correction and an MDL approach. Mirroring the ideas in Section 2, Section 4 proposes a novel MIC approach to hypothesis testing with multiple responses and shows that on synthetic data with significant sharing of features across responses, MIC sometimes outperforms standard FDR-controlling methods in terms of finding true positives for a given level of false positives. Section 5 concludes.

💡 Research Summary

The paper addresses the problem of selecting predictive features when multiple regression responses are present and potentially share underlying structure. It begins by reviewing ℓ₀‑regularized feature selection and interpreting it through the Minimum Description Length (MDL) principle, which treats model selection as a trade‑off between the length needed to encode the model (the “complexity” term) and the length needed to encode the data given the model (the “fit” term). In a single‑task setting this MDL view is equivalent to classical criteria such as BIC, but it provides a clear information‑theoretic justification for penalizing each added predictor.

The core contribution is the extension of this MDL framework to a multitask scenario, resulting in the “Multiple Inclusion Criterion” (MIC). MIC recognizes that a predictor that influences several responses should not be penalized independently for each task. Instead, the coding cost of a shared predictor grows sub‑linearly with the number of tasks—specifically, the cost is reduced roughly by a logarithmic factor in the number of responses that include the predictor. This formulation encourages the selection of features that are jointly useful across tasks while still allowing task‑specific variables when they provide sufficient explanatory power.

Empirical evaluation proceeds in two parts. First, synthetic data are generated with controllable levels of feature sharing. As the proportion of shared predictors increases, MIC consistently outperforms standard ℓ₀‑based selection: it achieves lower root‑mean‑square error with fewer selected variables, and the advantage becomes pronounced once more than half of the true predictors are shared. Second, real‑world biological data (gene‑expression profiles used to predict multiple drug responses) are analyzed. MIC identifies a larger set of biologically plausible genes than baseline methods while keeping the false‑positive rate below 5 %.

The paper then pivots to hypothesis testing. In the single‑response case, the authors draw a parallel between the Bonferroni correction and an MDL‑based penalty, showing that both can be viewed as ways of inflating the description length of a model to control Type I error. Extending this insight, they propose a multitask version of MIC for hypothesis testing: when a predictor is significant for several responses, its inclusion cost is again reduced, leading to higher statistical power. Simulations with shared signals demonstrate that MIC can discover more true positives at a fixed false‑positive budget than conventional false‑discovery‑rate procedures such as Benjamini–Hochberg.

The discussion acknowledges limitations. MIC assumes that some degree of sharing exists; if the true underlying structure is completely task‑specific, the sub‑linear penalty may cause over‑selection. Moreover, the logarithmic cost reduction is heuristic; a formal proof of optimality under a specific probabilistic model is lacking. Future work is suggested in three directions: (1) learning the amount of sharing from the data, possibly via a hierarchical Bayesian model; (2) extending MIC to nonlinear learners such as kernel methods or deep neural networks; and (3) providing tighter theoretical guarantees for the coding‑cost function.

Overall, the study makes a substantive contribution by marrying information‑theoretic model coding with multitask learning. It offers a principled, computationally tractable criterion that leverages shared structure to improve both predictive accuracy and hypothesis‑testing power, and it validates the approach on both synthetic benchmarks and real biological datasets.