Transfer Learning Using Feature Selection

We present three related ways of using Transfer Learning to improve feature selection. The three methods address different problems, and hence share different kinds of information between tasks or feature classes, but all three are based on the information theoretic Minimum Description Length (MDL) principle and share the same underlying Bayesian interpretation. The first method, MIC, applies when predictive models are to be built simultaneously for multiple tasks (simultaneous transfer'') that share the same set of features. MIC allows each feature to be added to none, some, or all of the task models and is most beneficial for selecting a small set of predictive features from a large pool of features, as is common in genomic and biological datasets. Our second method, TPC (Three Part Coding), uses a similar methodology for the case when the features can be divided into feature classes. Our third method, Transfer-TPC, addresses the sequential transfer’’ problem in which the task to which we want to transfer knowledge may not be known in advance and may have different amounts of data than the other tasks. Transfer-TPC is most beneficial when we want to transfer knowledge between tasks which have unequal amounts of labeled data, for example the data for disambiguating the senses of different verbs. We demonstrate the effectiveness of these approaches with experimental results on real world data pertaining to genomics and to Word Sense Disambiguation (WSD).

💡 Research Summary

The paper introduces three novel transfer‑learning‑based feature‑selection algorithms that are all grounded in the Minimum Description Length (MDL) principle and share a common Bayesian interpretation. The authors first motivate the need for transfer across tasks: in many real‑world domains (e.g., genomics, natural‑language processing) one must select a small predictive subset from a very large pool of candidate features, often under severe data‑scarcity conditions. Classical feature‑selection methods treat each task independently and thus ignore useful statistical regularities that exist across related tasks or feature classes.

The first algorithm, MIC (Minimum Incremental Cost), addresses the “simultaneous transfer” scenario where several tasks share the same feature universe and models are built at the same time. MIC encodes each feature with a three‑part code that indicates whether the feature is used in none, some, or all of the task models. By allowing a feature to be selectively shared, MIC reduces redundant inclusion and lowers the overall description length of the multi‑task model. The authors show that, on high‑dimensional genomic data (thousands of genes, few samples), MIC can achieve comparable or better predictive accuracy while selecting fewer than 10 % of the original features, thereby dramatically improving interpretability.

The second method, TPC (Three‑Part Coding), extends the same MDL coding idea to situations where features can be grouped into predefined classes (e.g., biological pathways, functional categories). TPC learns a prior distribution over classes and then encodes the selection of individual features within each class. This hierarchical coding exploits the intuition that features belonging to the same class tend to be co‑selected, which yields a more compact representation and a stronger statistical bias toward informative groups. Experiments on datasets with clear class structure demonstrate that TPC reduces the number of selected features by roughly 20 % relative to standard L1‑regularized approaches while maintaining or improving classification performance.

The third algorithm, Transfer‑TPC, tackles the “sequential transfer” problem where the target task may be unknown in advance and may have a markedly different amount of labeled data than the source tasks. Transfer‑TPC first estimates class‑level priors from a collection of source tasks, then uses these priors to guide feature selection on a new target task that may have only a handful of labeled examples. This approach is particularly useful for tasks such as Word Sense Disambiguation (WSD) of verbs, where some verbs have abundant sense‑annotated corpora and others have very few. By borrowing the class priors learned from the data‑rich verbs, Transfer‑TPC boosts the F1 score of the low‑resource verbs by an average of 8 % in the authors’ experiments.

All three methods share a common Bayesian derivation: the MDL code length corresponds to the negative log‑posterior of a model given a prior that reflects transferred knowledge. The paper details how the priors are updated online as new data arrive, and how the description‑length objective can be optimized efficiently using greedy forward selection combined with dynamic programming for the multi‑task case.

Empirical evaluation is conducted on two distinct domains. In the genomics experiments, the authors use expression data to predict disease status and demonstrate that MIC and TPC both achieve higher area‑under‑the‑ROC curves than baseline filter and wrapper methods while selecting dramatically fewer genes. In the WSD experiments, Transfer‑TPC is compared against a standard multitask learning baseline and a non‑transfer L1‑logistic regression model; Transfer‑TPC consistently outperforms both, especially when the target verb has fewer than 100 labeled instances.

In summary, the paper makes three key contributions: (1) a principled MDL‑based framework for simultaneous multi‑task feature sharing (MIC); (2) a hierarchical coding scheme that leverages feature‑class structure (TPC); and (3) a sequential transfer mechanism that can operate when the target task is unknown or data‑poor (Transfer‑TPC). The experimental results validate that each method delivers statistically significant improvements in predictive accuracy, model compactness, and interpretability across heterogeneous real‑world datasets. The work opens avenues for applying MDL‑driven transfer learning to other high‑dimensional, low‑sample domains such as proteomics, imaging genetics, and cross‑lingual NLP.