Domain Adaptation for Statistical Classifiers

The most basic assumption used in statistical learning theory is that training data and test data are drawn from the same underlying distribution. Unfortunately, in many applications, the “in-domain” test data is drawn from a distribution that is related, but not identical, to the “out-of-domain” distribution of the training data. We consider the common case in which labeled out-of-domain data is plentiful, but labeled in-domain data is scarce. We introduce a statistical formulation of this problem in terms of a simple mixture model and present an instantiation of this framework to maximum entropy classifiers and their linear chain counterparts. We present efficient inference algorithms for this special case based on the technique of conditional expectation maximization. Our experimental results show that our approach leads to improved performance on three real world tasks on four different data sets from the natural language processing domain.

💡 Research Summary

**
The paper tackles the fundamental assumption in statistical learning that training and test data are drawn from the same distribution, an assumption that often fails in real‑world natural‑language‑processing (NLP) applications. In many scenarios, abundant labeled data exist for a source domain (e.g., newswire text such as the Wall Street Journal), while the target domain (e.g., biomedical articles, email, meeting transcripts) has only a small amount of labeled data because annotation is expensive. The authors formalize this situation as a domain‑adaptation problem and critique existing approaches—primarily prior‑based methods that estimate a prior from out‑of‑domain data and then fix it when training on in‑domain data, as well as feature‑augmentation techniques that treat the out‑of‑domain model as a black‑box feature generator. These methods are asymmetric, require ad‑hoc hyper‑parameters, and do not scale gracefully to multiple out‑of‑domain sources.

To address these limitations, the authors propose a three‑component mixture model. They posit three latent distributions: (q^{(o)}) (truly out‑of‑domain), (q^{(g)}) (general, shared across domains), and (q^{(i)}) (truly in‑domain). The observed out‑of‑domain data are generated from a mixture of (q^{(o)}) and (q^{(g)}); the observed in‑domain data are generated from a mixture of (q^{(i)}) and (q^{(g)}). For each example a hidden binary indicator (z) denotes whether the example originates from the domain‑specific component (value 1) or the general component (value 0). This formulation treats the two data sets symmetrically and makes explicit the amount of “general” information that can be shared across domains.

The learning algorithm is based on Conditional Expectation‑Maximization (Conditional EM). In the E‑step, given current parameters, the posterior probability of each hidden indicator (z) is computed for every training instance. In the M‑step, the expected conditional log‑likelihood (augmented with Gaussian priors on the weight vectors) is maximized with respect to three sets of weight vectors: (\lambda^{(i)}) for the truly in‑domain component, (\lambda^{(o)}) for the truly out‑of‑domain component, and (\lambda^{(g)}) for the shared component. Because the underlying model is a maximum‑entropy (log‑linear) classifier, the M‑step reduces to the same convex optimization problem as ordinary maximum‑entropy training; the authors employ limited‑memory BFGS to solve it efficiently. The model also includes Beta priors over the Bernoulli parameters governing binary input features, allowing the algorithm to learn how “general” each feature is across domains.

The framework is instantiated for two widely used discriminative models:

1. Maximum Entropy (ME) classifiers (logistic regression). Here the conditional distribution (p(y\mid x;\lambda)) is expressed with three separate weight vectors, and the hidden indicator determines which vector is used for a given example.
2. Maximum Entropy Markov Models (MEMM), a linear‑chain conditional model used for sequence labeling (e.g., part‑of‑speech tagging). The same mixture idea is applied to the transition and emission potentials, enabling the model to share general transition patterns while adapting domain‑specific ones.

The authors evaluate the approach on four NLP data sets covering distinct domains (newswire, biomedical abstracts, meeting transcripts, and email) and three tasks (part‑of‑speech tagging, named‑entity recognition, and document summarization). Baselines include: (a) training only on the scarce in‑domain data, (b) prior‑based domain adaptation (training a maximum‑entropy model on out‑of‑domain data and using its parameters as a Gaussian prior for the in‑domain model), and (c) feature‑augmentation where predictions of an out‑of‑domain model are added as features for the in‑domain learner. Across all experiments, the proposed “Maximum Entropy Genre Adaptation” (MEGA) model consistently outperforms the baselines, achieving absolute accuracy gains of roughly 2–5 % and larger relative improvements when the domain shift is pronounced (e.g., biomedical vs. news). Moreover, the learned mixing proportions (\pi^{(i)}) and (\pi^{(o)}) provide an interpretable estimate of how much of each data set is “general” versus “domain‑specific,” eliminating the need for manual tuning.

Key contributions of the paper are:

• A symmetric probabilistic formulation of domain adaptation that explicitly models a shared general distribution and two domain‑specific distributions.
• An efficient Conditional EM learning algorithm that leverages existing convex optimization tools for maximum‑entropy models, making the approach practical for large‑scale NLP problems.
• General applicability to both flat classifiers and linear‑chain sequence models, demonstrating the flexibility of the mixture framework.
• Empirical evidence that the method outperforms prior‑based and feature‑augmentation baselines on multiple real‑world tasks, confirming its practical value.

The work opens several avenues for future research, such as extending the mixture to more than two domains, incorporating continuous or high‑dimensional features without binary discretization, and integrating the approach with deep neural architectures where hidden layers could also be partitioned into general and domain‑specific components. Nonetheless, the paper provides a solid, theoretically grounded, and empirically validated solution to the pervasive problem of domain adaptation in statistical classifiers.