A Unified Perspective on Multi-Domain and Multi-Task Learning

In this paper, we provide a new neural-network based perspective on multi-task learning (MTL) and multi-domain learning (MDL). By introducing the concept of a semantic descriptor, this framework unifies MDL and MTL as well as encompassing various classic and recent MTL/MDL algorithms by interpreting them as different ways of constructing semantic descriptors. Our interpretation provides an alternative pipeline for zero-shot learning (ZSL), where a model for a novel class can be constructed without training data. Moreover, it leads to a new and practically relevant problem setting of zero-shot domain adaptation (ZSDA), which is the analogous to ZSL but for novel domains: A model for an unseen domain can be generated by its semantic descriptor. Experiments across this range of problems demonstrate that our framework outperforms a variety of alternatives.

💡 Research Summary

The paper introduces a unified neural‑network framework that simultaneously addresses multi‑task learning (MTL) and multi‑domain learning (MDL) by exploiting a “semantic descriptor” for each task or domain. Each training example is represented by a pair (x, z) where x is the raw feature vector and z is a metadata vector describing the task or domain (e.g., school‑id and year‑group, or camera type and lighting condition). The model consists of two linear (or shallow) mappings: f P(x) ∈ ℝ^K that projects the input features into a latent space, and g Q(z) ∈ ℝ^K that projects the semantic descriptor into the same space. The prediction is the inner product ˆy = f P(x)·g Q(z). Training minimizes a standard loss (Euclidean for regression, hinge for classification) over all examples by back‑propagating through both sides, learning the weight matrices P (D × K) and Q (B × K).

A key contribution is showing that many classic MTL/MDL algorithms (RMTL, FEDA, MTFL, GO‑MTL) are special cases of this framework obtained by fixing the descriptor matrix Z and imposing particular constraints on P and Q (e.g., identity, low‑rank, or Kronecker‑product structures). Traditional methods assume a single categorical index for each task/domain (1‑of‑N encoding). By contrast, the proposed approach can handle arbitrary multivariate descriptors, allowing richer sharing of information when metadata is available.

Beyond standard MTL/MDL, the authors exploit the same architecture for zero‑shot learning (ZSL) and a newly defined zero‑shot domain adaptation (ZSDA). In ZSL, a novel class is represented only by its descriptor z*; the model computes f P(x)·g Q(z*) for each test instance and selects the class with the highest inner product, eliminating the need for any training examples of that class. ZSDA follows the same principle for domains: given a previously unseen domain’s descriptor, the model can synthesize a domain‑specific predictor without ever seeing data from that domain.

Experiments cover five settings: (1) standard MDL, (2) ZSDA (holding out one domain at a time), (3) MTL, (4) ZSL, and (5) combined multi‑domain multi‑task (MDMT). Implementations use Caffe; ReLU is applied to g Q(z) to encourage sparsity. The number of latent units K is set to D·log D. Baselines include single‑task linear/logistic regression and the four classic MTL methods re‑implemented with traditional 1‑of‑N descriptors. Results consistently show that the proposed method outperforms baselines, especially when the descriptor is multivariate (e.g., (school, year) or (camera, lighting)). In ZSDA, models built solely from the held‑out domain descriptor achieve substantially higher accuracy than a blind transfer baseline that trains a single model on all seen domains.

In summary, the paper provides a simple yet powerful two‑sided neural network that unifies a broad family of MTL/MDL algorithms, leverages rich semantic descriptors for better parameter sharing, and naturally extends to zero‑shot scenarios for both tasks and domains. This work highlights the importance of explicitly modeling task/domain metadata and opens avenues for rapid adaptation to new classes or environments without collecting additional labeled data.