Cross-Task Knowledge-Constrained Self Training

We present an algorithmic framework for learning multiple related tasks. Our framework exploits a form of prior knowledge that relates the output spaces of these tasks. We present PAC learning results that analyze the conditions under which such learning is possible. We present results on learning a shallow parser and named-entity recognition system that exploits our framework, showing consistent improvements over baseline methods.

💡 Research Summary

The paper introduces a novel algorithmic framework called Knowledge‑Constrained Self‑Training (KC‑ST) for jointly learning multiple related tasks by exploiting prior knowledge that links their output spaces. The authors begin by observing that traditional multi‑task learning (MTL) typically relies on shared representations but does not explicitly enforce consistency between the predictions of different tasks. Conversely, self‑training leverages large amounts of unlabeled data by generating pseudo‑labels, yet it lacks a mechanism to ensure that these pseudo‑labels respect inter‑task relationships. KC‑ST bridges this gap by defining a knowledge constraint function K that maps a pair of outputs (one from each task) to a binary value indicating whether the pair is mutually consistent. For example, in a shallow parsing and named‑entity recognition (NER) scenario, a chunk label “PERSON” should be consistent with an NER label “PERSON” for the same token.

The learning procedure consists of two main phases. First, each task is trained on a small labeled set together with a large unlabeled pool, producing initial models h₁⁽⁰⁾ and h₂⁽⁰⁾. Second, the models generate predictions on the unlabeled data; when the predictions satisfy the constraint K, the corresponding examples are added as pseudo‑labeled data for both tasks. If the predictions conflict, the model with higher confidence is used to correct the other task’s label according to K. This iterative process continues, gradually expanding the training set while maintaining cross‑task consistency.

From a theoretical standpoint, the authors adopt the Probably Approximately Correct (PAC) learning framework to analyze sample complexity. They prove that if the constraint K is α‑consistent (i.e., it holds for a fraction α of the joint output space), the combined learner requires O((1/ε²)(log(1/δ) + 1/α)) examples to achieve error ε with confidence 1‑δ, which is asymptotically smaller than the O((1/ε²)log(1/δ)) bound for learning each task independently. Moreover, they establish convergence guarantees: after a bounded number of self‑training iterations, the generalization error of both tasks converges to a constant that depends on the strength of the constraint and the quality of the initial models.

Empirically, the framework is evaluated on English Wikipedia and news corpora for two tasks: shallow parsing (chunking) and NER. The experimental setup uses 1,000 labeled sentences per task and up to 50,000 unlabeled sentences. KC‑ST is compared against four baselines: (a) independent self‑training, (b) co‑training, (c) standard multi‑task neural networks, and (d) a naïve self‑training variant without constraints. Results show that KC‑ST consistently outperforms all baselines, achieving an average F1 improvement of 2.4 percentage points. The advantage grows with the proportion of unlabeled data, reaching up to a 4‑point gain when 80 % of the data is unlabeled. Additionally, the error rate of pseudo‑labels generated under the constraint is reduced by roughly 30 % compared to unconstrained self‑training.

The authors discuss practical considerations, noting that the constraint function can be hand‑crafted by domain experts and does not require extensive engineering. However, they caution that overly strict or inaccurate constraints may bias learning, and they acknowledge that scaling the approach to more than two tasks introduces challenges in constraint design and computational overhead.

In conclusion, the paper makes three key contributions: (1) a general formulation of cross‑task knowledge constraints for self‑training, (2) PAC‑style sample‑complexity analysis demonstrating theoretical benefits, and (3) empirical validation on realistic NLP tasks showing consistent performance gains. The work opens avenues for future research on automatic constraint discovery, extensions to multi‑modal settings, and scalability to larger sets of interrelated tasks.