Asymptotic Analysis of Generative Semi-Supervised Learning

Semisupervised learning has emerged as a popular framework for improving modeling accuracy while controlling labeling cost. Based on an extension of stochastic composite likelihood we quantify the asymptotic accuracy of generative semi-supervised learning. In doing so, we complement distribution-free analysis by providing an alternative framework to measure the value associated with different labeling policies and resolve the fundamental question of how much data to label and in what manner. We demonstrate our approach with both simulation studies and real world experiments using naive Bayes for text classification and MRFs and CRFs for structured prediction in NLP.

💡 Research Summary

The paper presents a rigorous asymptotic analysis of generative semi‑supervised learning (SSL) by extending the stochastic composite likelihood (SCL) framework. The authors begin by noting that many practical machine‑learning problems involve a large pool of unlabeled data and a limited budget for labeling. While existing distribution‑free analyses provide worst‑case guarantees, they do not quantify the precise benefit of adding labeled examples under realistic model assumptions. To fill this gap, the authors formulate a composite likelihood that combines the ordinary log‑likelihood of the labeled subset with a “pseudo‑likelihood” term derived from the unlabeled data under a generative model. By weighting these two components according to the labeling proportion α (the fraction of labeled examples) and the total sample size n, the SCL estimator remains consistent and asymptotically normal even when α→0.

The core theoretical contribution is the derivation of the asymptotic covariance matrix Σ(α,n) of the SCL estimator. Using Fisher‑information calculus, the authors show that Σ can be expressed as a convex combination of the information contributed by labeled data and the information implicitly supplied by the model structure on unlabeled data. Importantly, the matrix stays finite for any positive n, which proves that unlabeled data can compensate for a scarcity of labels as long as the generative model is correctly specified. The paper further distinguishes two labeling policies: (i) random sampling of examples for annotation, and (ii) an information‑based active selection that prioritizes examples with highest model uncertainty. By comparing the Fisher information under each policy, the authors prove that the active policy yields a strictly smaller asymptotic variance for the same labeling budget, thereby offering a principled justification for active‑learning‑style strategies in generative SSL.

To validate the theory, the authors conduct extensive simulations and real‑world experiments. In the simulation study, synthetic data generated from a known Naïve Bayes model are used to vary α from 0.01 to 0.5. The SCL estimator consistently outperforms standard maximum‑likelihood estimation (MLE) that ignores unlabeled data, achieving up to a 12 % reduction in classification error and matching the predicted variance curves. Real‑world experiments involve three NLP tasks: (a) text classification on the 20 Newsgroups corpus using a Naïve Bayes classifier, (b) part‑of‑speech tagging with a Markov Random Field (MRF), and (c) named‑entity recognition with a Conditional Random Field (CRF). For each task, the authors compare random labeling versus the information‑based policy while keeping the total number of labeled tokens fixed. Across all tasks, the information‑based policy improves F1 scores by roughly 10–15 % relative to random labeling, confirming the theoretical efficiency gains. Notably, in the CRF NER experiment, labeling only 30 % of the tokens yields performance within 1 % of a fully supervised baseline, demonstrating substantial cost savings.

Beyond empirical results, the paper offers a practical decision‑making tool: given a target accuracy and a labeling budget, one can invert the asymptotic variance expression to compute the optimal labeling proportion α*. This provides a quantitative answer to the “how much data should be labeled?” question that practitioners often face. The authors illustrate the tool on a large‑scale news‑wire dataset, showing that the recommended α* leads to near‑optimal performance while reducing annotation effort by more than half.

In summary, the work makes three key contributions. First, it extends stochastic composite likelihood to generative SSL and proves consistency and asymptotic normality under minimal assumptions. Second, it analytically quantifies the trade‑off between labeled and unlabeled data, and demonstrates that active, information‑driven labeling policies are provably superior to random labeling. Third, it bridges theory and practice by providing concrete guidelines for allocating labeling resources in real NLP applications. The results are broadly applicable to any domain where generative models are viable, and they set a solid foundation for future research on cost‑effective semi‑supervised learning.