Learning to Predict Combinatorial Structures

The major challenge in designing a discriminative learning algorithm for predicting structured data is to address the computational issues arising from the exponential size of the output space. Existing algorithms make different assumptions to ensure efficient, polynomial time estimation of model parameters. For several combinatorial structures, including cycles, partially ordered sets, permutations and other graph classes, these assumptions do not hold. In this thesis, we address the problem of designing learning algorithms for predicting combinatorial structures by introducing two new assumptions: (i) The first assumption is that a particular counting problem can be solved efficiently. The consequence is a generalisation of the classical ridge regression for structured prediction. (ii) The second assumption is that a particular sampling problem can be solved efficiently. The consequence is a new technique for designing and analysing probabilistic structured prediction models. These results can be applied to solve several complex learning problems including but not limited to multi-label classification, multi-category hierarchical classification, and label ranking.

💡 Research Summary

The thesis “Learning to Predict Combinatorial Structures” tackles the fundamental difficulty of structured prediction when the output space grows exponentially with the problem size. Traditional discriminative methods such as structured SVMs, CRFs, and max‑margin Markov networks rely on assumptions that enable polynomial‑time inference; however, many combinatorial objects—cycles, partial orders, permutations, and various graph families—do not satisfy these assumptions, making exact learning intractable.

To overcome this barrier the author introduces two novel computational assumptions. The first, the Counting Assumption, posits that the total number of feasible structures for a given input can be computed efficiently (in polynomial time). Under this premise the author derives a Structured Ridge Regression framework that generalizes classical ridge regression to structured outputs. By using the exact count of all possible outputs, the expected loss and regularization terms can be evaluated without enumerating the exponential set, allowing closed‑form updates or efficient gradient‑based optimization. The thesis details linear model instantiations, online optimization schemes, and approximate decoding/enumeration techniques that preserve scalability. Empirical studies on multi‑label and hierarchical classification demonstrate that the counting‑based ridge regressor achieves higher accuracy and better generalization than conventional structured SVMs while retaining comparable computational cost.

The second, the Sampling Assumption, asserts that one can draw samples uniformly (or according to a known weighting) from the set of feasible structures in polynomial time. Leveraging this, the author builds a Probabilistic Structured Prediction approach based on log‑linear models. The key challenge in such models is the partition function and its gradient, which are intractable for exponential output spaces. By employing efficient samplers—constructed via Markov chain Monte Carlo (MCMC) techniques and meta‑chain constructions—the thesis provides unbiased estimators for the partition function and its gradient. A rigorous mixing‑time analysis guarantees that the samplers converge rapidly, and a reduction from counting to sampling further bridges the two assumptions. The resulting learning algorithm can be applied to label ranking, permutation prediction, and graph‑matching tasks. Experiments show that the sampling‑based probabilistic model not only matches or exceeds the predictive performance of state‑of‑the‑art methods but also reduces inference time dramatically.

Beyond the algorithmic contributions, the work offers a comprehensive theoretical treatment of learning complexity for combinatorial structures. Chapter 4 delineates hardness results, clarifies when efficient learning is possible, and formalizes the two assumptions as sufficient conditions for tractable learning. Subsequent chapters translate these conditions into concrete algorithms, provide detailed proofs of correctness, and discuss practical implementation issues such as kernelization, low‑dimensional embeddings, and online updates.

In summary, the thesis makes four major contributions: (1) it identifies counting and sampling as fundamental computational primitives that enable tractable learning for a broad class of combinatorial output spaces; (2) it introduces Structured Ridge Regression, a novel linear model that exploits exact counts to perform efficient discriminative learning; (3) it develops a sampling‑based probabilistic framework with provable MCMC guarantees for estimating partition functions and gradients; and (4) it validates both frameworks on real‑world problems, showing superior accuracy and scalability compared to existing structured prediction techniques. These results open new avenues for applying machine learning to complex combinatorial tasks such as graph generation, permutation‑based recommendation, and structured ranking, where traditional methods have previously been limited by computational infeasibility.