Formal Semantic Control over Language Models

This thesis advances semantic representation learning to render language representations or models more semantically and geometrically interpretable, and to enable localised, quasi-symbolic, compositional control through deliberate shaping of their latent space geometry. We pursue this goal within a VAE framework, exploring two complementary research directions: (i) Sentence-level learning and control: disentangling and manipulating specific semantic features in the latent space to guide sentence generation, with explanatory text serving as the testbed; and (ii) Reasoning-level learning and control: isolating and steering inference behaviours in the latent space to control NLI. In this direction, we focus on Explanatory NLI tasks, in which two premises (explanations) are provided to infer a conclusion. The overarching objective is to move toward language models whose internal semantic representations can be systematically interpreted, precisely structured, and reliably directed. We introduce a set of novel theoretical frameworks and practical methodologies, together with corresponding experiments, to demonstrate that our approaches enhance both the interpretability and controllability of latent spaces for natural language across the thesis.

💡 Research Summary

The dissertation “Formal Semantic Control over Language Models” presents a comprehensive framework for endowing neural language models with interpretable, geometrically structured semantic representations and enabling fine‑grained, quasi‑symbolic control over both sentence generation and natural language inference (NLI). Working within a variational auto‑encoder (VAE) paradigm, the author pursues two complementary research tracks: (i) sentence‑level semantic disentanglement and manipulation, using explanatory text as a testbed, and (ii) reasoning‑level control of inference behavior, focusing on explanatory NLI where two premises (explanations) lead to a conclusion.

The background chapters review formal semantic theory (including logical and material inference), semantic role labeling (SRL) as an interface between formal and distributional semantics, and modern transformer‑based language models. The author then surveys VAE variants—standard VAE, normalising‑flow AE, and vector‑quantised VAE (VQ‑VAE)—and discusses evaluation metrics for latent space geometry and text generation.

Chapter 4 introduces “Formal Semantic Geometry” via an SRL‑conditional VAE. By conditioning the encoder on SRL tags, each semantic role is allocated a dedicated latent subspace. The author demonstrates that latent arithmetic (addition, subtraction, interpolation) respects role boundaries, allowing controlled manipulation of meaning while preserving grammaticality. Experiments include latent traversal, arithmetic, and guided traversal, with proxy metrics such as isotropy and disentanglement scores.

Chapter 5 focuses on “Formal Semantic Disentanglement”. Two encoder designs are compared: an unsupervised INN (U‑INN) and a cluster‑supervised INN (C‑INN). The models are trained to separate ARG0, predicate, and other role embeddings. Supervised clustering yields higher disentanglement scores (measured by K‑NN, SVM, Naïve Bayes) and better invertibility in both encoding and decoding. Geometric data augmentation further improves robustness.

Chapter 6 proposes a graph‑based syntax encoder coupled with a heterogeneous latent decoder, forming an end‑to‑end VAE that jointly learns syntactic structure and semantic content. The encoder embeds constituency parse trees enriched with SRL roles, while the decoder reconstructs text. Results show improved syntactic fidelity (lower perplexity, higher BLEU) and complementary gains on a mathematical derivation probing task.

Chapter 7 presents “Semantic Discretisation” through a T5‑based VQ‑VAE (T5VQ‑VAE). By quantising the latent space into discrete codebooks, the model achieves controllable semantic token manipulation. The architecture is evaluated on auto‑encoding, text‑transfer, and inference tasks, demonstrating that discrete latent symbols enable smoother latent arithmetic and better downstream performance.

Chapter 8 develops “Explanatory Inference Control via Inference Types”. The author defines a taxonomy of inference types (Entailment, Contradiction, Neutral, etc.) and embeds this information into the model using three injection strategies: encoder‑prefix, decoder‑prefix, and decoder‑end. Quantitative results on the EntailmentBank show that decoder‑prefix injection yields the highest accuracy, and latent space visualisations reveal clear separation of inference‑type clusters.

Chapter 9 extends this to “Explanatory Inference Disentanglement”. A rule‑based neural tangent kernel encodes formal NLI rules, which are then combined with the VAE to produce quasi‑symbolic reasoning. Experiments with decoder‑only models (e.g., Qwen2.5‑0.5B) and with BM25‑guided sampling on a math‑reasoning benchmark illustrate that rule‑guided latent manipulation improves both logical consistency and answer correctness.

The concluding chapter summarises the contributions: (1) a formal‑semantic geometry that aligns latent dimensions with logical roles, (2) disentanglement techniques that isolate semantic factors, (3) a graph‑based syntax representation that preserves structural information, (4) discrete latent spaces for symbolic control, and (5) quasi‑symbolic NLI frameworks that make inference processes transparent and steerable. The author also discusses ethical considerations, limitations (dependency on high‑quality annotations, computational overhead, domain‑specific evaluation), and future directions such as scaling to multilingual corpora, integrating with retrieval‑augmented generation, and developing interactive control interfaces.

Overall, the thesis makes a significant stride toward interpretable, controllable language models by marrying formal semantics with deep generative architectures, though broader empirical validation and efficiency optimisation remain open challenges.