Monotonicity as an Architectural Bias for Robust Language Models

Large language models (LLMs) are known to exhibit brittle behavior under adversarial prompts and jailbreak attacks, even after extensive alignment and fine-tuning. This fragility reflects a broader challenge of modern neural language models: small, carefully structured perturbations in high-dimensional input spaces can induce large and unpredictable changes in internal semantic representations and output. We investigate monotonicity as an architectural inductive bias for improving the robustness of Transformer-based language models. Monotonicity constrains semantic transformations so that strengthening information, evidence, or constraints cannot lead to regressions in the corresponding internal representations. Such order-preserving behavior has long been exploited in control and safety-critical systems to simplify reasoning and improve robustness, but has traditionally been viewed as incompatible with the expressivity required by neural language models. We show that this trade-off is not inherent. By enforcing monotonicity selectively in the feed-forward sublayers of sequence-to-sequence Transformers – while leaving attention mechanisms unconstrained – we obtain monotone language models that preserve the performance of their pretrained counterparts. This architectural separation allows negation, contradiction, and contextual interactions to be introduced explicitly through attention, while ensuring that subsequent semantic refinement is order-preserving. Empirically, monotonicity substantially improves robustness: adversarial attack success rates drop from approximately 69% to 19%, while standard summarization performance degrades only marginally.

💡 Research Summary

The paper tackles the well‑known brittleness of large language models (LLMs) when faced with adversarial prompts and jailbreak attacks by introducing a structural inductive bias: monotonicity. Monotonicity, in this context, means that strengthening information—adding evidence, tightening constraints, or increasing risk—cannot cause a regression in the model’s internal semantic representations. While monotonic systems have long been used in control theory to guarantee stability, they have traditionally been considered incompatible with the expressive power required by neural language models.

The authors propose a selective enforcement of monotonicity only within the feed‑forward (FFN) sublayers of a sequence‑to‑sequence Transformer, leaving attention mechanisms completely unconstrained. This architectural split allows attention to handle explicit logical operations such as negation, contradiction, or exception, while the subsequent FFN refinement is forced to be order‑preserving.

Formally, a fixed semantic‑axis matrix A∈ℝ^{p×d} is introduced, and each hidden state h∈ℝ^{d} is mapped to semantic coordinates s=Ah. The FFN update is defined as
 F(h) = h + A^{†} g(Ah)
where A^{†} is a right‑inverse of A (e.g., the Moore‑Penrose pseudoinverse) and g:ℝ^{p}→ℝ^{p} is a coordinate‑wise monotone MLP (non‑decreasing activations and element‑wise non‑negative weight matrices). The paper proves that if g is monotone then F is A‑monotone, and that monotone functions are closed under composition, guaranteeing that the entire FFN stack remains monotone. Non‑negativity of weights is enforced via a Softplus re‑parameterization (W=log(1+exp(V))) so that constraints are satisfied throughout training without explicit projection.

Experimentally, the authors instantiate this design on the T5‑small architecture (≈60 M parameters, 6 encoder and 6 decoder layers). Roughly 40 % of the parameters (the FFN sublayers) are replaced with the monotone formulation; attention, residual connections, and layer normalizations stay unchanged. The matrix A is either learned from curated ordered prompt pairs (linear probes) or taken from a small set of fixed classifier directions, and is fixed before training. The monotone MLP g is initialized near zero so that the model starts close to the original T5.

Results show that standard summarization metrics (ROUGE, BLEU) degrade only marginally (≈0.2‑0.5 % absolute drop), indicating that the expressive capacity of the model is largely preserved. In stark contrast, robustness improves dramatically: across a suite of ten publicly known jailbreak prompts and custom adversarial suffixes, the attack success rate falls from ~69 % for the baseline T5 to ~19 % for the monotone version. Qualitative analyses of ordered prompts (e.g., “The medication has side effects” ⪯ “The medication has severe side effects”) demonstrate that the monotone model’s internal representations respect the partial order, whereas the baseline exhibits regressions.

The paper also confirms that leaving attention unconstrained allows the model to represent explicit logical negation and contradictions, while the monotone FFN prevents implicit feature cancellation that would otherwise cause semantic reversals. Training overhead is minimal; the Softplus re‑parameterization eliminates the need for projection steps, and the overall training time remains comparable to the original T5.

In conclusion, the study provides strong evidence that selective monotonicity is a viable architectural bias for LLMs: it preserves task performance while substantially increasing resistance to adversarial manipulation. This approach differs from reactive defenses (adversarial training, output filtering) by embedding safety directly into the model’s structure. Future work may explore scaling the method to larger models, alternative semantic‑axis constructions, and formal verification of monotone behavior in broader NLP tasks.