Lookahead-then-Verify: Reliable Constrained Decoding for Diffusion LLMs under Context-Free Grammars

Diffusion Large Language Models (dLLMs) have demonstrated promising generative capabilities and are increasingly used to produce formal languages defined by context-free grammars, such as source code and chemical expressions. However, as probabilistic models, they still struggle to generate syntactically valid outputs reliably. A natural and promising direction to address this issue is to adapt constrained decoding techniques to enforce grammatical correctness during generation. However, applying these techniques faces two primary obstacles. On the one hand, the non-autoregressive nature of dLLMs renders most existing constrained decoding approaches inapplicable. On the other hand, current approaches specifically designed for dLLMs may allow intermediate outputs that are impossible to complete into valid sentences, which significantly limits their reliability in practice. To address these challenges, we present LAVE, a constrained decoding approach specifically designed for dLLMs. Our approach leverages a key property of dLLMs, namely their ability to predict token distributions for all positions in parallel during each forward pass. Whenever a new token is proposed by model, LAVE performs lookahead using these distributions to efficiently and reliably verify the validity of the proposed token. This design ensures reliable constraints by reliably preserving the potential for intermediate outputs to be extended into valid sentences. Extensive experiments across four widely used dLLMs and three representative benchmarks demonstrate that LAVE consistently outperforms existing baselines and achieves substantial improvements in syntactic correctness, while incurring negligible runtime overhead.

💡 Research Summary

Diffusion Large Language Models (dLLMs) have emerged as a powerful alternative to traditional autoregressive language models, offering parallel token prediction and fast inference. However, when tasked with generating formal languages defined by context‑free grammars (CFGs)—such as source code or chemical SMILES strings—dLLMs still suffer from high syntax error rates. Existing constrained decoding techniques are largely designed for autoregressive models, relying on left‑to‑right generation and complete prefixes, which makes them incompatible with the non‑autoregressive nature of dLLMs. Recent dLLM‑specific approaches attempt to enforce CFG constraints but often produce “unreliable” intermediate outputs that cannot be extended to any valid sentence, undermining practical reliability.

The paper introduces LAVE (Lookahead‑then‑Verify), a constrained decoding framework tailored to dLLMs. The key insight is that dLLMs predict probability distributions for all masked positions simultaneously at each denoising step. When the model proposes a token for a specific mask, LAVE performs a lookahead: it samples a small number (N) of complete prefixes by filling the remaining masks according to the model’s distributions. These sampled prefixes are then fed to a conventional CFG parser (e.g., Earley or CYK) to check whether at least one of them can be further extended into a valid full sentence. If such an extendable prefix exists, the proposed token is accepted; otherwise it is rejected and the model must propose an alternative. This verification guarantees the “reliable constraint” property: every intermediate output remains extendable to at least one valid sentence.

Two practical challenges are addressed. First, exhaustive enumeration of all possible completions is infeasible due to the combinatorial explosion of mask positions and vocabulary size. Empirical studies show that sampling as few as 5–10 prefixes is sufficient to make accurate accept/reject decisions, keeping the verification overhead negligible. Second, the iterative propose‑verify loop can become stuck when the model repeatedly suggests tokens that fail verification. LAVE incorporates a lightweight recovery mechanism that slightly perturbs the current context—e.g., re‑ordering masks or re‑weighting candidate tokens—to escape such dead‑ends while preserving constraint reliability.

The authors formally define reliability: a constraint is reliable if at every decoding step the current partial output can still be completed into at least one valid sentence; otherwise it is unreliable. They demonstrate that prior dLLM‑specific methods (e.g., Mündler et al.) over‑approximate masked spans as having unlimited length, leading to acceptance of tokens that render the intermediate output impossible to finish correctly.

Extensive experiments are conducted on four state‑of‑the‑art dLLMs (Dream‑Coder, LLaDA2, Gemini Diffusion, Dream‑v0‑Instruct‑7B) across three representative benchmarks: HumanEval‑CPP (code generation), CodeXGLUE (multi‑language code tasks), and SMILES‑Gen (chemical expression generation). LAVE consistently outperforms baselines (including DINGO, the prior CFG‑based dLLM decoder, and unconstrained generation). Syntax correctness rates improve dramatically, often reaching near‑perfect levels (e.g., 99.2 % on HumanEval‑CPP). Functional correctness, measured by test case pass rates, also sees multi‑fold gains. Runtime overhead is minimal, averaging a 1–3 % increase over unconstrained decoding, confirming the method’s practicality for real‑time applications.

Ablation studies reveal that both the lookahead sampling and the recovery component contribute positively: reducing N below 3 degrades reliability, while disabling recovery leads to higher failure rates in complex contexts. Sensitivity analysis shows the approach is robust to variations in the attempt budget τ and other hyperparameters.

In summary, LAVE leverages the parallel prediction capability of diffusion models to implement a “propose‑lookahead‑verify‑recover” loop that enforces CFG constraints reliably during non‑autoregressive generation. It eliminates the unreliability of previous methods, achieves near‑zero syntax errors, improves functional outcomes, and does so with negligible computational cost, representing a significant step forward for deploying dLLMs in safety‑critical or formally‑specified generation tasks.