Large Language Models Can Take False First Steps at Inference-time Planning

Large language models (LLMs) have been shown to acquire sequence-level planning abilities during training, yet their planning behavior exhibited at inference time often appears short-sighted and inconsistent with these capabilities. We propose a Bayesian account for this gap by grounding planning behavior in the evolving generative context: given the subtle differences between natural language and the language internalized by LLMs, accumulated self-generated context drives a planning-shift during inference and thereby creates the appearance of compromised planning behavior. We further validate the proposed model through two controlled experiments: a random-generation task demonstrating constrained planning under human prompts and increasing planning strength as self-generated context accumulates, and a Gaussian-sampling task showing reduced initial bias when conditioning on self-generated sequences. These findings provide a theoretical explanation along with empirical evidence for characterizing how LLMs plan ahead during inference.

💡 Research Summary

This paper tackles a puzzling discrepancy in large language models (LLMs): although they acquire sequence‑level planning abilities during pre‑training, their behavior at inference time often appears short‑sighted and inconsistent. The authors propose a Bayesian account that attributes this gap to the evolving generative context. In the Bayesian formulation, the posterior over a response sequence s given a prompt I and a domain prior ϕ is proportional to the product of a planning likelihood P(I | s, ϕ) and a domain prior P(s | ϕ). Human‑written prompts (I_C) are drawn from natural language, whereas the model’s internal language (I_M) differs subtly. Consequently, conditioning on I_C yields a higher entropy in the planning likelihood, allowing the prior to exert a stronger influence early in generation. This “prior‑biased” phase leads to short‑term planning that does not reliably predict later tokens.

As generation proceeds, the model’s self‑generated tokens become part of the context, effectively turning into a new I_M that increasingly resembles the model’s own distribution. This accumulation reduces the entropy of the planning likelihood, suppresses the prior, and shifts the model toward a likelihood‑driven plan. The authors term this dynamic transition a “planning shift,” followed by a gradual “planning convergence” as the model’s internal plan stabilizes. The theory predicts two observable signatures: (1) an early phase of short‑term planning that weakens with larger prediction horizons, and (2) a bias‑then‑debias trajectory where initial responses are biased by the prior but become unbiased as self‑generated context accumulates.

To validate these predictions, the paper presents two controlled experiments. Experiment 1 uses a random‑number generation task (simulating height guesses) with Llama‑3.1‑8B‑Instruct and Qwen‑2.5‑7B‑Instruct. Models generate 60 numeric estimates per trial, and embeddings of each token are extracted. Linear LASSO regressions predict future tokens from current embeddings at varying offsets Δt. Results show that embeddings capture a substantial proportion of variance for short offsets (high R²), confirming short‑term planning, while R² rises dramatically (up to ~0.8–1.0) as more self‑generated context accumulates, evidencing the predicted planning shift and convergence.

Experiment 2 introduces a Gaussian‑sampling task with known ground‑truth distribution, allowing assessment of bias. The models initially produce responses that deviate from the true distribution (bias), but as self‑generated sequences accumulate, the responses move toward the ground‑truth (debias). This demonstrates the bias‑then‑debias dynamics predicted by the Bayesian model.

Overall, the paper offers a coherent theoretical framework that reconciles LLMs’ training‑time planning capabilities with their inference‑time behavior. By highlighting the role of distribution mismatch between human prompts and the model’s internal language, and by showing how self‑generated context dynamically reshapes planning, the work provides actionable insights for prompt engineering, decoding strategies, and future model design aimed at mitigating short‑sightedness and enhancing long‑range coherence.