Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models

Numerous applications of large language models (LLMs) rely on their ability to perform step-by-step reasoning. However, the reasoning behavior of LLMs remains poorly understood, posing challenges to research, development, and safety. To address this gap, we introduce landscape of thoughts (LoT), the first landscape visualization tool to inspect the reasoning trajectories with certain reasoning methods on any multi-choice dataset. We represent the textual states in a trajectory as numerical features that quantify the states’ distances to the answer choices. These features are then visualized in two-dimensional plots using t-SNE. Qualitative and quantitative analysis with the landscape of thoughts effectively distinguishes between strong and weak models, correct and incorrect answers, as well as different reasoning tasks. It also uncovers undesirable reasoning patterns, such as low consistency and high uncertainty. Additionally, users can adapt LoT to a model that predicts the property they observe. We showcase this advantage by adapting LoT to a lightweight verifier that evaluates the correctness of trajectories. Empirically, this verifier boosts the reasoning accuracy and the test-time scaling effect. The code is publicly available at: https://github.com/tmlr-group/landscape-of-thoughts.

💡 Research Summary

Paper Overview
The authors introduce “Landscape of Thoughts” (LoT), a visualization and analysis framework designed to make the step‑by‑step reasoning processes of large language models (LLMs) transparent, scalable, and quantitatively measurable. While chain‑of‑thought prompting and related techniques have shown that LLMs can solve complex multi‑choice problems, the internal dynamics of their reasoning remain opaque. Existing debugging practices rely on manual inspection of generated thoughts, which is neither scalable nor suitable for dataset‑level conclusions. LoT fills this gap by converting textual intermediate states into numerical features that capture each state’s relative distance to every answer choice, projecting these high‑dimensional features into a two‑dimensional space with t‑SNE, and visualizing the resulting density maps (landscapes).

Feature Construction
For a given question x and a trajectory of thoughts t₁,…,tᵢ, the state sᵢ =