Extracting and Following Paths for Robust Relational Reasoning with Large Language Models

Large language models (LLMs) possess vast semantic knowledge but often struggle with complex reasoning tasks, particularly in relational reasoning problems such as kinship or spatial reasoning. In this paper, we present Path-of-Thoughts (PoT), a novel framework for solving relation reasoning that decomposes the task into three key stages: graph extraction, path identification, and reasoning. Unlike previous approaches, PoT efficiently extracts a reasoning graph that identifies crucial entities, relations, and attributes within the context. Subsequently, PoT identifies query-relevant reasoning paths within the graph, facilitating downstream reasoning of potential answers. Experimental evaluations across four datasets of relational reasoning demonstrate that PoT surpasses state-of-the-art baselines by a significant margin (up to 21.3%) without requiring fine-tuning or extensive LLM calls. Furthermore, unlike prior neuro-symbolic methods, PoT exhibits improved resilience against LLM extraction errors and input ambiguity by leveraging the compositional nature of graphs.

💡 Research Summary

The paper tackles a well‑known weakness of large language models (LLMs): despite their massive semantic knowledge, they often fail at multi‑hop relational reasoning tasks such as kinship inference or spatial navigation. To address this, the authors introduce Path‑of‑Thoughts (PoT), a three‑stage neuro‑symbolic framework that separates understanding from reasoning.

Stage 1 – Graph Extraction
Given a story (context + question), PoT prompts an LLM once to extract all entities, relations, and attributes as directed triples (head, relation, tail). The prompting design is heavily structured: sections are marked with special characters, output delimiters (brackets/parentheses) enforce a fixed schema, and the set of admissible relations is predefined. This reduces hallucinations, missing nodes, and inconsistent formatting, yielding a clean reasoning graph G = (N, E) that resembles a cognitive map of the problem.

Stage 2 – Path Identification
From the graph, PoT isolates the two entities mentioned in the question and enumerates every possible connecting path between them. Unlike prior methods that feed the whole graph into a chain‑of‑thought prompt, PoT extracts only the sub‑graph that is directly relevant to the query. Importantly, multiple independent paths are retained; each path becomes a separate reasoning unit. This “path‑wise independent reasoning” prevents a mistake on one route from contaminating the others and provides natural redundancy when the extracted graph contains errors.

Stage 3 – Reasoning
For each identified path PoT can employ either (a) an LLM with chain‑of‑thought or self‑consistency prompting, or (b) a symbolic solver such as Answer Set Programming (ASP). The symbolic option is especially powerful when domain‑specific logical rules are available (e.g., kinship composition rules). After processing all paths, PoT aggregates the results via majority voting or probabilistic weighting to produce the final answer.

Empirical Evaluation
The authors test PoT on four benchmark datasets: CLUTRR (kinship), StepGame, SPARTUN, and a fourth spatial reasoning set. Compared with strong baselines—including vanilla CoT, Tree‑of‑Thoughts, Graph‑of‑Thoughts, DSR‑LM, LINC, and other neuro‑symbolic pipelines—PoT achieves up to a 21.3 percentage‑point accuracy gain (average gain ≈ 12.7 pp). Notably, PoT requires no fine‑tuning and makes only a single LLM call for graph extraction, dramatically reducing computational overhead.

Robustness to Extraction Errors
To probe resilience, the authors inject systematic noise (flipped relations, omitted triples) into the input. Because PoT reasons over multiple independent paths, the performance drop stays under 5 % even under severe noise, whereas single‑chain methods degrade sharply. This demonstrates that the compositional graph representation and path‑wise reasoning effectively isolate and mitigate extraction errors.

Limitations and Future Work
The paper acknowledges two main limitations: (1) extracting nuanced or implicit relations (e.g., double negation, indirect references) remains challenging for the single‑call LLM extractor; (2) the number of possible paths can explode combinatorially with large entity sets, raising computational costs. The authors propose future directions such as multi‑shot prompting with self‑verification loops to improve extraction fidelity, and graph compression or heuristic path sampling to keep the search tractable.

Conclusion
Path‑of‑Thoughts offers a principled way to combine the linguistic flexibility of LLMs with the interpretability and compositionality of graph‑based symbolic reasoning. By explicitly constructing a reasoning graph, isolating query‑relevant sub‑graphs, and performing independent path‑wise inference, PoT achieves superior accuracy, robustness to noisy extractions, and efficiency without model fine‑tuning. The framework thus represents a significant step forward for reliable relational reasoning in LLM‑driven applications.