Reasoning by Commented Code for Table Question Answering

Table Question Answering (TableQA) poses a significant challenge for large language models (LLMs) because conventional linearization of tables often disrupts the two-dimensional relationships intrinsic to structured data. Existing methods, which depend on end-to-end answer generation or single-line program queries, typically exhibit limited numerical accuracy and reduced interpretability. This work introduces a commented, step-by-step code-generation framework that incorporates explicit reasoning into the Python program-generation process. The approach decomposes TableQA reasoning into multi-line executable programs with concise natural language comments, thereby promoting clearer reasoning and increasing the likelihood of generating correct code. On the WikiTableQuestions benchmark, the proposed method achieves 70.9% accuracy using Qwen2.5-Coder-7B-Instruct, surpassing the Repanda baseline (67.6%). Integrating the proposed framework with a robust end-to-end TableQA model via a lightweight answer-selection mechanism yields further improvements. This combined approach achieves up to 84.3% accuracy on the WikiTableQuestions benchmark.

💡 Research Summary

Table Question Answering (TableQA) remains a challenging task for large language models (LLMs) because tables encode information in a two‑dimensional layout that does not align with the linear token‑by‑token generation process of LLMs. Existing solutions fall into three broad families: (1) text‑to‑SQL approaches that translate a natural‑language question into a SQL query, (2) end‑to‑end models that directly generate an answer from a serialized table‑question pair, and (3) program‑based methods that produce executable code (often a single Pandas expression). Each of these families suffers from important drawbacks. Text‑to‑SQL assumes homogeneous column types and struggles with noisy real‑world tables that contain mixed numeric strings, annotations, or irregular formatting. End‑to‑end models are vulnerable to context‑length issues such as the “lost‑in‑the‑middle” phenomenon and cannot guarantee exact arithmetic because numerical operations are simulated through token generation. Program‑based methods like Repanda improve efficiency by generating a single‑line Pandas query, yet they sacrifice interpretability: the reasoning steps (filtering, parsing, aggregation) are implicit and cannot be inspected or debugged.

The paper proposes a new framework called Commented Code for TableQA, which integrates explicit reasoning directly into the code generation process. When given a table and a question, the LLM is prompted to output a single Python function that (a) begins with a mandatory one‑line # PLAN comment summarizing the overall strategy, (b) optionally includes additional step comments such as # FILTER, # PARSING, # AGGREGATE only when needed, and (c) follows each comment with the corresponding executable Pandas operation. The function must return the final answer in a list. This design forces the model to articulate a high‑level plan before writing code, to decompose the problem into clear sub‑steps, and to handle common sources of table noise (duplicate column names, mixed numeric formats, date parsing, missing values) using safe Pandas utilities like pd.to_numeric(..., errors='coerce'). By grounding reasoning in executable code, numerical calculations are performed deterministically, eliminating the approximation errors typical of pure language‑model generation.

The authors evaluate the approach on the WikiTableQuestions benchmark, a widely used dataset containing heterogeneous tables and complex reasoning requirements. Using the open‑source Qwen2.5‑Coder‑7B‑Instruct model with the proposed prompt template, the system achieves 70.9 % accuracy, surpassing the Repanda baseline (67.6 %). To further exploit complementary strengths, the authors introduce a lightweight answer‑selection module that combines the commented‑code model with a strong end‑to‑end TableQA system (e.g., Table‑R1). The selector evaluates the answers from both models and chooses the most plausible one, raising the overall accuracy to 84.3 %. Error analysis shows that the code‑based component virtually eliminates arithmetic mistakes; most residual errors stem from missing or sub‑optimal planning comments, which are easier to diagnose thanks to the explicit stepwise structure.

Key contributions include: (1) a structured instruction format that mandates a planning comment and aligns each reasoning step with a concrete Pandas operation; (2) a single‑pass generation pipeline that retains token efficiency while providing the interpretability of multi‑step reasoning; (3) empirical evidence that explicit, executable reasoning improves both numerical reliability and overall performance, especially when combined with existing end‑to‑end models.

The paper also discusses limitations and future work. The current implementation focuses on Pandas and single‑table queries; extending the framework to handle joins, multi‑table databases, or non‑tabular inputs (e.g., OCR‑extracted tables) will require additional language constructs. Moreover, while the prompt design works well for a 7‑billion‑parameter model, scaling to larger LLMs may demand more sophisticated token budgeting to keep the comment‑code block within context limits. Future directions include (a) integrating automatic error‑diagnosis loops that re‑generate or patch code when execution fails, (b) exploring richer planning languages (e.g., pseudo‑SQL + comments) for more complex relational reasoning, and (c) applying the approach to other structured‑data tasks such as knowledge‑base question answering or spreadsheet automation.

In summary, Commented Code for TableQA demonstrates that embedding explicit, human‑readable reasoning directly into executable code bridges the gap between interpretability and numerical correctness. The method outperforms prior single‑line code generators, and when paired with powerful end‑to‑end models, it reaches state‑of‑the‑art performance on WikiTableQuestions. This work suggests a promising path forward for TableQA: a tight coupling of planning, stepwise reasoning, and deterministic execution within a compact, single‑pass generation framework.