Beyond Naïve Prompting: Strategies for Improved Context-aided Forecasting with LLMs

Real-world forecasting requires models to integrate not only historical data but also relevant contextual information provided in textual form. While large language models (LLMs) show promise for context-aided forecasting, critical challenges remain: we lack diagnostic tools to understand failure modes, performance remains far below their potential, and high computational costs limit practical deployment. We introduce a unified framework of four strategies that address these limitations along three orthogonal dimensions: model diagnostics, accuracy, and efficiency. Through extensive evaluation across model families from small open-source models to frontier models including Gemini, GPT, and Claude, we uncover both fundamental insights and practical solutions. Our findings span three key dimensions: diagnostic strategies reveal the “Execution Gap” where models correctly explain how context affects forecasts but fail to apply this reasoning; accuracy-focused strategies achieve substantial performance improvements of 25-50%; and efficiency-oriented approaches show that adaptive routing between small and large models can approach large model accuracy on average while significantly reducing inference costs. These orthogonal strategies can be flexibly integrated based on deployment constraints, providing practitioners with a comprehensive toolkit for practical LLM-based context-aided forecasting.

💡 Research Summary

The paper tackles the practical problem of context‑aided time‑series forecasting, where textual side‑information is essential for accurate predictions. While large language models (LLMs) have shown promise in this setting, naïve direct prompting (DP) suffers from three major limitations: (1) a lack of diagnostic tools to understand why a model fails, (2) sub‑optimal predictive accuracy, and (3) high inference costs that hinder deployment. To address these gaps, the authors propose a unified framework consisting of four complementary strategies—FxDP, CorDP, IC‑DP, and RouteDP—each targeting one of the three orthogonal dimensions of diagnostics, accuracy, and efficiency.

FxDP (Forecast Effect Explanation Direct Prompt) modifies the DP workflow by first asking the LLM to explain how the supplied textual context should affect the forecast, then to produce the numeric forecast. This two‑step chain‑of‑thought prompt separates verbal reasoning from numerical execution, enabling a fine‑grained diagnosis. The authors define three possible outcomes: (i) correct reasoning and correct execution, (ii) correct reasoning but incorrect execution (the “Execution Gap”), and (iii) incorrect reasoning. By augmenting the Context‑is‑Key (CiK) benchmark with ground‑truth effect annotations, they show that even frontier models (GPT‑5.2, Gemini‑2.5‑Pro, Claude‑Sonnet‑4.5) often explain the effect accurately (≈ 85‑92% agreement) yet still miss the correct numbers, confirming a pervasive Execution Gap that is independent of model size.

CorDP (Forecast Correction Direct Prompt) takes an existing probabilistic forecast (e.g., a sampled posterior) as input and asks the LLM to adjust it in light of the textual context. This “predict‑then‑correct” paradigm preserves the statistical strengths of the base forecaster while leveraging LLM reasoning to remove context‑induced bias. Empirically, CorDP improves the region‑of‑interest CRPS (R‑CRPS) by 25‑50% across all model scales, with the largest relative gains observed for models under 2.5 B parameters.

IC‑DP (In‑Context Direct Prompt) enriches the prompt with a few exemplars of similar context‑aided forecasting tasks. By providing concrete analogues, the LLM can perform case‑based reasoning, akin to meta‑learning. This strategy yields a 20‑30% reduction in forecast error for every model evaluated, and even large models see an additional 10‑15% boost, demonstrating that exemplar‑driven prompting is universally beneficial.

RouteDP (Direct Prompt with Model Routing) addresses efficiency. A lightweight router first assesses task difficulty using simple heuristics and a small labeled difficulty set. Easy tasks are delegated to a compact model (e.g., Qwen‑2.5‑0.5 B), while hard tasks are sent to a heavyweight model (e.g., GPT‑5.2). This dynamic allocation cuts average inference cost by 40‑60% while maintaining accuracy comparable to always using the large model; in some configurations, overall forecast quality improves by up to 46% due to better specialization.

The authors evaluate all four strategies on the CiK benchmark, which contains 71 carefully crafted context‑aided forecasting tasks across seven domains (climatology, economics, energy, mechanics, public safety, transportation, retail). The benchmark requires textual context for any chance of success, eliminating memorization shortcuts. Experiments span a wide model family spectrum: open‑source Qwen (0.5 B–72 B), Llama‑3 (1 B–405 B), and commercial frontier models (GPT‑5.2, Gemini‑2.5‑Pro, Claude‑Sonnet‑4.5). For each model, 25 Monte‑Carlo samples are drawn to produce probabilistic forecasts, and performance is measured with R‑CRPS, which respects region‑of‑interest windows and hard constraints described in the text.

Key findings include: (i) the Execution Gap is a universal phenomenon; (ii) FxDP provides a reliable diagnostic that can be combined with correction or exemplar strategies to close the gap; (iii) CorDP and IC‑DP each independently deliver 25‑50% accuracy gains, and together they achieve the highest improvements; (iv) RouteDP offers a practical path to deploy LLM‑based forecasting at scale, dramatically reducing compute while preserving or even enhancing accuracy; (v) the four strategies are orthogonal and can be mixed‑and‑matched depending on deployment constraints (e.g., latency, budget, interpretability).

In summary, the paper makes three major contributions: a diagnostic framework exposing the Execution Gap, two accuracy‑boosting prompting techniques (correction and in‑context exemplars), and an efficient routing mechanism for cost‑effective inference. The work opens several avenues for future research, such as learning‑based routers, tighter integration of explanations into the forecasting loss, and extension to multimodal context (images, tables). Overall, the study provides a comprehensive toolkit that moves LLM‑based context‑aided forecasting from a promising concept toward a deployable solution.