ConformalNL2LTL: Translating Natural Language Instructions into Temporal Logic Formulas with Conformal Correctness Guarantees

Linear Temporal Logic (LTL) is a widely used task specification language for autonomous systems. To mitigate the significant manual effort and expertise required to define LTL-encoded tasks, several methods have been proposed for translating Natural Language (NL) instructions into LTL formulas, which, however, lack correctness guarantees. To address this, we propose a new NL-to-LTL translation method, called ConformalNL2LTL that achieves user-defined translation success rates on unseen NL commands. Our method constructs LTL formulas iteratively by solving a sequence of open-vocabulary question-answering (QA) problems using large language models (LLMs). These QA tasks are handled collaboratively by a primary and an auxiliary model. The primary model answers each QA instance while quantifying uncertainty via conformal prediction; when it is insufficiently certain according to user-defined confidence thresholds, it requests assistance from the auxiliary model and, if necessary, from the user. We demonstrate theoretically and empirically that ConformalNL2LTL achieves the desired translation accuracy while minimizing user intervention.

💡 Research Summary

The paper addresses a critical gap in the emerging field of natural‑language‑to‑formal‑specification translation for autonomous robots. While recent works have demonstrated that large language models (LLMs) can generate Linear Temporal Logic (LTL) formulas from natural language (NL) commands, none of them provide any guarantee that the generated formula faithfully captures the intended task. ConformalNL2LTL is introduced as the first NL‑to‑LTL translation framework that can achieve a user‑specified success probability (1‑α) on previously unseen commands, by explicitly quantifying and managing the uncertainty of LLM outputs.

The core idea is to decompose the translation into a sequence of interdependent question‑answering (QA) steps. At each step k, a prompt ℓ(k) is constructed that contains the original NL instruction ξ, the robot’s skill set A, and the partially built LTL formula ϕ(k‑1). This prompt is fed to a primary LLM (ψp), which is asked to propose the next logical operator or atomic proposition (AP). Rather than taking a single deterministic answer, ψp is sampled multiple times (e.g., two or more) and the empirical frequencies of the responses are used as a coarse confidence estimate. Conformal Prediction (CP) is then applied: using a calibration set, a quantile (\bar q) and a semantic threshold ζ are computed such that the resulting prediction set C(ℓ(k), ψp) contains the correct answer with probability at least 1‑α. If C is a singleton, the answer is accepted; otherwise the framework invokes an auxiliary LLM (ψaux) and repeats the CP construction. The intersection C_inter = C(ℓ(k), ψp) ∩ C(ℓ(k), ψaux) is examined. If the intersection is a singleton, the answer is accepted; if it still contains multiple candidates, a human operator is asked to select the correct one. If the intersection is empty or does not contain the correct answer, the translation is declared a failure. The selected token s(k) is appended to the growing formula ϕ(k) and the process repeats until a termination token (e.g., “/”) appears.

Theoretical analysis shows that, under the standard exchangeability assumptions required for CP, the overall translation algorithm satisfies Pσ∼D(ϕ ≡ ξ) ≥ 1‑α, i.e., the probability that the produced LTL formula is semantically equivalent to the original NL task meets the user‑specified confidence level. Importantly, the method works for both closed‑source LLMs (e.g., GPT‑4) and open‑source models, because it relies only on sampled outputs, not on internal logits.

Empirical evaluation is extensive. The authors construct calibration and test datasets covering a variety of robot domains, skill sets, and NL task structures. ConformalNL2LTL achieves translation success rates of 99% on unseen commands while requiring human assistance in only 0.378% of cases. Compared to baseline NL‑to‑LTL methods that simply take the highest‑confidence LLM output, the proposed approach yields higher accuracy and dramatically lower human‑in‑the‑loop rates. The auxiliary LLM reduces the need for human help from 36.5% (primary‑only) to about 4%. Integration with the TL‑RRT* planner demonstrates that the higher‑quality specifications translate into more reliable robot plans, and the method remains competitive even when the CP assumptions are violated (distribution shift experiments).

In summary, ConformalNL2LTL introduces a principled uncertainty‑aware pipeline for NL‑to‑LTL translation, leveraging conformal prediction to provide statistical guarantees on correctness and to orchestrate collaboration between a primary LLM, an auxiliary LLM, and a human operator. The work bridges the gap between the flexibility of LLM‑driven language understanding and the rigor required for formal verification in autonomous systems, opening avenues for safe, interpretable, and scalable language‑guided robot programming.