Lang2Manip: A Tool for LLM-Based Symbolic-to-Geometric Planning for Manipulation

Simulation is essential for developing robotic manipulation systems, particularly for task and motion planning (TAMP), where symbolic reasoning interfaces with geometric, kinematic, and physics-based execution. Recent advances in Large Language Models (LLMs) enable robots to generate symbolic plans from natural language, yet executing these plans in simulation often requires robot-specific engineering or planner-dependent integration. In this work, we present a unified pipeline that connects an LLM-based symbolic planner with the Kautham motion planning framework to achieve generalizable, robot-agnostic symbolic-to-geometric manipulation. Kautham provides ROS-compatible support for a wide range of industrial manipulators and offers geometric, kinodynamic, physics-driven, and constraint-based motion planning under a single interface. Our system converts language instructions into symbolic actions and computes and executes collision-free trajectories using any of Kautham’s planners without additional coding. The result is a flexible and scalable tool for language-driven TAMP that is generalized across robots, planning modalities, and manipulation tasks.

💡 Research Summary

Lang2Manip presents a unified pipeline that bridges large language model (LLM)‑generated symbolic task plans with the Kautham motion‑planning framework, enabling robot‑agnostic, language‑driven manipulation in simulation. The authors begin by highlighting the central role of simulation in task‑and‑motion planning (TAMP) and noting that existing pipelines (e.g., PyBullet‑based extensions) are tightly coupled to specific robot URDFs, scene configurations, or particular planners. This coupling hampers scalability when new manipulators or planning paradigms are introduced.

To address this, the paper leverages Kautham, an open‑source ROS‑compatible platform built on OMPL that supports a wide variety of industrial arms (KUKA, ABB, UR5, Franka Panda, etc.) and offers geometric, kinodynamic, physics‑driven, and constraint‑based planners through a single, standardized interface. Kautham’s XML problem files describe robots, obstacles, initial and goal configurations, and planner choices, while its ROS package (kautham_ros) provides real‑time integration and visualization via Qt5 and RViz.

The Lang2Manip architecture consists of two layers. The LLM‑guided symbolic planning layer defines a fixed action grammar A = {pick, place, move, push}. Each action follows a template a(o, p, r, κ) where o is the target object, p optional pose parameters, r refinement (e.g., grasp direction), and κ the preferred planner. The LLM receives a composite prompt containing (1) the user’s natural‑language task description, (2) a system prompt that specifies the action schema and required JSON output format, and (3) a textualized state observation generated by Kautham. The state observation converts the current robot joint values, obstacle poses, and spatial relationships into a natural‑language description, allowing the LLM to reason about the environment. Using GPT‑4 in experiments (but model‑agnostic), the LLM outputs a JSON sequence of actions that are robot‑independent.

The Kautham‑based motion planning and execution layer parses the JSON plan, grounds each symbolic action using grasp‑planning and inverse‑kinematics plugins, and translates it into concrete joint‑space goals. The chosen planner (e.g., RRT, RRTConnect) is invoked via the XML problem definition, producing a collision‑free trajectory. Results are visualized in the Kautham GUI and RViz, facilitating debugging and analysis. Because the planner and robot are selected at runtime via the XML file, the same symbolic plan can be executed on any supported manipulator with any of Kautham’s planners without additional code changes.

Key contributions include: (1) a robot‑agnostic interface that automatically maps LLM‑generated symbolic actions to geometric motion planning, (2) a reusable textual state observation that enables LLMs to incorporate up‑to‑date scene information, and (3) demonstration of seamless swapping among multiple planners and manipulators within a single framework.

The paper’s limitations are notable. Experimental validation is minimal; quantitative metrics such as success rate, planning time, or robustness to dynamic obstacles are not reported, leaving open the question of how the system performs on complex, multi‑object rearrangement tasks. The approach also relies heavily on the correctness of LLM‑produced parameters (e.g., grasp direction, target pose) without an explicit verification or correction step. Finally, while Kautham offers a rich set of sampling‑based planners, integration with newer optimization‑based or learning‑based planners is not explored.

In summary, Lang2Manip offers a practical solution to the “semantic gap” between high‑level language instructions and low‑level motion execution. By decoupling symbolic planning from robot specifics and leveraging Kautham’s modular planner suite, it provides a scalable foundation for research in language‑conditioned TAMP. Future work that adds rigorous benchmarking, dynamic environment handling, and adaptive planner selection would further solidify its role as a standard tool for language‑driven robotic manipulation.