ALRM: Agentic LLM for Robotic Manipulation

Large Language Models (LLMs) have recently empowered agentic frameworks to exhibit advanced reasoning and planning capabilities. However, their integration in robotic control pipelines remains limited in two aspects: (1) prior \ac{llm}-based approaches often lack modular, agentic execution mechanisms, limiting their ability to plan, reflect on outcomes, and revise actions in a closed-loop manner; and (2) existing benchmarks for manipulation tasks focus on low-level control and do not systematically evaluate multistep reasoning and linguistic variation. In this paper, we propose Agentic LLM for Robot Manipulation (ALRM), an LLM-driven agentic framework for robotic manipulation. ALRM integrates policy generation with agentic execution through a ReAct-style reasoning loop, supporting two complementary modes: Code-asPolicy (CaP) for direct executable control code generation, and Tool-as-Policy (TaP) for iterative planning and tool-based action execution. To enable systematic evaluation, we also introduce a novel simulation benchmark comprising 56 tasks across multiple environments, capturing linguistically diverse instructions. Experiments with ten LLMs demonstrate that ALRM provides a scalable, interpretable, and modular approach for bridging natural language reasoning with reliable robotic execution. Results reveal Claude-4.1-Opus as the top closed-source model and Falcon-H1-7B as the top open-source model under CaP.

💡 Research Summary

The paper introduces ALRM (Agentic LLM for Robotic Manipulation), a novel framework that integrates large language models (LLMs) into robotic manipulation pipelines through an agentic, closed‑loop architecture. Recognizing two major gaps in prior work—(1) the lack of modular, reflective execution mechanisms that allow LLM‑driven planners to monitor and revise actions in real time, and (2) the scarcity of benchmarks that test multistep reasoning and linguistic diversity—the authors design a system that addresses both.

ALRM is built around a ReAct‑style “thought‑action‑observation” loop. Three core components interact: (i) a Task Planner Agent that incrementally decomposes a high‑level natural‑language instruction into sub‑tasks, (ii) a Task Executor Agent that translates each sub‑task into concrete robot actions, and (iii) an API Server that connects the executor to a Gazebo‑ROS‑MoveIt simulation environment. The planner receives observations after each executor step, enabling dynamic replanning and error correction without restarting the whole task.

The framework supports two complementary execution modes. In Code‑as‑Policy (CaP), the LLM generates a single block of Python code that calls predefined robot‑control functions; the code is then run in one shot. CaP minimizes the number of LLM calls and works with models that lack tool‑calling capabilities, but it is brittle—any syntax or logic error aborts the entire sub‑task. In Tool‑as‑Policy (TaP), the LLM issues a sequence of tool calls (e.g., get_objects, pick, place, compute_grasp). Each call returns a structured observation that the planner can use to adjust subsequent calls, offering greater robustness at the cost of higher latency and reliance on accurate tool‑calling. Both modes are guided by high‑level templates and best‑practice prompts to improve consistency.

To evaluate ALRM, the authors create a new benchmark consisting of 56 manipulation tasks across three simulated environments (kitchen utensils, boxes, fruits). Tasks are stratified into four difficulty levels, ranging from simple lexical variations to high‑level reasoning (e.g., “pick the two fruits with the lowest calories”). Each scene contains more objects than needed, forcing the agent to identify and manipulate only the relevant items.

Ten state‑of‑the‑art LLMs—both closed‑source (Claude‑4.1‑Opus, GPT‑4, etc.) and open‑source (Falcon‑H1‑7B, DeepSeek‑V3.1, Llama‑2, etc.)—are tested under both CaP and TaP. Success rate and average execution latency are reported. Claude‑4.1‑Opus achieves the highest performance among closed‑source models, with 93.5 % success in TaP and 92.6 % in CaP. Among open‑source models, Falcon‑H1‑7B matches DeepSeek‑V3.1’s 84.3 % success in CaP while halving the latency, making it the most efficient open‑source option. Overall, CaP yields lower latency (≈30‑40 % faster) but is more prone to failure on complex reasoning tasks; TaP, with its step‑wise feedback, shows slightly higher robustness on the hardest tasks.

The analysis demonstrates that ALRM successfully combines modular policy generation, interpretability, and closed‑loop reflection, offering a flexible platform that can accommodate models with or without tool‑calling support. By providing both code‑generation and tool‑calling pathways, the framework maximizes applicability across a wide range of LLMs and robotic platforms. The authors conclude with future directions: transferring the system to real‑world robots, incorporating multimodal inputs (vision, speech), and extending to collaborative multi‑robot scenarios. Overall, ALRM represents a significant step toward general‑purpose, language‑driven robotic agents capable of complex, multi‑step manipulation.