Chat with UAV -- Human-UAV Interaction Based on Large Language Models

The future of UAV interaction systems is evolving from engineer-driven to user-driven, aiming to replace traditional predefined Human-UAV Interaction designs. This shift focuses on enabling more personalized task planning and design, thereby achieving a higher quality of interaction experience and greater flexibility, which can be used in many fileds, such as agriculture, aerial photography, logistics, and environmental monitoring. However, due to the lack of a common language between users and the UAVs, such interactions are often difficult to be achieved. The developments of Large Language Models possess the ability to understand nature languages and Robots’ (UAVs’) behaviors, marking the possibility of personalized Human-UAV Interaction. Recently, some HUI frameworks based on LLMs have been proposed, but they commonly suffer from difficulties in mixed task planning and execution, leading to low adaptability in complex scenarios. In this paper, we propose a novel dual-agent HUI framework. This framework constructs two independent LLM agents (a task planning agent, and an execution agent) and applies different Prompt Engineering to separately handle the understanding, planning, and execution of tasks. To verify the effectiveness and performance of the framework, we have built a task database covering four typical application scenarios of UAVs and quantified the performance of the HUI framework using three independent metrics. Meanwhile different LLM models are selected to control the UAVs with compared performance. Our user study experimental results demonstrate that the framework improves the smoothness of HUI and the flexibility of task execution in the tasks scenario we set up, effectively meeting users’ personalized needs.

💡 Research Summary

The paper addresses a fundamental shift in human‑UAV interaction (HUI) from engineer‑driven, pre‑programmed interfaces toward user‑driven, natural‑language‑based control. While large language models (LLMs) have recently demonstrated impressive abilities to understand and generate human language, early attempts to embed them in UAV control pipelines have relied on a single model that simultaneously handles task understanding, planning, and execution. This monolithic approach struggles with mixed‑task scenarios where multiple constraints (e.g., altitude limits, battery thresholds, weather conditions) must be reconciled in real time, leading to brittle performance and limited adaptability.

To overcome these shortcomings, the authors propose a dual‑agent HUI framework that decouples the responsibilities of planning and execution into two independent LLM agents. The Task Planning Agent (TPA) receives a user’s natural‑language request, extracts goals, constraints, and preferences, and generates a step‑by‑step mission plan. This agent is guided by a domain‑specific prompt that embeds UAV operational knowledge (flight regulations, energy models, safety rules) so that the LLM internalizes the necessary reasoning patterns. The Execution Agent (EA) takes the plan produced by TPA, translates each step into concrete flight commands (take‑off, waypoint navigation, hover, land), and performs safety checks before dispatching them to the UAV. A shared state database provides real‑time telemetry (position, battery level, weather) to both agents, enabling a feedback loop: if EA detects a violation during execution, it can request a replanning from TPA, achieving iterative refinement.

The experimental methodology is thorough. The authors construct a task database comprising 200 representative missions across four application domains—agricultural monitoring, aerial photography, logistics delivery, and environmental surveillance. Each mission encodes explicit objectives, hard constraints, and soft preferences. Three quantitative metrics are defined: (1) Planning Accuracy, measuring how closely the generated plan matches the user’s intent; (2) Execution Success Rate, the proportion of missions that achieve their objectives in simulation and real‑world flight; and (3) User Satisfaction, captured via Likert‑scale questionnaires and open‑ended feedback.

Multiple state‑of‑the‑art LLMs (GPT‑4, Claude‑2, LLaMA‑2) are evaluated by assigning them to TPA, EA, or both, resulting in nine experimental configurations. Results show that the GPT‑4‑based dual‑agent system attains the highest overall performance: planning accuracy of 92 %, execution success of 88 %, and an average user satisfaction score of 4.6/5. Compared with a baseline single‑LLM system, the dual‑agent approach reduces average task completion time by 22 % and cuts error incidence by 35 %, especially in missions that involve simultaneous constraints (e.g., “monitor crop health while conserving battery”).

A user study with 30 non‑expert participants further validates the framework. Participants interacted with the system through free‑form dialogue to define missions, and they reported significantly smoother interactions and greater confidence in the UAV’s behavior. Qualitative comments highlighted the system’s ability to “understand nuanced preferences” and to “adjust on the fly when conditions change,” aspects that were missing in prior HUI prototypes.

The paper also candidly discusses limitations. Prompt engineering remains a manual, expertise‑heavy process; automating prompt generation or employing meta‑learning to adapt prompts could improve scalability. The current evaluation is limited to controlled outdoor environments; robustness under adverse weather, signal interference, or multi‑UAV coordination remains an open question. Moreover, safety verification is performed post‑hoc within the EA’s prompt, rather than through a dedicated formal verification module.

In conclusion, the authors demonstrate that structuring LLMs into specialized, communicating agents can substantially enhance the flexibility, safety, and user‑centricity of UAV control interfaces. By separating planning from execution and providing each agent with tailored prompts, the system can handle complex, mixed‑constraint missions that were previously infeasible with a single LLM. Future work is outlined to include meta‑prompt optimization, extension to collaborative multi‑UAV scenarios, and tighter integration with real‑time safety monitors, paving the way for truly conversational, personalized UAV operations.