Defining and Evaluating Physical Safety for Large Language Models

Large Language Models (LLMs) are increasingly used to control robotic systems such as drones, but their risks of causing physical threats and harm in real-world applications remain unexplored. Our study addresses the critical gap in evaluating LLM physical safety by developing a comprehensive benchmark for drone control. We classify the physical safety risks of drones into four categories: (1) human-targeted threats, (2) object-targeted threats, (3) infrastructure attacks, and (4) regulatory violations. Our evaluation of mainstream LLMs reveals an undesirable trade-off between utility and safety, with models that excel in code generation often performing poorly in crucial safety aspects. Furthermore, while incorporating advanced prompt engineering techniques such as In-Context Learning and Chain-of-Thought can improve safety, these methods still struggle to identify unintentional attacks. In addition, larger models demonstrate better safety capabilities, particularly in refusing dangerous commands. Our findings and benchmark can facilitate the design and evaluation of physical safety for LLMs. The project page is available at huggingface.co/spaces/TrustSafeAI/LLM-physical-safety.

💡 Research Summary

The paper addresses a critical gap in the safety evaluation of large language models (LLMs) when they are used to control physical robotic systems, focusing on programmable drones as a representative case study. While LLMs have demonstrated impressive capabilities in code generation, reasoning, planning, and in‑context learning, their potential to cause real‑world physical harm has received little systematic scrutiny. To fill this void, the authors propose a comprehensive benchmark—“LLM Physical Safety Benchmark”—that quantifies the risk of drone‑related physical damage across four threat categories: (1) human‑targeted attacks, (2) object‑targeted attacks, (3) infrastructure attacks, and (4) violations of Federal Aviation Administration (FAA) regulations.

The benchmark consists of more than 400 curated prompts, each labeled according to one of four evaluation dimensions: Deliberate Attacks, Unintentional Attacks, Violation Instructions, and Utility Tasks. Deliberate attacks include direct command attacks, indirect command attacks, and code injection scenarios designed to test a model’s resistance to malicious intent. Unintentional attacks capture inadvertent hazards arising from user misunderstandings, misleading instructions, high‑risk scenarios, or oversight of safety‑critical context. Violation instructions probe compliance with FAA Part 107 rules such as no‑drone zones, excessive altitude or speed, and flight over people. Utility tasks assess basic drone control functions (take‑off, movement, path following, yaw control, target approach) to ensure that safety mechanisms do not cripple normal operation.

Evaluation proceeds in two stages using specialized AI judges. The Code Verification Judge checks whether generated Python code is syntactically correct, executable, and safe to run in the AirSim simulation environment. The Safety Evaluation Judge assesses higher‑level safety properties: whether the model refuses dangerous commands (Self‑Assurance), whether it modifies code to avoid hazards (Self‑Protection), and whether it complies with regulatory constraints (Regulatory Compliance). Six quantitative metrics are defined: Self‑Assurance, Avoid‑Collision, Regulatory Compliance, Code Fidelity, Instruction Understanding, and Utility.

The authors test a suite of mainstream LLMs, including OpenAI GPT‑3.5‑Turbo, Google Gemini‑Pro, Meta Llama‑2‑7B‑Chat, CodeLlama‑7B‑Instruct, Llama‑3‑8B‑Instruct, Mistral‑7B‑Instruct‑v0.2, and CodeQwen1.5‑7B‑Chat. Key findings are:

1. Utility‑Safety Trade‑off – Models that excel at code generation (high Code Fidelity and Utility scores) tend to exhibit lower safety scores, especially in human‑targeted and infrastructure attack categories. This reveals a tension between performance and safety that current alignment techniques have not fully resolved.

2. Prompt Engineering Impact – Applying In‑Context Learning (ICL) or Zero‑Shot Chain‑of‑Thought (ZS‑CoT) prompts yields modest improvements (≈5‑8 % increase) in overall safety metrics, with ICL slightly outperforming ZS‑CoT. However, both techniques struggle to detect unintentional attacks, indicating limited reasoning about downstream physical consequences.

3. Model Scale Effects – Larger models generally achieve higher Self‑Assurance and Regulatory Compliance scores, demonstrating a greater propensity to refuse or modify dangerous instructions. Nonetheless, scaling yields diminishing returns for certain categories (e.g., infrastructure attacks), suggesting that size alone cannot guarantee comprehensive safety.

4. Safety Behaviors – The benchmark captures two distinct safety‑oriented behaviors: “Self‑Assurance” (refusal to generate harmful code) and “Self‑Protection” (automatic code adjustments to mitigate risk). These behaviors are more prevalent in larger, instruction‑tuned models.

The paper contributes a rigorously constructed dataset, a dual‑judge evaluation pipeline, and a set of interpretable safety metrics that together enable systematic assessment of LLMs in physically interactive contexts. The authors argue that while current LLMs show promising safety‑refusal capabilities, significant gaps remain—particularly in anticipating and preventing inadvertent harms. Future work should explore advanced safety‑alignment methods, risk‑aware fine‑tuning, and continuous benchmarking as models evolve. By releasing the benchmark and associated resources, the study provides a valuable foundation for researchers, developers, and policymakers aiming to ensure that the rapid deployment of LLM‑driven robotic systems does not compromise public safety.