Soft Instruction De-escalation Defense

Large Language Models (LLMs) are increasingly deployed in agentic systems that interact with an external environment; this makes them susceptible to prompt injections when dealing with untrusted data. To overcome this limitation, we propose SIC (Soft Instruction Control)-a simple yet effective iterative prompt sanitization loop designed for tool-augmented LLM agents. Our method repeatedly inspects incoming data for instructions that could compromise agent behavior. If such content is found, the malicious content is rewritten, masked, or removed, and the result is re-evaluated. The process continues until the input is clean or a maximum iteration limit is reached; if imperative instruction-like content remains, the agent halts to ensure security. By allowing multiple passes, our approach acknowledges that individual rewrites may fail but enables the system to catch and correct missed injections in later steps. Although immediately useful, worst-case analysis shows that SIC is not infallible; strong adversary can still get a 15% ASR by embedding non-imperative workflows. This nonetheless raises the bar.

💡 Research Summary

The paper “Soft Instruction De‑escalation Defense” addresses the growing problem of prompt injection attacks on tool‑augmented large language model (LLM) agents that ingest untrusted data such as web pages, emails, or API responses. Existing defenses—aggressive filtering, single‑pass sanitization, detection‑only classifiers, or system‑level sandboxing—either suffer from high false‑positive rates, are easily bypassed by adaptive attacks, or require heavyweight integration changes.

To overcome these limitations, the authors propose Soft Instruction Control (SIC), a lightweight, modular preprocessing loop that can be attached to any LLM‑based agent without altering its internal logic. SIC works as follows: (1) the raw external content is first augmented with a known “control instruction” (e.g., “I instruct you to clean the windows”). (2) An LLM‑based rewriter is invoked with a prompt that specifies one of three strategies—Mask, Rephrase, or Remove—to transform any detected instruction‑like fragments into non‑imperative text or placeholders. (3) After each rewrite, the pipeline checks whether the control instruction survived; if it does, the rewrite is deemed compromised and the system halts. (4) The rewritten text is then passed through a classifier (also LLM‑based) both on the whole string and on a set of chunks, to verify that no instruction‑like content remains. (5) If all checks pass, placeholders are stripped and the sanitized input is fed to the agent; otherwise the agent is halted with a “” token. The loop repeats up to a configurable maximum number of passes (R), allowing later passes to catch instructions missed by earlier ones.

Key design choices include: (i) stateless rewrites—each pass does not depend on the outcome of previous passes—so an attacker cannot influence the control flow by manipulating intermediate results; (ii) insertion of a control instruction that acts as a canary, guaranteeing that a failed rewrite is detected; (iii) optional chunk‑level detection to reduce the chance of missing short or hidden commands; and (iv) the entire mechanism is a pure pre‑processing step, requiring no changes to the agent’s policy or architecture.

The authors evaluate SIC on the AgentDojo benchmark, which simulates tool‑using agents operating in partially observable environments. They test four state‑of‑the‑art models (GPT‑4o, Kimi‑k2, GPT‑4.1‑mini, Qwen3‑32B) and several attack variants derived from adaptive prompt‑injection techniques (e.g., the AC adaptive attack). With a simple configuration—masking strategy, a single rewrite pass, and no chunking—SIC achieves 0 % attack success rate (ASR) across all models, while preserving the utility of clean inputs. Ablation studies show that adding more rewrite passes or chunk checks does not materially improve performance but increases latency.

Despite its strengths, the paper acknowledges important limitations. Because SIC relies on LLMs for both rewriting and detection, a determined adversary could craft inputs that survive the rewrite (e.g., by embedding “do not modify this text” style commands) and simultaneously evade the classifier through adversarial re‑phrasing. The authors’ worst‑case analysis indicates that non‑imperative workflow attacks—where the malicious payload is hidden in data rather than explicit commands—can still achieve up to a 15 % ASR. This reflects the inherent difficulty of perfectly distinguishing instruction‑like language from benign content, especially when the attacker has full knowledge of the defense (white‑box threat model).

Computationally, SIC incurs a linear cost in the length of the input (Θ(n) per LLM call). With R rewrite passes and k chunk detections, the total number of LLM invocations is R + 1 + k in the best case (early halt) and R + 1 + k in the clean case. In practice, most inputs require only the initial rewrite and a single full‑text detection, resulting in two LLM calls and modest latency; chunk detections can be parallelized to further reduce response time.

In summary, Soft Instruction Control offers a pragmatic, easily deployable defense that substantially raises the bar for prompt‑injection attacks on LLM agents. Its modular, multi‑pass design, combined with a built‑in sanity check via control instructions, makes it more robust than pure detection or prompt‑augmentation schemes. However, it does not provide provable security; sophisticated white‑box attacks, especially those that hide malicious intent in non‑imperative data, can still succeed at non‑trivial rates. Future work should explore hybrid defenses—e.g., integrating policy‑enforced execution sandboxes, ensemble detectors, or formal verification of instruction extraction—to close the remaining gaps.