Wink: Recovering from Misbehaviors in Coding Agents

Autonomous coding agents, powered by large language models (LLMs), are increasingly being adopted in the software industry to automate complex engineering tasks. However, these agents are prone to a wide range of misbehaviors, such as deviating from the user’s instructions, getting stuck in repetitive loops, or failing to use tools correctly. These failures disrupt the development workflow and often require resource-intensive manual intervention. In this paper, we present a system for automatically recovering from agentic misbehaviors at scale. We first introduce a taxonomy of misbehaviors grounded in an analysis of production traffic, identifying three primary categories: Specification Drift, Reasoning Problems, and Tool Call Failures, which we find occur in about 30% of all agent trajectories. To address these issues, we developed a lightweight, asynchronous self-intervention system named Wink. Wink observes agent trajectories and provides targeted course-correction guidance to nudge the agent back to a productive path. We evaluated our system on over 10,000 real world agent trajectories and found that it successfully resolves 90% of the misbehaviors that require a single intervention. Furthermore, a live A/B test in our production environment demonstrated that our system leads to a statistically significant reduction in Tool Call Failures, Tokens per Session and Engineer Interventions per Session. We present our experience designing and deploying this system, offering insights into the challenges of building resilient agentic systems at scale.

💡 Research Summary

The paper “Wink: Recovering from Misbehaviors in Coding Agents” presents a comprehensive study of failure modes in large‑language‑model (LLM) powered coding assistants and introduces a lightweight, asynchronous self‑intervention system called Wink that can automatically correct these failures at scale.
First, the authors analyze 42,920 production trajectories collected from a Visual Studio Code extension used by thousands of developers at Meta. From this data they derive a taxonomy of three high‑level misbehaviors that together affect roughly 29 % of all sessions: (1) Specification Drift – the agent deviates from the user’s explicit instructions, further split into “Did Not Follow Instructions” (DNF) and “Unrequested Changes” (UC); (2) Reasoning Problems – primarily infinite loops where the same tool is called repeatedly without progress; and (3) Tool Call Failures – malformed or missing parameters, calls to non‑existent tools, or failure to recover from a bad call.
To detect these issues in real time, the team builds binary classifiers for each category using state‑of‑the‑art LLMs (Claude Sonnet 4, GPT‑4o, Gemini 2.5, etc.). They prioritize precision (≥ 80 %) to avoid false alarms, and find Claude Sonnet 4 to be the most reliable across all categories. The classifiers ingest the full trajectory up to the current step (user messages, assistant reasoning, tool actions, and observations) and output a flag plus the identified misbehavior type.
Wink’s architecture runs this detection asynchronously alongside the main coding agent, which follows a ReACT‑style loop (reason → action → observation). After every k steps the observer checks whether a detection result is ready; if a misbehavior is reported, Wink retrieves a pre‑written corrective prompt from a store. The prompt consists of clear “DO” and “DON’T” instructions that nudge the agent to reflect, adjust its plan, and issue a correct tool call. This guidance is injected as a system‑message into the next input to the main LLM, ensuring the correction does not block the primary workflow and adds negligible latency.
The authors evaluate Wink on more than 10,000 real‑world trajectories. With a single intervention, 90 % of the misbehaviors are resolved; for cases requiring multiple interventions, the recovery rate remains around 80 %. An online A/B test in production shows statistically significant reductions in tool‑call failures, tokens per session, and engineer‑initiated interventions per session. A model‑version comparison reveals that while newer models (e.g., Claude Opus 4.5) improve some aspects of specification drift, they do not uniformly reduce all failure types, underscoring the need for system‑level safeguards like Wink.
The paper contributes several key insights: (i) a data‑driven taxonomy grounded in actual developer feedback; (ii) the feasibility of high‑precision LLM classifiers for real‑time error detection; (iii) an effective, non‑blocking self‑intervention loop that can be layered onto existing agent harnesses; and (iv) empirical evidence that such a loop yields measurable productivity gains at scale. Limitations include the focus on a single IDE extension and the reliance on pre‑crafted corrective prompts, which may need adaptation for other toolchains or domains. Nonetheless, Wink demonstrates a practical pathway to more resilient, autonomous coding agents and offers design principles that can be generalized to other AI‑driven automation systems.