Steering LLMs via Scalable Interactive Oversight

As Large Language Models increasingly automate complex, long-horizon tasks such as \emph{vibe coding}, a supervision gap has emerged. While models excel at execution, users often struggle to guide them effectively due to insufficient domain expertise, the difficulty of articulating precise intent, and the inability to reliably validate complex outputs. It presents a critical challenge in scalable oversight: enabling humans to responsibly steer AI systems on tasks that surpass their own ability to specify or verify. To tackle this, we propose Scalable Interactive Oversight, a framework that decomposes complex intent into a recursive tree of manageable decisions to amplify human supervision. Rather than relying on open-ended prompting, our system elicits low-burden feedback at each node and recursively aggregates these signals into precise global guidance. Validated in web development task, our framework enables non-experts to produce expert-level Product Requirement Documents, achieving a 54% improvement in alignment. Crucially, we demonstrate that this framework can be optimized via Reinforcement Learning using only online user feedback, offering a practical pathway for maintaining human control as AI scales.

💡 Research Summary

The paper addresses the emerging “supervision gap” that arises when large language models (LLMs) become strong executors of complex, long‑horizon tasks while human users remain weak supervisors due to limited domain knowledge, time, and cognitive bandwidth. To bridge this gap, the authors propose a Scalable Interactive Oversight (SIO) framework that decomposes a user’s high‑level intent into a recursive decision tree. At each leaf node, the system asks low‑burden, closed‑form questions (selection or ranking) and the user responds with simple signals such as preference, “don’t care”, or “don’t know”. These responses are aggregated into a cumulative preference state, which updates the tree and guides subsequent interactions. The loop continues until all nodes are resolved, after which a Product Requirements Document (PRD) is generated from the accumulated preferences.

The authors evaluate SIO using the “sandwich” protocol, which positions a non‑expert user, the LLM, and an expert evaluator in a three‑role setting. Real‑world websites are crawled to construct ground‑truth PRDs, and 37 test cases are created. Baselines include direct PRD generation with existing vibe‑coding tools (Claude‑Code, Gemini‑CLI) and vanilla multi‑turn free‑form dialogue with strong models (GPT‑5, Claude‑Sonnet‑4.5, Gemini‑2.5‑Pro). Across all PRD modules (product overview, core functionality, non‑functional requirements, business rules, UX design), SIO achieves a 54 % improvement in alignment scores relative to the baselines, demonstrating that structured, low‑effort feedback can dramatically increase the fidelity of model outputs to user intent.

Beyond inference, the paper shows that the interaction logs can serve as reward signals for reinforcement learning. By training the model with online user feedback alone, the system learns to pose better questions and integrate preferences more effectively, yielding an additional ~12 % gain over non‑interactive baselines. This confirms that interactive supervision not only guides a single generation but also produces high‑quality supervision data for continual model improvement.

Key contributions are: (1) formalizing the supervision asymmetry between weak human supervisors and strong LLM executors; (2) introducing the SIO framework that recursively decomposes intent and amplifies weak human signals through preference propagation; (3) demonstrating that interactive supervision can be leveraged for reinforcement learning, enabling models to self‑optimize their alignment strategies from real‑time user feedback. The work provides a practical pathway for maintaining human control as AI systems scale to ever more complex tasks, making advanced LLM capabilities accessible to non‑experts while preserving safety and alignment.