Intelligent support for Human Oversight: Integrating Reinforcement Learning with Gaze Simulation to Personalize Highlighting

Interfaces for human oversight must effectively support users’ situation awareness under time-critical conditions. We explore reinforcement learning (RL)-based UI adaptation to personalize alerting strategies that balance the benefits of highlighting critical events against the cognitive costs of interruptions. To enable learning without real-world deployment, we integrate models of users’ gaze behavior to simulate attentional dynamics during monitoring. Using a delivery-drone oversight scenario, we present initial results suggesting that RL-based highlighting can outperform static, rule-based approaches and discuss challenges of intelligent oversight support.

💡 Research Summary

The paper addresses the challenge of supporting human operators who must maintain situation awareness while supervising multiple autonomous systems under time‑critical conditions. Highlighting critical information on a monitoring interface can improve awareness, but excessive or poorly timed highlights lead to alarm fatigue, attentional overload, and reduced performance. To balance the benefits of alerts against their cognitive costs, the authors propose an adaptive highlighting approach that learns a personalized policy through reinforcement learning (RL) and integrates a simulated model of human visual attention (gaze).

The authors use a delivery‑drone oversight scenario as a testbed. Four drones are displayed on a dashboard, each with eight attributes (e.g., velocity, battery level, rotor status). The interface must convey the current attribute values (the “attribute state”) while also maintaining an estimate of the operator’s belief about those values (the “user state”). A binary vector represents which attribute icons are currently highlighted. The RL agent’s action consists of selecting a subset of icons to highlight at each time step.

To train the agent without costly real‑world user studies, the authors embed a gaze simulation based on the temporal saliency model TASED‑Net, which they fine‑tuned on prior eye‑tracking data from drone‑monitoring tasks. At each simulation step the model receives a rendered image of the interface (including any highlights) and outputs a probability distribution over all icons indicating where the user is likely to look next. A single icon is sampled from this distribution to represent the user’s next fixation; the corresponding attribute value is then copied into the user’s belief state, while all other beliefs decay implicitly because the underlying drone attributes continue to evolve.

The environment is formalized as a Markov Decision Process (MDP). The state combines the actual attribute values, the user’s belief, and the current highlights. Transition dynamics consist of (1) deterministic evolution of drone attributes, (2) application of the agent’s highlight decision, and (3) stochastic update of the user belief via the gaze model. The reward function has two components: (i) a weighted L1 error between the true attribute values and the user’s belief, reflecting the cost of inaccurate situation awareness, and (ii) a fixed penalty H for each highlighted icon, representing the attentional cost and risk of alarm fatigue. The total reward is the negative sum of these terms, so maximizing reward encourages the agent to highlight only those attributes that are both important (high weight) and poorly known to the user, while keeping the number of highlights low.

The authors train the policy using Proximal Policy Optimization (PPO), a state‑of‑the‑art policy gradient algorithm. They set a relatively high highlight penalty (H = 500) to force the agent to consider long‑term benefits of a highlight rather than immediate error reduction. Training hyper‑parameters follow standard PPO settings with minor adjustments; full details are provided in an appendix.

Qualitative evaluation shows that the learned policy can outperform a naïve rule‑based baseline that highlights every critical event for a fixed duration (e.g., five seconds). In a representative scenario, one drone experiences high wind speed while simultaneously recovering a rotor failure. The policy chooses to highlight the rotor icon; the simulated user first looks at wind speed (due to residual gaze probability) and then at the highlighted rotor, after which the policy removes all highlights. A rule‑based system would have continued to highlight wind speed unnecessarily, potentially distracting the operator. However, the authors also report failure cases where the policy does not highlight a genuinely critical attribute because the gaze model predicts low fixation probability or the highlight penalty is too high. This underscores the sensitivity of the learned behavior to the fidelity of the gaze simulation and the chosen cost parameters.

The paper concludes that integrating human visual‑attention models into RL environments is a promising avenue for creating adaptive, personalized oversight interfaces. Nevertheless, several open challenges remain: (1) validating the approach with real users to ensure that simulated gaze behavior matches actual operator behavior, (2) systematically studying how different gaze‑model fidelities affect learning efficiency and policy quality, (3) extending the reward formulation to incorporate higher‑order situation‑awareness concepts (e.g., Level 3 SA, predictive understanding), and (4) exploring multimodal alerts (auditory, haptic) and safety‑critical constraints for deployment in real‑world supervisory control systems. The authors view this work as a first step toward intelligent, user‑centric monitoring tools for increasingly autonomous AI systems.