How Users Perceive Mixed-Initiative AI: Attitudes Toward Assistance in Problem Solving

In mixed-initiative systems, the mode of AI assistance delivery can be as consequential as the assistance itself. We investigated two assistance delivery modes: on-demand help (users request via Button) and pre-scheduled help (assistance delivered at user-selected intervals, with user actions resetting the Timer). To evaluate these modes, we selected Rush Hour puzzles as the human-AI collaborative task because they capture elements of real-world problem solving such as analysis, resource management, and decision-making under constraints. To enhance ecological validity, we imposed monetary costs for both time and AI assistance, simulating scenarios where people must balance implicit or explicit trade-offs such as time pressure, financial limitations, or opportunity costs. Although task performance was comparable across modes, participants who used the pre-scheduled (Timer) mode reported more positive perceptions of the AI, even when their ending budget was low. This suggests that assistance delivery mode can shape user experience independent of task outcomes, indicating that human-AI systems may need to consider how AI assistance is delivered alongside improving task performance.

💡 Research Summary

The paper investigates how the timing of AI assistance influences user experience in mixed‑initiative human‑AI collaboration. Two canonical assistance‑timing mechanisms were compared: an on‑demand “Button” mode, where users explicitly request help and specify the number of AI‑generated moves, and a “Timer” mode, where users pre‑define an inactivity threshold and a move count; the AI then automatically intervenes after the user remains idle for the specified period. Both modes were embedded in a controlled study using the Rush Hour sliding‑block puzzle, a task chosen because it captures key aspects of real‑world problem solving—multi‑step planning, time pressure, resource management, and the need to decide when to seek external guidance.

To simulate realistic constraints, participants were given a monetary budget that decayed at a rate of 1 cent per second, and each AI intervention cost an additional 5 cents. This design mirrors scenarios such as time‑sensitive work, financial limits, or opportunity costs in domains ranging from software debugging to medical decision support. Participants (N = 66) were randomly assigned to either the Button or Timer condition and could not switch modes during a puzzle. Objective performance metrics—completion time, total budget spent, and number of AI moves used—showed no statistically significant differences between the two groups, indicating that the timing mechanism did not affect raw problem‑solving efficiency.

Subjective measures, however, revealed a clear divergence. Post‑task questionnaires assessed perceived AI competence, helpfulness, autonomy intrusion, and overall satisfaction. Users in the Timer condition consistently reported higher ratings across all dimensions, even when their final budget was near zero. The authors interpret this as evidence that the pre‑scheduled, system‑initiated assistance—though still bounded by user‑defined parameters—creates a sense of predictability and controlled agency. Participants felt the AI intervened at appropriate moments without them having to constantly monitor or decide when to press a button, thereby preserving a feeling of “guided autonomy.”

The study contributes three main insights. First, it introduces an experimental framework that decouples when assistance is delivered from what assistance is delivered, allowing researchers to isolate timing effects. Second, it demonstrates that user experience—encompassing perceived autonomy, AI competence, and helpfulness—is as critical as performance outcomes in shaping preferences for AI assistance. Third, it provides empirical support for inactivity‑triggered (timer‑based) assistance as a viable design alternative for collaborative AI in multi‑step reasoning tasks.

The authors situate their work within a broader literature on mixed‑initiative interaction, adjustable autonomy, and agency in human‑AI systems. They reference Horvitz’s “elegant coupling” concept, Parasuraman’s levels of automation, and recent adjustable‑autonomy frameworks that emphasize dynamic initiative allocation. By focusing on a sequential, repeatable task rather than a single‑shot interaction, the paper extends prior research that often conflated timing with content or examined only one‑time assistance events.

Practical implications are highlighted for domains where timing is crucial. In navigation, health reminders, or educational tutoring, offering users the ability to set inactivity thresholds can improve satisfaction without sacrificing performance. Designers are encouraged to treat timing, initiative, and control mechanisms as first‑class design concerns, alongside algorithmic improvements. Moreover, evaluation protocols should incorporate experiential metrics (e.g., perceived control, trust) in addition to traditional performance metrics to capture the full impact of mixed‑initiative designs.

In summary, even under identical cost and capability conditions, the mode of assistance delivery significantly shapes users’ attitudes toward AI. System‑initiated, timer‑based help—when parameterized by the user—enhances perceived helpfulness and maintains a sense of agency, suggesting that future human‑AI systems should carefully consider how assistance is offered, not just what assistance is offered.