Implicit Movement-Based Cues as a Trigger for Proactive AR Assistance

Abstract

Augmented reality (AR) assistants leverage spatial understanding and multi-modal input in order to deliver context-aware guidance and instruction. Some systems are proactive, able to fluidly anticipate user needs and predict user goals. However, a key challenge is determining when to intervene, as intervening too early or late can disturb user focus. To address this issue, analysis on the HoloAssist dataset discovered that implicit movement-based cues can serve as a basis for modeling user uncertainty in order to predict ideal moments of intervention. I present a novel AR system that continuously tracks user movement and intervenes at moments of high uncertainty based on these spatial cues. Through visual language model analysis and a decision making algorithm, the system detects movement cues and determines when to proactively give instruction through an AR interface. To evaluate this system, I conducted a within-subjects study (N = 16) involving four physical assembly tasks, measuring user performance and subjective experience. I compared the system against three alternative instructional conditions (Explicit-Asking, Pause, and Step Completion). Contrary to our initial hypothesis, the system was significantly more distracting than the other conditions. Since natural physical exploration heavily overlapped with the movement cues, the system triggered too often with unpredictable interventions, interrupting user focus. However, the system did not significantly degrade task completion time, knowledge retention, perceived workload, or user trust. I discuss the implications of these findings, ultimately proposing guidelines for designing more robust, proactive AR guidance systems.