Effort, No Progress: Formulating Implicit Movements for Just-In-Time Instructions

Abstract

When observing the execution of manual tasks, instructors provide follow-up instructions instead of waiting for mistakes to occur or for users to explicitly ask for help [37]. We manually coded 500 video clips—captured from an egocentric view of the instructee—to identify the implicit cues that prompt these interventions. Our annotations revealed a systematic classification of uncertainty-insinuating cues expressed through hand and head movements [4]. Based on this finding, we propose a just-in-time instruction system that tracks user and object movements to emulate two groups of implicit cues: goal-agnostic cues and goal-dependent cues. We implemented three methods to encode the most frequently observed cues: for the first group, we detect repeated movements of the head or of the objects currently being manipulated; for the second group, we leverage measures of task progress and error to infer uncertainty. By identifying when progress is suboptimal, our system can infer that the user is uncertain and deliver timely instructions.