Code It, See It, Touch It: Vision-Powered AR Programming Tool for Children

Abstract

Children increasingly interact with intelligent systems through Augmented Reality (AR), but few authoring tools enable them to create expressive experiences grounded in the physical world. We present an AR authoring tool that enables children to create interactive experiences by linking visual code blocks to physical objects recognized by vision-based AI models. The system is built on a block-based programming interface and leverages YOLOv11 to identify physical objects through the device camera. Building on the accessibility of block-based programming, our system encourages creativity and exploration by extending traditional visual authoring tools into the physical environment. Through user studies with 20 children aged 7–16 in the United States and Argentina, we found that children could successfully author interactive AR stories using object detection, creatively extending their environments into hybrid digital-physical play. We discuss the system design, implementation, and implications for educational AI tools that blend embodied programming with computer vision. Our findings suggest that the system supports playful, explorative learning, while also highlighting challenges in real-time object tracking accuracy and responsiveness.