A Real-Time Virtual Reality Visualization and Control System for Boston Dynamics Spot

Abstract

This project combines the Boston Dynamics Spot robotic platform with virtual reality to develop a more intuitive remote inspection experience. Two means of visualizing camera data from Spot in real-time in a VR interface were developed and tested. The first approach uses point cloud data from the robot’s stereo cameras. Because of a bottleneck in data transmission latency, the second approach focuses on improved latency at the tradeoff of a slightly less natural two-dimensional display of data. This approach surrounds the human user in six planes that show the live JPEG-compressed image feed from each of the robot’s cameras: gripper, front right, front left, right, left, and back. The location of the user is tracked and sent back as movement commands to Spot; as the user moves in their space, the robot moves analogously in its space. The user can send additional commands, such as sitting and standing, via handheld controller buttons. Considerations for privacy in a remote inspection were also implemented, including features that protect sensitive information from being transmitted and ensure the robot can not approach certain areas. Possible applications for this project lie in its ability to facilitate more natural feeling remote inspections, including those of dangerous areas, spaces where direct access is difficult, and private or secure locations.