Feeling the Future: Dexterous Bolt Manipulation in Robotics Using Optical Tactile Sensing

Abstract

Tactile sensing is an emerging technology that expands robotic understanding of physical interactions. Though optical sensing remains a leader in the field for its precision and low cost, tactile sensing methods allow for feature detection by measuring force related features without requiring that the contact surfaces are directly visible to a camera. This is useful for environments in which the function of visual sensors becomes inhibited, such as in darkness or small manipulation tasks, where detailed features that classify an object may be more difficult to view optically. This project experiments exclusively with tactile sensing, without the aid of optical sensors, on an end effector for bolt manipulation tasks. The two experiments include (1) finding the block where an M8 bolt stands and navigating to the correct position for grasping it, and (2) aligning and placing the bolt into a threaded hole. The end effector is a redesign of the LEAP hand, fitted with three fingers optimized for weight and efficiency for M8 bolt manipulations. Two GelSight DIGIT sensors were integrated onto the end effector fingers for experimentation. The software infrastructure enables streaming synchronized tactile images and robot state data through the Deoxys framework. The preprocessing pipeline extracts scalar contact features from DIGIT deformation images, forming the input to two learning pipelines trained on teleoperated demonstrations. Through experimentation using a Franka Research 3 robot arm integrated with the end effector, the research found that tactile sensing improved the learning capabilities of the robot. Experiment 1 showed that even with a consistent start location of the robot arm, end effector, block, and bolt, the feedback from the tactile sensor improved the learning capabilities of the task with a validation loss ¼ of that without the tactile features. Experiment 2 showed that a logistic regression classifier trained on tactile contact features achieved 80% accuracy in detecting when the bolt was threaded, with a 96% success recall. These findings support further experimentation exclusively with tactile sensors and further development of the braille-analogous search system as a strategy for visually constrained manipulation tasks.