Learning to See from Procedural 3D

Abstract

Synthetic data is often considered to be a subpar substitute for real data. In this thesis, we challenge this assumption by demonstrating the effectiveness of procedural synthetic data for perception and representation learning. In the first half of this thesis, we use controllable procedural generation to discover the optimal properties of synthetic stereo depth data, and apply these insights to develop a novel stereo dataset, WMGStereo-150k. Our procedural data is 80 times more sample-efficient than the classic stereo dataset SceneFlow and enables zero-shot generalization to the real world. In the second half of this thesis, we learn general image representations from procedural 3D data. We introduce ProcRep, a method to pretrain image encoders on multitask dense prediction using only synthetic data. For semantic segmentation, ProcRep demonstrates better performance than standard ImageNet-pretrained backbones, despite using over 10 times less data. Our key insight is that not all aspects of realism are important for learning. By sacrificing certain dimensions of realism for more diversity, we can learn more efficiently than from the real world. We aim to generalize this principle to a wide range of practical tasks in computer vision and robotics.