Toward Content-Aware Scene Retrieval Using Natural Language

Abstract

This thesis establishes a methodology for building natural language query-based retrieval systems for large datasets of 3D scenes. We present our results through the introduction of a proof-of-concept search engine, SUNCG Search, indexed on the SUNCG dataset of approximately forty-five thousand richly-annotated scenes. We find that for the retrieval tasks of dataset exploration and subset identification, our novel approach of identifying binary spatial relationships between objects greatly decreases the time users spend on these tasks. Additionally, we introduce unique solutions to information retrieval's prototypical difficulties -- result visualization and query interface design -- with specific application to the 3D scene use case.