Compositional Generalization in Systematic Tasks

Abstract

Language acquisition and reasoning, two skills fundamental to human intelligence, both require internalizing and applying a collection of rules and structures derived from a limited set of examples. Crucially, this requires not only the ability to identify important structures in those examples, but also the ability to systematically combine learned structures in novel combinations, or systematicity. Interestingly, despite recent successes in the sphere of language generation, statistical learners like today's foundation models still fail to consistently reason systematically, suggesting a possible incompatibility between the current learning paradigm or model architectures and the acquisition of systematicity. We aim to develop a better understanding of the extent to which statistical learners are capable of generalizing to novel combinations, which we call compositional generalization, in systematic tasks. To that end, we identify some ways in which model architecture or optimization choices can encourage the inductive biases needed for models to generalize compositionally, as well as challenges in more realistic task settings.