Rethinking Robot Planning: Using LLMs for Task and Motion Planning

Abstract

The idea of a general-purpose household robot, one that can listen and understand natural language instructions like ``clear the table", before executing that task, remains one of the central challenges of modern robotics. Building robots capable of this requires solving Task and Motion Planning (TAMP) problems, where a planner needs to jointly reason about what actions to take and how to physically execute them. Purely symbolic planners handle these problems poorly, and valid high-level symbolic plans frequently fail when geometric constraints are taken into account. This thesis investigates how large language models (LLMs) can be incorporated into the planning loop to produce high-level plans that are more robust to these geometric failures. We develop and evaluate various LLM-based approaches within the SeSamE framework, and introduce average sampling attempts per abstract action as a new metric for plan quality. Our key finding is that failure feedback is essential. Without it, LLMs produce the same geometrically naive plans as classical symbolic planners. Approaches that incorporate failure feedback are comparable to oracle-level performance in smaller environments. An ablation study of our failure-feedback approach shows that geometric information tends to improve performance for packing-style problems, while symbolic state information can surprisingly hurt that performance, suggesting that not all failure context is equally useful. LLM-generated heuristics, despite theoretical promise, do not produce reliable results in our environments. Overall, our results demonstrate that LLMs can serve as effective high-level planners for TAMP when integrated in an intelligent manner.