System-1/2 Safe Autonomy

Abstract

This thesis studies the integration of foundation-model semantics into safe robotic autonomy through a System 1/System 2 perspective. The core idea is that semantic models should not directly choose robot actions. Instead, they should modify the robot's internal representation of the world, while structured planning mechanisms remain responsible for safe execution. The main implemented contribution of the thesis is a recursively feasible safe exploration backbone, reimplemented and extended in a ROS~2-based C++/Python system. This backbone includes frontier-biased target selection, graph-maintenance machinery, and map-growth-based termination, and is validated in simulation under geometric safety constraints. The broader architecture is designed to support semantic hazard insertion, semantic revision, and memory-guided navigation. A major outcome of the thesis, however, is that the full real-world SYS1/SYS2 pipeline remains far from robust deployment. Bringing together a vision-language model, grounding pipeline, SLAM system, and recursively feasible planner on a physical Unitree go2 quadruped exposed substantial bottlenecks, including semantic latency, semantic mislocalization, redundancy in semantic detections, segmentation failures, odometry instability, compute constraints, and nominal motion-stack fragility. These issues show that the challenge is not only whether foundation models can recognize meaningful semantic conditions, but whether their outputs can be grounded, localized, and acted on reliably in a real robotic system. The thesis therefore makes two contributions. First, it provides a working recursive-feasibility exploration backbone and a concrete architectural formulation for semantic safe autonomy. Second, it offers an empirical critique of the systems bottlenecks that currently limit this approach in practice. More broadly, the thesis argues that semantically capable safe autonomy is a promising direction, but still requires major advances in state estimation, embodied affordance reasoning, semantic memory consistency, system latency, and lower-level safety mechanisms before it can be deployed robustly in the real world.