Triggering the Inevitable: Automated Discovery of Rare Failures in Hybrid Robotic Control Systems

Abstract

Rare failures in hybrid robotic systems can occur as infrequently as once in 108 trajectories, making them invisible to standard testing yet catastrophic when encountered in deployment. Existing search methods are structurally rigid: they adapt parameters within a fixed architecture but cannot reconfigure their search logic in response to what they learn about a specific failure condition, leaving all prior approaches facing exponential sample complexity as the failure region shrinks. This paper ex- amines whether an AI agent, given only environment documentation and a failure specification, can overcome this rigidity to discover rare hybrid failures as reliably as a privileged Oracle with direct access to internal code. Our Agentic Failure Finder achieves a mean of 1.00 trajectory to failure across all three benchmark environments and 15 randomly sampled evaluation seeds (drawn from the range 0–1031), matching the Oracle benchmark without any privileged access. A non-agentic LLM baseline, by contrast, achieves 0% solve rate on the temporally-grounded ConveyorBelt failure despite having full access to the failure specification—demonstrating that comprehension alone is insufficient without iterative execution feedback. The one-time synthesis cost is recovered within a single evaluation seed on the ConveyorBelt and within five seeds on the Blocks domain. These results show that an execution-based agentic loop overcomes the structural rigidity of both blind random search and one-shot LLM synthesis, achieving Oracle-level reliability without Oracle-level access.