Execution Trace Analysis for Understanding AI Agent Behavior: A Case Study

Abstract

In our research, we identify the drawbacks of currently applied outcome-based approaches to LLM agent evaluation and propose to adopt the log-based one. Due to the development and improvement of the agent itself from a predictor into a highly sophisticated system capable of multiple-step reasoning, tool usage, and interacting with the environment, more complex metrics should be applied to such a model’s assessment. In addition, the outcome-based approaches may not provide insight regarding why the AI system succeeds/fails, which means that the problems related to the agent scaffold, the environment, or other determiners might not be identified. This paper develops an extensive framework for logs’ analysis in the context of the HAL leaderboard. The current experiments are performed under carefully designed settings, which include changes in model parameters, scaffolds, and reasoning modes and SciCode dataset for code generation tasks. Log-based approach enables the creation of the classification taxonomy of errors, covering such classes as model errors, scaffold errors, infrastructure errors, and error propagation. As demonstrated by the experiment results, the impact of the scaffold complexity on the agent’s accuracy is rather high. Higher scaffold complexity leads to the increased costs due to excessive planning, a higher number of steps needed and context accumulation, thus reducing the agent’s accuracy despite its strong model. In addition, a significant reasoning effort in combination with poorly designed scaffolds may have an adverse effect on performance. Therefore, the advantages of log-based analysis become evident.