Leveraging Large Language Models and Reinforcement Learning for Strategic Decision-Making in Complex Video Games

Abstract

This thesis presents a novel AI framework that integrates Large Language Models (LLMs) with Reinforcement Learning (RL) to enable autonomous gameplay in Pokémon Emerald, a complex and open-ended video game environment. We develop a modular agent architecture that separates perception, planning, memory, and action execution. The system is intended to operate with minimal domain-specific background knowledge. Vision-Language Models (VLMs) are used to interpret game frames and generate structured observations, which are stored and summarized within a dynamic memory module. These observations guide high-level strategic planning and policy refinement, enabling long-term decision-making. The agent executes actions in a game loop, coordinating real-time environment interaction, visualized through a web-based interface. Through this research, we explore the potential of LLMs for grounded understanding in interactive environments. We aim to demonstrate how language-guided reinforcement learning can be effectively integrated to solve nonlinear games. Our findings suggest that hybrid LLM-RL agents can outperform conventional RL methods to complete complex tasks. This offers new pathways for sample-efficient, interpretable, and generalizable agents.