Adventures with CubBot: Pedagogically Constrained LLM Use and Learning Outcomes in an Undergraduate Machine Learning Course

Abstract

As generative artificial intelligence (genAI) becomes more integrated into computing education pipelines, it is necessary to investigate how students use these systems for their academic workflows and how such tools influence their learning outcomes. Existing literature focuses on these objectives within the context of introductory programming, namely within CS1 and CS2 course settings, rarely incorporating advanced CS classes--especially those that do not emphasize programming skill acquisition as a principal educational goal. This work bridges this gap by exploring the relationship between the use of genAI by students and their conceptual understanding of advanced mathematical and algorithmic principles in COS324, an undergraduate machine learning course. We address two primary research questions, both with regard to our specific course context: (1) how do students interact with genAI in an upper-level CS course that emphasizes conceptual mastery and for what objectives do they use these tools? And (2) how does such use of genAI relate to their learning outcomes?

To meet these aims, we deploy a course-specific, custom-interfaced large language model (LLM), CubBot, and examine collected chat logs. We further perform a randomized, controlled assessment with a subset of participants, where half maintain access to CubBot and half do not during the duration of the test, to assess the tool's influence on conceptual problem solving. We find that, within this particular course context, students mainly use the pedagogically constrained LLM for clarification of concepts and how to apply them, and that more use does not necessarily indicate better academic performance. Overall, this research contributes to the expanding body of literature on genAI in CS education by providing one of the first empirical investigations into the relationship of these tools with conceptual learning outcomes in a high-level CS course that does not mainly focus on coding proficiency, also leveraging actual interaction logs.