In Tune with Intelligence: Design, Implementation, and Embodiment of a Real-Time Interactive AI System

Abstract

This thesis presents the design and evaluation of a real-time AI-driven desktop robot specialized in musical assistance. By integrating speech-to-speech AI with modular tools, translating natural language into system behavior, this system introduces an embodied interface unlike conventional voice models [20, 30, 23]. In this system, internal states are displayed through a dynamic visual agent to improve interaction transparency. The conversational model acts as a high-level orchestration layer that translates user intent into frontend actions, executing the use of domain-specific musical tools. The final presented design is deployed as both a desktop and a physically embedded platform functioning as a standalone unit. This enabled a comparative analysis of system performance metrics that revealed performance bottlenecks introduced by embedded deployment. Experimental results showed that conversational latency and backend processing remained largely consistent across platforms. This demonstrated that AI inference is not the limiting factor. However, significant degradation observed in visual responsiveness and tool execution demonstrated that performance drops emerged from frontend rendering constraints and execution environment limitations. This indicated that user-perceived responsiveness in real-time AI systems is governed primarily by interface realization rather than model capability. This work provides systems-level insights into embodied AI development. The challenges of embodying AI do not lie in intelligence alone, but rather in the coordination of distributed components operating under hardware constraints. Interaction models designed for desktop systems do not necessarily translate to embedded platforms and can require significant redesigns to maintain usability. This project establishes a foundation for real-time, physically embodied AI assistants and demonstrates the effectiveness of conversational AI as an orchestration layer for interactive systems.