TuBERT: Multimodal Speech Emotion Recognition for Real-Time Avatar Control

Abstract

This paper presents TuBERT: a lightweight, on-device, multimodal speech emotion recognition (SER) model designed for video game livestreams. Current real-time SER approaches either demand computing power beyond the reach of most consumers or rely solely on low-level audio descriptors. This limits their accessibility and accuracy. TuBERT alleviates both limitations by combining MFCC-based audio features with DistilBERT text embeddings derived from live Vosk speech transcription, training a single-GRU recurrent neural network on the MELD dataset to classify speech into five emotions: neutral, joy, surprise, anger, and sadness. On the MELD test set, TuBERT achieves a weighted accuracy of 60.11% and an F1-score of 58.35%. Fine-tuned on IEMOCAP, it reaches 60.29% weighted accuracy, matching models in existing literature that, unlike TuBERT, are not designed for real-time use. TuBERT runs at a real-time factor (RTF) of 0.425 on the CPU of a consumer Apple M4 chip, lower than any documented RTF for a real-time SER. TuBERT is then packaged in a GUI that enables streamers to assign custom sprites for display in a broadcast window to each detected emotion, requiring only a microphone as input. Qualitative feedback from users, suggestions for future research, and implications of TuBERT are also discussed.