Emotional Risks and Ethical Challenges in AI Companionship

Abstract

What happens when people start treating AI less like an assistant and more like a companion? This thesis aim to address that gap in two parts. The first part presents the design and implementation of a data donation infrastructure for collecting anonymous ChatGPT conversations. The infrastructure contains a browser extension that captures structured conversations directly from the ChatGPT interface and a Streamlit-based donation platform that allows participants to review, redact, export, and optionally donate their data. These tools were designed to work hand in hand to give participants full control over the data they are donating, from curating an export JSON file to modifying individual chats to redact or delete information they are uncomfortable sharing. The second part presents methodological and empirical approach to developing data analysis pipeline. Since the primary donation workflow resulted in no submissions, the empirical analysis pivots to WildChat as a large-scale source of real-world ChatGPT interactions. A weakly supervised, embedding-based detector was developed to identify companionship- related conversation, which was then used to construct a 25,000-conversation corpus. This corpus was analyzed using thematic coding, topic discovery, and measures of relationship dynamics at the conversation level. Within the constructed corpus, companionship-related use is not dominated by explicitly romantic or sexual exchange, but rather, it is primarily organized around emotionally supportive talk, daily-life conversation, and role-play or fictional interaction. The analysis also suggests that assistants frequently supply the interactional signals that make these conversations feel relational, including affectionate language, continuity cues, and question asking. These findings imply that the most socially consequential risks of AI companionship may arise not only from extreme edge cases, but also from routine, emotionally supportive interactions.