Confiding in AI: What Users Like, What Users Need, and the Limits of Preference

Abstract

Humans are developing more complex and persistent social and emotional relationships with AI. As these generative systems increasingly serve as sources of comfort, validation, and relational support, a tension arises: do the interactions users prefer also improve their wellbeing? Few controlled, longitudinal studies have examined this relationship between the long-term psychosocial outcomes of affective AI use and what users perceive as satisfactory. To investigate this tension, I conduct a mixed-methods behavioral analysis of 36,027 turns from 52 Replika users in a 21-day randomized controlled study by Guingrich and Graziano (2025). I first develop a bidirectional conversational coding schema spanning 16 bot-side and 28 user-side codes, which I then link to participant-reported wellbeing and preference outcomes. My analysis finds that preference does not correlate with wellbeing; however, both surface independently in the transcripts. User-side behavioral markers including closer relational framing, negative self-talk, positive appraisal of the chatbot, emotional distress, and repetitive negative thinking track with preference. A largely distinct set of user-distress markers correlates with decline in self-esteem, social health, and loneliness. In addition, I apply OpenAI's Model Spec Eval to an adversarial set of 32 Replika response turns drawn from days containing crisis-level participant self-disclosures. The Model Spec reliably flags crisis-level non-compliance but inconsistently catches the non-acute, emotionally distressed interactions that accumulate toward crisis. Additionally, sycophancy is almost never flagged. Together, these findings suggest that user satisfaction is not a reliable proxy for user wellbeing in digital companionship, and that current evaluation frameworks need greater sensitivity to behavioral patterns across the full arc of companion AI interaction, not only at acute moments.