Just You, Me, and ChatGPT: The Effect of LLM-Assisted Pre-Play Communication in Economic Games

Abstract

This paper investigates the effect of LLM-assisted pre-play communication on outcomes in economic games. Participants play a series of trust and coordination games under three treatments: a control condition which allows them to write a message to each other, an LLM-edit condition which edits the messages for tone and clarity, and an LLM-summary condition which summarizes the messages and identifies opportunities for mutual benefit. I find that the effects of the LLM-assistance on payoffs are highly dependent on context. In trust games, the LLM-edit condition worsens payoffs, with the most profound effect on prisoner’s dilemma games with high temptation to defect. While the edit treatment does not have an effect overall in coordination games, it might have a positive effect on payoffs in the stag hunt. The summary treatment does not have an effect overall in either trust or coordination games. An exploratory portion of analysis reveals that both treatments improve payoffs when messages are written in poor English, and suggests that message content can be used to make using LLM-assistance more beneficial. Overall, the results suggest we should be cautious of the integration of LLM-assisted communication in situations in which trusting a partner is difficult, while more optimistic about its use in situations in which the interests of participants are fully aligned.