The Art of Convincing: Investigating Persuasive Strategies Employed by Humans vs LLMs Through Natural Language Processing

Abstract

This thesis explores the persuasive quality of natural language, specifically in regards to the persuasive strategies that humans and AI models employ as a persuader, and are susceptible to as a persuadee. Starting from a partially annotated dataset of human-to-human persuasive text, we explore various approaches to the multi-class classification problem of labeling text with persuasive strategies, including in-context learning methodologies and fine-tuning a RoBERTa encoder model. We then write software to displace the human with an LLM, in both the persuader and persuadee roles. This allows us to directly compare the strategies employed by human-as-persuader and LLM-as-persuader texts, and likewise for persuasion outcomes between human-as-persuadee and LLM-as-persuadee. We find that the LLM persuader tends to avoid employing emotional or personal persuasive tactics as compared to the human persuader, and that the LLM persuadee is significantly more difficult to convince overall. This research is important as LLMs begin to pervade more aspects of human lives - understanding LLMs’ ability to persuade and to be persuaded helps create a safer, more technologically-literate future.