AI Summarization and Customer Decision-Making in E-Commerce

Abstract

Customer reviews are widely used across online shopping platforms to provide additional insight into quality and perceived usefulness of many products and services. However, consumers are often unable to read the full volume of reviews for highly reviewed products. To combat this, online retailers have begun implementing artificial intelligence (AI) summaries of customer reviews, claiming that they help consumers make better decisions and increase decision-making speed. However, there is little public-facing empirical research that supports these claims. Furthermore, existing research has to rely on strong assumptions and proxies that weaken the interpretability of its findings. Using a controlled experiment, we answer whether exposure to AI summaries of customer reviews helps customers interpret the price and quality of goods in order to make better decisions for themselves. Participants complete a series of price estimation and purchasing decision simulation tasks and describe their general sentiment towards AI. Participants are then assigned sentiment classifications using the RoBERTa model for sentiment analysis. We find that AI summaries have a small, positive but statistically insignificant effect on price estimation accuracy. We also find that AI summaries did not significantly decrease time spent on either task, but individuals with negative sentiment towards AI spent significantly longer on settings with AI summaries compared to those without. Most strikingly, individuals with positive sentiment towards AI demonstrated significantly lower simulated purchasing rates on settings with AI summaries. These findings challenge claims made by e-commerce companies and existing research that suggest AI summaries cause increases in sales volume and faster decision-making.