A Comparison of LLM-Based and Stock-Return-Based Financial Sentiment Classifiers

Abstract

In this paper, we build on existing applications of large language models, including both chat models and embedding models, for the purpose of sentiment classification of financial news articles. We evaluate two types of classification models: general GPT-based models that do not incorporate information about stock returns and a specialized model that is trained on stock returns. Through comparisons of classification performance and sentiment-based portfolio performance, we show that GPT-based models perform as well as context-specific, stock-return-based models. One drawback of using a chat model for sentiment classification is that it does not generate a ranking over the set of articles. Thus, we are limited to acting on all articles in our trading strategies. We show that a logistic regression model that utilizes embeddings as features and labels generated directly by GPT as targets can be used to improve on the performance of direct-GPT-based trading strategies because it allows us to rank the articles by confidence in positive or negative labels. Finally, we explore model performance across various categories of news and find that articles about earnings expectations or earnings reports are the most informative. All models classify high-impact earnings news accurately, and, as a result, earnings-related news is found to be a significant driver of returns in our sentiment-based portfolios.