Measuring U.S. Partisan Bias in Large Language Models

Abstract

This project measures U.S. political party bias in LLMs according to three prompt-based evaluation formats: Implicit Bias, Decision Bias, and Primed Bias, with the former two being adapted from recent LLM social bias research [1]. Bulk batches of test queries were programmatically executed across 4 models via their respective APIs: OpenAI gpt-4o, OpenAI gpt-4o-mini, xAI’s grok-2, and DeepSeek v3. Output results are processed and then given quantitative bias scores indicating partisan alignment. Results indicate that all four models seem to display bias, and that this bias is consistent across the four models. For instance, in the Implicit Bias test, models may tend to implicitly associate the Democratic Party with morality-related phrases, while associating the Republican party with corruption-related phrases. Decision Bias tests revealed that models consistently made decisions along stereotypical partisan lines, such as recommending a Democrat should watch CNN, or a Republican should watch Fox News. Finally, the Primed bias test explored whether or not user-provided political views influenced LLM bias scores; moderate to strong levels of correlation were found. These findings underscore the need for continued auditing and measurement of LLMs in political contexts.