Court v. Classifier: A Data-Driven Evaluation of Language and Decision-Making on the U.S. Supreme Court

Abstract

This thesis investigates the language, behavior, and decision-making of U.S. Supreme Court justices through a computational lens. Grounding my study in structured and curated datasets—including justice- and case-level variables, authored opinions, and over 1,600 transcribed oral arguments—I analyze how justices speak, write, and vote. I begin with an empirical study of voting patterns, opinion authorship, and judicial trends across natural court eras. I then turn to oral argument behavior, quantifying the participation of justices across alignments and outcomes. Building on these insights, I implement a series of predictive classifiers, replicating and extending a previous statistical model to include oral argument features. While the inclusion of these features yields modest and at times inconclusive improvements in accuracy, they underscore the complexity of predicting voting patterns based on oral argument behavior, given the distinct rhetorical styles and engagement patterns of individual justices. Nonetheless, the findings allude to promising directions for future modeling of case outcomes using alternative features derived from oral arguments. Finally, I experiment with prompting large language models (LLMs) to classify tones of judicial questioning due to the limitations of more traditional natural language processing techniques. I also simulate justice voting behavior with LLMs on unseen cases, assessing the capabilities of generative AI for legal reasoning. Through our experimentation, the LLMs proved to be limited in their capacity for legal judgement, though they also demonstrate opportunity to be better leveraged when provided additional guidance through fine-tuning. Altogether, this study offers a data-driven portrait of the Supreme Court and its justices, rooted in empirical data and powered by modern machine learning methods.