SCouT++: Improved Representations of Transformers for Counterfactual Reasoning

Abstract

In a world where decisions shape our future, from personal choices to policy implementations, the concept of counterfactual reasoning emerges as a powerful tool for informed decision-making. This thesis explores the application of counterfactual reasoning in healthcare, particularly focusing on personalized medicine and the evaluation of medical interventions. Traditional Randomized Control Trials (RCTs) face limitations in addressing individual variability and are often impractical for rare diseases. In this work, we tackle this challenge by leveraging the Synthetic Control (SC) method, specifically SCouT- a novel transformer model introduced by Dedhia et al., where the SC problem is approached from the lens of a sequence-to-sequence learning task. We introduce SCouT++, exploiting masked language modeling and auto-regressive prediction to enhance the representation learning process and extend the model’s applicability to scenarios with limited pre-intervention data. Through experimentation on synthetic and real-world datasets, this research serves as a proof-of-concept of SCouT++ in providing reliable counterfactual estimations, positioned for further research and improvement on methods, ultimately advancing the frontier of personalized healthcare and contributing to the design of more efficient medical interventions.