Text-to-Design: Automated Conversion of Scientific Methods into Experimental Specification

Abstract

Reproducing experiments in behavioral and cognitive science requires a precise extraction of factors, conditions, constraints, and trial logic from experimental design. In academic practice, this information is often embedded within verbose methods sections, padded with writing motivated by persuasion and defensibility rather than reproducibility, making manual extraction a slow, error-prone process difficult to scale. In this thesis, we present Text-to-Design, a multi-stage LLM pipeline automatically converting the natural language found in factorial design methodology to executable SweetPea specifications. The pipeline decomposes the task into LLM-powered summarization, LLM-powered compilation into intermediate representation, deterministic SweetPea translation, and two layers of LLM verification and correctness checks. When evaluated against 54 published cognitive science papers, the pipeline achieved an 88.9% pass rate, compared to a one-shot baseline rate of 25.9%. The pipeline generates a structured corpus of the stage-by-stage pipeline states for each paper, but also records what experimental details are commonly under-specified. These contributions help demonstrate the feasibility of large-scale automated experimental design information extraction for dataset construction and furthering foundation models for cognitive science.