Automated Social Science with Large Language Models: Framework, Experiments, and Future Outlook

The authors introduce the motivation for leveraging large language models (LLMs) in social‑science experiments, noting that traditional AB‑testing workflows involve three steps—hypothesis, experiment, analysis—and that LLMs can automate these stages while reducing human bias and ethical concerns.

They propose an "Automated Social Science" framework consisting of seven modules: (1) Specify Social Scenario (human‑in‑the‑loop), (2) Hypothesis Generation, (3) Agent Building, (4) Design Interaction, (5) Experiment Running, (6) Data Collection, and (7) Model Estimation. The framework uses LLM‑driven agents to represent participants and automatically conducts experiments.

To illustrate the workflow, a concrete example is given where a researcher defines a trading scenario. Using a structural causal model, the LLM generates hypotheses about buyer budget and seller price, creates buyer and seller agents with attributes, designs a simple bidding interaction, runs parallel simulations, and collects outcome data directly from the agents.

The authors validate the approach with an auction experiment involving three simulated bidders. The LLM assigns budget variables, runs the auction, and estimates effect sizes, showing that a $1 increase in bidder budget raises the final price by roughly 33 % and that the simulated results align with classic auction theory predictions.

Finally, the significance of LLMs as human surrogates is discussed: they capture nuanced human behavior, enable low‑cost large‑scale experiments, mitigate confounding variables, and avoid ethical issues. Future research directions include building better LLM scientists for hypothesis generation and experimental design, improving human‑behavior modeling, and applying LLM‑based simulations to public‑policy analysis.