Can AI agents read a social science paper and write the code from scratch to reproduce its results? No access to original code. Just text + data. New paper with Ben Kohler, @David Zollikofer, @Johanna Einsiedler, and @Alexander Hoyle

How we do it. Agents: → extract methods from the paper → reimplement the analysis → reproduce every table cell → compare to original → trace errors

Our testbed is 48 published papers in econ and poli sci – with verified reproducibility, thanks to the amazing @I4R. The challenge: fill in 14,214 empty cells in 222 redacted tables.

The contenders. Heavy hitters: GPT-5.3 / 5.4; Claude Opus 4.6 Underdog: GLM-5 (open weights) Two of the jockies (GPT-5.4 and GLM-5) steer multiple horses. Each LLM takes a turn strapped into the @OpenCode and SWE-Agent harnesses (on top of GPT-5.4 running in its native Codex CLI).

Main result: Agents recover most published findings. Sign of coefficients correct: ~80–90% Within 95% CI: >70% (best models: >80%) That’s real progress toward automated reproducibility.

Best setup: GPT-5.4 running on @OpenCode: • ~91% correct coefficient signs • >80% within 95% CI Other LLMs (Claude, GLM-5) and scaffolds (Codex, SWE-agent) trail behind. E.g. SWE-agent + GPT-5.4: → ~78% correct coeff. sign (worst of the lot) Same base model. Very different outcomes.

Better agent systems don’t just “think better” They: use more tokens ‍ run longer explore more Performance = capability × compute

Here’s the real surprise. Failures are mostly not AI mistakes. They come from the papers. Underspecified methods. Missing details. Ambiguity.

Example: Paper says “controls included” Code actually uses specific variables, filters, transformations That gap forces the agent to guess. Different agents → different guesses → different results

Upshot Even without code, agents can: • reconstruct regressions • rebuild pipelines • recover datasets • match published tables The original papers were in Stata or R. The agents rewrote everything in Python. This is reimplementation, not just rerunning code.

Quick sidenote: We believe that successful reproductions are not due to training-data leakage. In a side analysis, we compared performance on some reproduction packages published before/after the model knowledge cutoffs, and results were similar.

What does this mean for research? The bottleneck is shifting. Not model capability → documentation quality Reproducibility depends on how precisely methods are written.

A deeper question: If they are not the source of truth, then what are papers for? One idea: → Code defines what was done. → Paper explains why.

Another idea: Agentic reproduction systems can serve a diagnostic function for verifying paper/code alignment.

Where this goes next. What if agents: • reconstruct missing data? • infer methods from questions? • run robustness checks automatically? This is not just reproducibility. It’s the start of automated science.

There are still many gaps to close. But for the first time: AI can read the paper and write the code. And that changes how we think about scientific verification. https://elliottash.com/wp-content/uploads/2026/04/Kohler-Zollikofer-Einsiedler-Hoyle-Ash-Read-Paper-Write-Code-Agentic-Reproduction-Social-Science-Results.pdf…

@Andy Hall @Alex Imas @Ethan Mollick @scott cunningham @Chris Blattman @D. Yanagizawa-Drott @Soumitra Shukla @Brian Nosek (@briannosek@nerdculture.de) @Metaphysical Science @Stuart Buck @Andrew @Andrej Karpathy @Yiqing Xu @Daniel Kang @Sayash Kapoor @Arvind Narayanan @benediktstroeblQuick

And here is the reproduction code: https://github.com/benjamin-kohler/social_science_replicability… to be added soon: a guide to help run it on your own paper / dataset.