Decades of social science RCTs have generated a wealth of rigorous impact estimates. By carefully isolating cause and effect, they offer insight into what policy alternatives achieve – and, more generally, into the forces driving society and the economy.
But this body of evidence is complex and hard to use. Individual studies can be noisy and difficult to generalize, while traditional methods for synthesizing them into balanced policy advice are slow, expensive, quickly outdated, and lose context.
A better science is one that makes better predictions: the more accurately we can forecast the results of trials that haven't yet resolved, the more we can claim to know. Advances in AI point in exactly this direction, demonstrating dramatic capabilities to synthesize vast amounts of knowledge and generate nuanced predictions. Indeed, recent work suggests that large language models can forecast the effects of survey experiments no worse than human forecasters, and that purpose-built, fine-tuned methods forecast causal effect sizes more accurately than off-the-shelf models – all indications that the approach has real headroom. But many challenges remain: for example, models trained on the published literature inherit publication bias, and machine prediction introduces a trade-off between accuracy and interpretability.
We aim to accelerate research toward a future where policy choices are guided by the best available evidence, and where new experiments are designed to maximize collective learning. To this end, we are announcing a prediction league: an open contest in which forecasting methods are scored against experimental results they have never seen.
As a first step, we are launching a pilot prize in which contestants train on a curated set of open access literature.
