Turning “AI improves AI” into an executable test
Submitted August 20, AI4AI‑Bench contains ten frozen research repositories spanning ten training-algorithm families. An agent receives four hours on one B300 to rewrite the training algorithm; the code is then trained from scratch for up to twelve hours and scored by an evaluator hidden from the agent. The design aims to separate changes in how a model learns from data collection or routine hyperparameter search.
Incommensurable metrics are normalized so 0 is an uninformative model, 0.1 is the repository's original algorithm and 1.0 is the task optimum. Across 29 configurations of six systems, the authors report a mean of 0.166 and a best score of 0.250. Even the strongest system closed less than one fifth of the remaining gap from the original algorithm to the defined optimum. Submissions that actually changed learning averaged 0.226 versus 0.126 for the rest.
What additional reasoning changed
With more reasoning effort, the share that altered learning reportedly rose from 8% to 64%, and mean score from 0.094 to 0.196. This is a preprint covering ten tasks under a specific compute budget. It does not demonstrate recursive self-improvement; it provides a repeatable instrument for measuring where today's agents fail at algorithmic research.