Evaluating the path to a repository, not only the repository
The NLPCC 2026 Shared Task organizers posted AgentActionBench on September 10. Many existing evaluations inspect only the final repository, which makes it difficult to distinguish a mistaken reading of a paper from a failure in environment setup or execution. The new benchmark uses an MCP-based Action Recorder to capture files, commands and tool activity, then scores the trace with rubrics tailored to each paper.
The collection covers 150 papers: 120 in machine learning and 30 in AI for Science. Humans annotated a ten-percent subset, while model-assisted expansion produced more than 10,000 rubric items. In the authors' experiments, current systems remained limited and execution—running code, resolving errors and producing results—was the main bottleneck rather than merely writing a plausible plan.
Reliable rubric generation is not the same as reproduced science
The authors report strong Pearson and Spearman correlations between generated rubrics and human annotations. That supports scalable process evaluation, but it does not prove that an experimental claim was scientifically reproduced. A set of 150 papers and a ten-percent human subset may not represent every discipline, and the paper is a preprint awaiting peer review.
Evaluations of research agents should track pinned environments, data provenance, execution logs, recovery after failure, numerical reproduction tolerances and human confirmation of central claims. A long action trace by itself is not evidence of a correct result.