Capability benchmark · Computer science
Station recovers more withheld research findings in three open-ended ML tasks
Experimental research reports recovering findings from three ICLR oral papers under open-ended task descriptions.
Summary
Station evaluated six-agent research loops on three ML rediscovery tasks with papers withheld and internet disabled. It recovered 62.7% of graded findings versus 14.4–20.6% for baselines at matched experiment budgets. Codex judged the reports; one author checked a small blinded sample.
AI role
Six agents from three model families pursue experiments with supervisor feedback and periodic reflection, then consolidate research reports.
Narrative role
Adds evidence that sustained agent experimentation can recover mechanistic findings beyond optimizing a supplied score, with inspectable reports and limited expert checking.
Caveat
Training exposure to oracle papers cannot be excluded. Model composition and workflows differ; one-task expert checking does not independently validate all findings. These are rediscoveries, not a measured research speedup.