End-to-end AI Scientist Trials (Methods/Applied Research)

(Check the last updated timestamp on this page, since this field moves fast)

Given the rise of agentic AI and its use in academic research, the computing committee ran a few tests of cutting AI products and their ability to completely (or almost completely) automate the research process end-to-end, from literature review and hypothesis generation to final paper writeup. As of this writing, the models appear to still have a ways to go.

Some caveats about these trials:

• All models worked on the same open research question. Performance of these products is likely dependent on the nature of the research (theoretical vs methods/applied) as well as the field/sub-field of research.
• These trials were run with automatic or deterministic prompting
• Models had access to relatively powerful compute resources for the purposes of running experiments (e.g. the Wright GPU Cluster)
• Trials were stopped after one round of peer review by the same model