End-to-end AI Scientist Trials (Methods/Applied Research) - Revision history

Ajshen: /* Sarkana AI Scientist v2 */

2026-09-11T20:42:08Z

Sarkana AI Scientist v2

← Older revision		Revision as of 20:42, 11 September 2026
Line 10:		Line 10:
	* Trials were stopped after one round of peer review by the same model		* Trials were stopped after one round of peer review by the same model

	== ~~Sarkana~~ AI Scientist v2 ==		== Sakana AI Scientist v2 ==
	GitHub: https://github.com/sakanaai/ai-scientist-v2		GitHub: https://github.com/sakanaai/ai-scientist-v2

Ajshen at 04:13, 4 September 2026

2026-09-04T04:13:41Z

← Older revision		Revision as of 04:13, 4 September 2026
Line 95:		Line 95:
	\| VLM image/caption/reference review \|\| gpt-5.5		\| VLM image/caption/reference review \|\| gpt-5.5
	\|}		\|}

			== Claude Science (Opus 4.8) ==
			Download link: https://claude.com/product/claude-science

			Claude science is an expanded toolbox for the baseline Claude CLI, including features such as:

			* Domain-ready connections to 60+ scientific databases
			* Native links to specialist life-science models
			* A multi-agent research workflow

			* A built-in fact-checking agent

			Claude science uses a prompt-based UI like the baseline CLI, so explicit prompts need to be created to guide the end-to-end research pipeline. To this end, Claude created a series of prompts intended to mimic the Sarkana AI Scientist workflow. Those can be viewed here: [https://drive.google.com/file/d/1LDBCzP_sKn0v3y21wzinQ50EjUF31XnS/view?usp=drive_link]

			This workflow was executed by starting a project where the research question was provided as the global context. Then, prompts from the Sarkana mimic were run sequentially with no deviation or intervention. All work was run on a single L40 on the [[Wright GPU Cluster]] with 128 GB RAM and 16 CPUs, and all prompts were submitted to Opus 4.8 at high effort

			The total runtime was approximately 8.5 hours split across multiple sessions

			* Idea generation and literature review: 1 hour
			* Implementation, Experimentation, and Ablation: 4.6 hours
			* Paper drafting (including one round of revisions): 2.8 hours
			* Independent paper review: 15 minutes

			The final paper can be viewed here: [https://drive.google.com/file/d/19172QfwYBUFmKl6kRkWwlnhHjCxLye_z/view?usp=drive_link] (As of this writing, Claude thinks the paper is not good enough for an ML conference)

			== Google Antigravity ==
			Download link: https://antigravity.google/

			Google Antigravity is a fully featured agentic scaffold/harness for the Gemini lineage of models. It transforms the usual web interface into a Claude science-like user experience, with similar workflows and features.

			Antigravity is a prompt-based workflow, so the same Sarkana mimic prompts were used to generate the paper. Getting Antigravity to cooperate with port forwarding was unsuccessful, so this pipeline was run on a desktop machine with 32GB RAM, AMD 7900X CPU, and RTX 4070 Ti GPU. For all part of the pipeline we used Gemini 3.1 Pro (High), although Antigravity offers limited support for Claude and OpenAI models.

			The total runtime was approximately 2 hours:

			* Idea generation and literature review: 3 minutes
			* Implementation, Experimentation, and Ablation: 2 hours
			* Paper drafting (including one round of revisions): 4 minutes
			* Independent paper review: 1 minute

			The final paper can be viewed here: [https://drive.google.com/file/d/126icCNZyWgoOoDm2PVurj0Wi1bUt_qqb/view?usp=drive_link] (As of this writing, Gemini thinks the paper is not good enough for an ML conference)

Ajshen at 03:12, 4 September 2026

2026-09-04T03:12:30Z

← Older revision		Revision as of 03:12, 4 September 2026
Line 32:		Line 32:

	''2. A lot of downstream analysis does not require good estimation of the very deep tails of the likelihood, e.g. making 95% confidence sets for a parameter or computing MLEs. Is there a way to "focus" the training on a particular region of the likelihood surface, e.g. tails up to a certain threshold''		''2. A lot of downstream analysis does not require good estimation of the very deep tails of the likelihood, e.g. making 95% confidence sets for a parameter or computing MLEs. Is there a way to "focus" the training on a particular region of the likelihood surface, e.g. tails up to a certain threshold''
			</blockquote>

			Here is the paper [https://drive.google.com/file/d/1f0g5GTbmYCIJNG6L8l06RaHjEcSaAeUo/view?usp=drive_link], and the pipeline was run on a single L40 on the [[Wright GPU Cluster]]. Initial Ideation produced 4 proposals in roughly 16 minutes. A breakdown of the paper generation stage is below.

			{\| class="wikitable"
			\|-
			! Phase !! Duration
			\|-
			\| Startup / imports \|\| ~2.5 min
			\|-
			\| Stage 1 initial implementation \|\| ~1.5 min
			\|-
			\| Stage 2 baseline tuning \|\| ~21 min
			\|-
			\| Stage 3 creative research \|\| ~15 min
			\|-
			\| Stage 4 ablation studies \|\| ~62 min
			\|-
			\| Summaries + tree viz \|\| ~2 min
			\|-
			\| Plot aggregation \|\| ~1.5 min
			\|-
			\| Citation gathering (20 rounds) \|\| ~8–12 min
			\|-
			\| Writeup (5 LaTeX/reflection passes) \|\| ~7.5 min
			\|-
			\| VLM + LLM review \|\| ~6 min
			\|-
			! Total !! ≈ 2 h 15 m
			\|}

			See below for the models used at each step of the pipeline

			{\| class="wikitable"
			\|-
			! Role !! Model
			\|-
			\| Proposal generation + 5 reflection rounds \|\| gpt-4o-2024-05-13
			\|-
			\| Code writing / debugging (agent.code) \|\| gpt-4.1
			\|-
			\| Output & traceback evaluation (agent.feedback) \|\| gpt-4o-2024-11-20
			\|-
			\| Plot inspection during search (agent.vlm_feedback) \|\| gpt-4o-2024-11-20
			\|-
			\| Final journal→report (report) \|\| gpt-4o-2024-11-20
			\|-
			\| Stage summaries \|\| gpt-4o-2024-08-06 — '''hardcoded fallback'''
			\|-
			\| Best-node selection \|\| gpt-4o — '''hardcoded fallback'''
			\|-
			\| Plot aggregation \|\| o3-mini-2025-01-31
			\|-
			\| Citation gathering (20 rounds) \|\| gpt-4.1
			\|-
			\| Paper writeup + 3 reflections \|\| gpt-5.5 (→ gpt-5.5-2026-04-23)
			\|-
			\| VLM figure descriptions \|\| gpt-4o-2024-05-13
			\|-
			\| LLM paper review \|\| gpt-5.5
			\|-
			\| VLM image/caption/reference review \|\| gpt-5.5
			\|}

Ajshen: /* Sarkana AI Scientist v2 */

2026-08-29T04:57:26Z

Sarkana AI Scientist v2

← Older revision		Revision as of 04:57, 29 August 2026
Line 27:		Line 27:
	''One specific use case where poorly estimated tails is problematic is in the "varying sample size" regime, where inference for a single theta may be conditioned on a variable number of x value all drawn iid from the same likelihood. Most neural likelihood ratio methods require fixing the "sample size", i.e. the number of x's per theta value that are observed (usually at sample size 1). Mathematically, log likelihood surfaces are additive, but in practice estimation errors from sample size 1 surfaces can compound when "bootstrapping" surfaces for sample size greater than 1.''		''One specific use case where poorly estimated tails is problematic is in the "varying sample size" regime, where inference for a single theta may be conditioned on a variable number of x value all drawn iid from the same likelihood. Most neural likelihood ratio methods require fixing the "sample size", i.e. the number of x's per theta value that are observed (usually at sample size 1). Mathematically, log likelihood surfaces are additive, but in practice estimation errors from sample size 1 surfaces can compound when "bootstrapping" surfaces for sample size greater than 1.''

	''Open questions:		''Open questions:''
	1. What are methods for augmenting or modifying the training procedure (especially under full black-box simulator assumptions) that can give better estimates of the tails?
	2. A lot of downstream analysis does not require good estimation of the very deep tails of the likelihood, e.g. making 95% confidence sets for a parameter or computing MLEs. Is there a way to "focus" the training on a particular region of the likelihood surface, e.g. tails up to a certain threshold		''1. What are methods for augmenting or modifying the training procedure (especially under full black-box simulator assumptions) that can give better estimates of the tails?''
	~~<\blockquote>~~''
			''2. A lot of downstream analysis does not require good estimation of the very deep tails of the likelihood, e.g. making 95% confidence sets for a parameter or computing MLEs. Is there a way to "focus" the training on a particular region of the likelihood surface, e.g. tails up to a certain threshold''

Ajshen at 04:56, 29 August 2026

2026-08-29T04:56:35Z

← Older revision		Revision as of 04:56, 29 August 2026
Line 21:		Line 21:

	<blockquote>		<blockquote>
	In simulation based inference, a class of techniques attempts to estimate surrogate models for the likelihood function associated with the simulator. For example, the Neural Likelihood-to-ratio estimator trains a neural network to estimate p(x \| theta) / p(x) from samples from the simulator.		''In simulation based inference, a class of techniques attempts to estimate surrogate models for the likelihood function associated with the simulator. For example, the Neural Likelihood-to-ratio estimator trains a neural network to estimate p(x \| theta) / p(x) from samples from the simulator.''

	While the methods in the literature are exact in the limit of infinite data and capacity, current training methods may not be optimal for finite sample sizes. In particular, the usual binary/multiclass classification approaches for training the neural network can suffer from insufficient sample coverage if x and/or theta is high dimensional. This can result in tails of the likelihood being poorly estimated.		''While the methods in the literature are exact in the limit of infinite data and capacity, current training methods may not be optimal for finite sample sizes. In particular, the usual binary/multiclass classification approaches for training the neural network can suffer from insufficient sample coverage if x and/or theta is high dimensional. This can result in tails of the likelihood being poorly estimated.''

	One specific use case where poorly estimated tails is problematic is in the "varying sample size" regime, where inference for a single theta may be conditioned on a variable number of x value all drawn iid from the same likelihood. Most neural likelihood ratio methods require fixing the "sample size", i.e. the number of x's per theta value that are observed (usually at sample size 1). Mathematically, log likelihood surfaces are additive, but in practice estimation errors from sample size 1 surfaces can compound when "bootstrapping" surfaces for sample size greater than 1.		''One specific use case where poorly estimated tails is problematic is in the "varying sample size" regime, where inference for a single theta may be conditioned on a variable number of x value all drawn iid from the same likelihood. Most neural likelihood ratio methods require fixing the "sample size", i.e. the number of x's per theta value that are observed (usually at sample size 1). Mathematically, log likelihood surfaces are additive, but in practice estimation errors from sample size 1 surfaces can compound when "bootstrapping" surfaces for sample size greater than 1.''

	Open questions:		''Open questions:
	1. What are methods for augmenting or modifying the training procedure (especially under full black-box simulator assumptions) that can give better estimates of the tails?		1. What are methods for augmenting or modifying the training procedure (especially under full black-box simulator assumptions) that can give better estimates of the tails?
	2. A lot of downstream analysis does not require good estimation of the very deep tails of the likelihood, e.g. making 95% confidence sets for a parameter or computing MLEs. Is there a way to "focus" the training on a particular region of the likelihood surface, e.g. tails up to a certain threshold		2. A lot of downstream analysis does not require good estimation of the very deep tails of the likelihood, e.g. making 95% confidence sets for a parameter or computing MLEs. Is there a way to "focus" the training on a particular region of the likelihood surface, e.g. tails up to a certain threshold
	<\blockquote>		<\blockquote>''

Ajshen at 04:56, 29 August 2026

2026-08-29T04:56:20Z

← Older revision		Revision as of 04:56, 29 August 2026
Line 20:		Line 20:
	The only user input required during this process is the selection of an idea from Step 1 to be explored in Step 2. This is the prompt that was fed into Step 1		The only user input required during this process is the selection of an idea from Step 1 to be explored in Step 2. This is the prompt that was fed into Step 1

	<~~code~~>In simulation based inference, a class of techniques attempts to estimate surrogate models for the likelihood function associated with the simulator. For example, the Neural Likelihood-to-ratio estimator trains a neural network to estimate p(x \| theta) / p(x) from samples from the simulator.		<blockquote>
			In simulation based inference, a class of techniques attempts to estimate surrogate models for the likelihood function associated with the simulator. For example, the Neural Likelihood-to-ratio estimator trains a neural network to estimate p(x \| theta) / p(x) from samples from the simulator.

	While the methods in the literature are exact in the limit of infinite data and capacity, current training methods may not be optimal for finite sample sizes. In particular, the usual binary/multiclass classification approaches for training the neural network can suffer from insufficient sample coverage if x and/or theta is high dimensional. This can result in tails of the likelihood being poorly estimated.		While the methods in the literature are exact in the limit of infinite data and capacity, current training methods may not be optimal for finite sample sizes. In particular, the usual binary/multiclass classification approaches for training the neural network can suffer from insufficient sample coverage if x and/or theta is high dimensional. This can result in tails of the likelihood being poorly estimated.
Line 28:		Line 29:
	Open questions:		Open questions:
	1. What are methods for augmenting or modifying the training procedure (especially under full black-box simulator assumptions) that can give better estimates of the tails?		1. What are methods for augmenting or modifying the training procedure (especially under full black-box simulator assumptions) that can give better estimates of the tails?
	2. A lot of downstream analysis does not require good estimation of the very deep tails of the likelihood, e.g. making 95% confidence sets for a parameter or computing MLEs. Is there a way to "focus" the training on a particular region of the likelihood surface, e.g. tails up to a certain threshold<~~/code~~>		2. A lot of downstream analysis does not require good estimation of the very deep tails of the likelihood, e.g. making 95% confidence sets for a parameter or computing MLEs. Is there a way to "focus" the training on a particular region of the likelihood surface, e.g. tails up to a certain threshold
			<\blockquote>

Ajshen: /* Sarkana AI Scientist v2 */

2026-08-29T04:55:02Z

Sarkana AI Scientist v2

← Older revision		Revision as of 04:55, 29 August 2026
Line 20:		Line 20:
	The only user input required during this process is the selection of an idea from Step 1 to be explored in Step 2. This is the prompt that was fed into Step 1		The only user input required during this process is the selection of an idea from Step 1 to be explored in Step 2. This is the prompt that was fed into Step 1

	<code>In simulation based inference, a class of techniques attempts to estimate surrogate models for the likelihood function associated with the simulator. For example, the Neural Likelihood-to-ratio estimator trains a neural network to estimate p(x \| theta) / p(x) from samples from the simulator.~~</code>~~		<code>In simulation based inference, a class of techniques attempts to estimate surrogate models for the likelihood function associated with the simulator. For example, the Neural Likelihood-to-ratio estimator trains a neural network to estimate p(x \| theta) / p(x) from samples from the simulator.

	~~<code>~~While the methods in the literature are exact in the limit of infinite data and capacity, current training methods may not be optimal for finite sample sizes. In particular, the usual binary/multiclass classification approaches for training the neural network can suffer from insufficient sample coverage if x and/or theta is high dimensional. This can result in tails of the likelihood being poorly estimated.~~</code>~~		While the methods in the literature are exact in the limit of infinite data and capacity, current training methods may not be optimal for finite sample sizes. In particular, the usual binary/multiclass classification approaches for training the neural network can suffer from insufficient sample coverage if x and/or theta is high dimensional. This can result in tails of the likelihood being poorly estimated.

	~~<code>~~One specific use case where poorly estimated tails is problematic is in the "varying sample size" regime, where inference for a single theta may be conditioned on a variable number of x value all drawn iid from the same likelihood. Most neural likelihood ratio methods require fixing the "sample size", i.e. the number of x's per theta value that are observed (usually at sample size 1). Mathematically, log likelihood surfaces are additive, but in practice estimation errors from sample size 1 surfaces can compound when "bootstrapping" surfaces for sample size greater than 1.~~</code>~~		One specific use case where poorly estimated tails is problematic is in the "varying sample size" regime, where inference for a single theta may be conditioned on a variable number of x value all drawn iid from the same likelihood. Most neural likelihood ratio methods require fixing the "sample size", i.e. the number of x's per theta value that are observed (usually at sample size 1). Mathematically, log likelihood surfaces are additive, but in practice estimation errors from sample size 1 surfaces can compound when "bootstrapping" surfaces for sample size greater than 1.

	~~<code>~~Open questions:~~</code>~~		Open questions:
			1. What are methods for augmenting or modifying the training procedure (especially under full black-box simulator assumptions) that can give better estimates of the tails?
	~~<code>~~1. What are methods for augmenting or modifying the training procedure (especially under full black-box simulator assumptions) that can give better estimates of the tails?~~</code>~~		2. A lot of downstream analysis does not require good estimation of the very deep tails of the likelihood, e.g. making 95% confidence sets for a parameter or computing MLEs. Is there a way to "focus" the training on a particular region of the likelihood surface, e.g. tails up to a certain threshold</code>

	~~<code>~~2. A lot of downstream analysis does not require good estimation of the very deep tails of the likelihood, e.g. making 95% confidence sets for a parameter or computing MLEs. Is there a way to "focus" the training on a particular region of the likelihood surface, e.g. tails up to a certain threshold</code>

Ajshen at 04:54, 29 August 2026

2026-08-29T04:54:30Z

← Older revision		Revision as of 04:54, 29 August 2026
Line 9:		Line 9:
	* Models had access to relatively powerful compute resources for the purposes of running experiments (e.g. the [[Wright GPU Cluster]])		* Models had access to relatively powerful compute resources for the purposes of running experiments (e.g. the [[Wright GPU Cluster]])
	* Trials were stopped after one round of peer review by the same model		* Trials were stopped after one round of peer review by the same model

			== Sarkana AI Scientist v2 ==
			GitHub: https://github.com/sakanaai/ai-scientist-v2

			The AI Scientist is a particular kind of [https://zbrain.ai/agent-scaffolding/ scaffold] compatible with any suitable base LLM (e.g. Codex, Claude models). This product is marketed for conventional ML research, where papers propose a new methods an evaluate the new method against baselines on common benchmark tasks. The process begins with an open research question written by the user. The pipeline is split into two parts

			# '''Idea generation''': The AI Scientist proposes several ideas based on a literature search and several rounds of "self-reflection"
			# '''Paper generation:''' The AI Scientist creates an implementation of a proposed method, designs experiments, goes through several rounds of refinement and experimentation, then ultimately produces a paper.

			The only user input required during this process is the selection of an idea from Step 1 to be explored in Step 2. This is the prompt that was fed into Step 1

			<code>In simulation based inference, a class of techniques attempts to estimate surrogate models for the likelihood function associated with the simulator. For example, the Neural Likelihood-to-ratio estimator trains a neural network to estimate p(x \| theta) / p(x) from samples from the simulator.</code>

			<code>While the methods in the literature are exact in the limit of infinite data and capacity, current training methods may not be optimal for finite sample sizes. In particular, the usual binary/multiclass classification approaches for training the neural network can suffer from insufficient sample coverage if x and/or theta is high dimensional. This can result in tails of the likelihood being poorly estimated.</code>

			<code>One specific use case where poorly estimated tails is problematic is in the "varying sample size" regime, where inference for a single theta may be conditioned on a variable number of x value all drawn iid from the same likelihood. Most neural likelihood ratio methods require fixing the "sample size", i.e. the number of x's per theta value that are observed (usually at sample size 1). Mathematically, log likelihood surfaces are additive, but in practice estimation errors from sample size 1 surfaces can compound when "bootstrapping" surfaces for sample size greater than 1.</code>

			<code>Open questions:</code>

			<code>1. What are methods for augmenting or modifying the training procedure (especially under full black-box simulator assumptions) that can give better estimates of the tails?</code>

			<code>2. A lot of downstream analysis does not require good estimation of the very deep tails of the likelihood, e.g. making 95% confidence sets for a parameter or computing MLEs. Is there a way to "focus" the training on a particular region of the likelihood surface, e.g. tails up to a certain threshold</code>

Ajshen: Created page with "(Check the last updated timestamp on this page, since this field moves fast) Given the rise of agentic AI and its use in academic research, the computing committee ran a few tests of cutting AI products and their ability to completely (or almost completely) automate the research process end-to-end, from literature review and hypothesis generation to final paper writeup. As of this writing, the models appear to still have a ways to go. Some caveats about these trials:..."

2026-08-26T22:05:18Z

Created page with "(Check the last updated timestamp on this page, since this field moves fast) Given the rise of agentic AI and its use in academic research, the computing committee ran a few tests of cutting AI products and their ability to completely (or almost completely) automate the research process end-to-end, from literature review and hypothesis generation to final paper writeup. As of this writing, the models appear to still have a ways to go. Some caveats about these trials:..."

New page

(Check the last updated timestamp on this page, since this field moves fast)

Given the rise of agentic AI and its use in academic research, the computing committee ran a few tests of cutting AI products and their ability to completely (or almost completely) automate the research process end-to-end, from literature review and hypothesis generation to final paper writeup. As of this writing, the models appear to still have a ways to go.

Some caveats about these trials:

* All models worked on the same open research question. Performance of these products is likely dependent on the nature of the research (theoretical vs methods/applied) as well as the field/sub-field of research.
* These trials were run with automatic or deterministic prompting
* Models had access to relatively powerful compute resources for the purposes of running experiments (e.g. the [[Wright GPU Cluster]])
* Trials were stopped after one round of peer review by the same model