AI Agents: A Primer

Agentic AI tools are AI systems that don't just answer questions: they take actions. An agent can search for papers, read your files, write and run code, launch jobs on a cluster, and look at the results, all in a loop, until the task you gave it is done. For statistical research, the most mature agents today are AI coding agents such as Claude Code, OpenAI's Codex CLI, and Google's Gemini CLI. They run in your terminal and work directly on your repository.

These tools can save an enormous amount of time, and they can also waste an enormous amount of time (and money). The difference is almost entirely in how they are used. This article explains how agents work, where their costs come from, and a set of concrete practices for getting good results from them in research.

Based on a StatBytes talk
This article is adapted from the StatBytes talk Piloting the Suit: Workflow tips for agentic AI in statistical research. A small demo repository that goes with it is at github.com/Alex-Shen-CMU/statbytes-agentic-example. Most examples below use Claude Code, but almost everything applies to other coding agents too.

See [1] for a collection of commonly used agent skills and agent instruction files relevant for statistics research.

What Is an Agentic AI?

An agent has two parts: a large language model (LLM) that does the thinking, and a harness that gives the model a way to act on the world.

The LLM (the brain)

The harness (the body)

Text in, text out: At its core, the model predicts the next token given everything before it. Reasons and plans: It decides what to do next, including which tool to call. Stateless: It knows only what is currently in its context window. It has no memory of yesterday's session unless something puts that information back in front of it.

Runs the loop: It sends the conversation to the model, receives a tool call, executes it, and sends back the result. Executes tools: Web search, reading PDFs, loading data, running code, editing files. Manages context and memory: It decides what goes into the model's context and loads standing instructions. Enforces permissions: It decides what the agent may do without asking, and sandboxes what it runs.

The model and harness talk through tool calls and observations. The model emits something like {search: "Conformal Prediction"}. The harness runs the search and returns "10 results: arXiv, ..." as an observation, which becomes part of the conversation for the next step. Around this exchange sit two more parties: you, who supply the goal, the constraints, and the final judgment, and the world, meaning papers, datasets, your files, compute, and the web.

An agent is an LLM plus a harness, connected by tool calls and observations.

In one sentence: an agent is an LLM in a loop, with tools, running until the task is done.

The agent loop

A single request like "What are the extensions of conformal prediction for dependent data?" becomes many turns of the same loop:

1. Gather context: search for papers, read PDFs, load data.
2. Take action: analyze, compute, draft.
3. Verify: cross-check sources and numbers, run the tests.
4. Repeat until the model decides no more tool calls are needed, then hand you a summary and sources to review.

One detail matters a great deal for cost: every trip around the loop re-sends the entire conversation to the model. This is covered in The Cost of Using Agents below.

The agent loop: gather context, act, verify, repeat.

What can agents do?

Although they are marketed as coding tools, coding agents are general-purpose research assistants that happen to live next to your code.

Software	Computation and data	Research writing and reading
Write code: simulations, samplers, neural estimators Debug: read tracebacks, bisect regressions Refactor: turn a script into a package, port R to Python and back Write tests: unit tests, coverage checks, CI	Run experiments: launch, monitor, collect results Analyze data: EDA, summaries, diagnostic plots Make figures: iterate on a plot, including from a screenshot of a figure you like Reproduce: re-run a paper's code from its repository	Literature: search, summarize, compare methods Write LaTeX: fix compile errors, build tables from CSVs Review: check a derivation, revise a paper draft Admin: issues, emails, documentation, READMEs

AI Coding Agents

The landscape

Coding agents fall roughly into two groups. They share the same core loop, and differ in harness, model, and pricing. The landscape changes monthly, so treat any list like this as a snapshot.

Terminal-native	IDE and cloud
Claude Code (Anthropic) Codex CLI (OpenAI) Gemini CLI (Google) Aider, OpenCode (open source)	Cursor: an AI-first editor GitHub Copilot: agent mode inside VS Code and other editors Cline and similar editor extensions Cloud agents (Devin, Jules, Copilot coding agent): work on a remote copy of your repository and hand back a pull request

Terminal-native agents are a natural fit for statistics research, because so much of our work already happens in a shell: Rscript, python, make, git, ssh, and Slurm.

Core features: read, write, execute

Every coding agent is built on three capabilities:

• Read (cat, grep, glob): navigate the repository and find the 3 relevant files out of your 300.
• Write (edit, create): make targeted diffs rather than rewriting whole files. You can require approval for each edit.
• Execute (bash): run Rscript, pytest, make, git, and so on, then read the output.

The output of each command becomes new context, and the agent iterates until the tests pass. This is the key leap over chat assistants: the agent runs the code and sees what happens. There is no more copy-paste ping-pong between a browser tab and your terminal.

Read, write, execute, and feed the output back in.

Features beyond the basics

Modern harnesses add a lot on top of the core loop:

• Web search: look up documentation, papers, and error messages.
• MCP servers: the Model Context Protocol connects the agent to outside services such as GitHub, Slack, Notion, or Zotero.
• Test writing: unit tests, edge cases, and regression tests.
• SSH and Slurm: with SSH access configured, the agent can run sbatch and squeue on a cluster and pull results back.
• Subagents: parallel workers that each start with a fresh context and report back only a summary.
• Multimodal input: screenshots, plots, and PDFs of papers. The agent can also render a plot and look at its own output to fix overlapping labels before you ever see it.
• Memory files: CLAUDE.md (or AGENTS.md for some other agents) holds your project conventions and is loaded into every session.
• Skills and hooks: reusable workflows and automatic actions (see Skills below).

Features modern coding agents provide on top of the core loop.

The Cost of Using Agents

Agents have two costs: tokens (money and usage limits) and your time. They are linked, since a confused agent burns both, but they are not the same thing. A cheap, fast agent can still waste your afternoon.

Tokens

Recall that every turn of the loop re-sends the whole conversation. If the context grows by a roughly constant amount each turn, then the context size grows linearly and the total number of tokens processed grows quadratically in the number of turns. Prompt caching lowers the price of re-reading earlier context, but it does nothing about the context window's hard limit.

Context grows linearly, total tokens grow quadratically.

The biggest token sinks are:

• Long contexts: every turn re-reads the whole history.
• Dumping big outputs: a 50,000-line log, a full CSV, or print(model) on a large object, all of which stay in context for every later turn.
• High reasoning effort: thinking tokens are billed too.
• Open-ended prompts: the agent explores the whole repository to guess what you meant.
• Lost threads: re-explaining everything after the context is compacted, polluted, or gone because you started a new session.

Context rot

The context window is the agent's entire working memory. In a long session it fills up with a system prompt, memory files, and your task, plus a lot of noise: files it read that turned out to be irrelevant, logs and tracebacks, failed attempts, and old tangents. The useful work is often a small slice at the end.

That noise costs tokens on every turn, and it also degrades quality. Models get measurably worse as their context fills with irrelevant material (often called "context rot"). They lose instructions given early in the session, repeat fixes that already failed, and contradict earlier decisions. Eventually the agent hits the context limit and has to compact (summarize) its history, which loses detail.

A full context window is mostly noise.

Your time

Where does the wall-clock time of a "2-hour" agent session actually go? In an inefficient session, most of it goes to clarifying, waiting, back-and-forth, and fixing things. In an efficient one, you spend a few minutes on a spec and a plan, the agent does the work in the background, and you spend a few minutes reviewing the diff.

Where the time goes in an inefficient vs. an efficient session.

The main time sinks are:

• Wrong direction: the agent confidently builds the wrong thing, and you notice 30 minutes later.
• Opaque code: you end up debugging 600 lines you didn't write and don't understand.
• Memory loss: sessions don't share context, so threads get lost and rebuilt repeatedly.
• Ambiguity ping-pong: a vague ask produces a vague result, so you ask again.

The Role of the Researcher

A common misconception

Because agents talk like humans, it is tempting to assume they think like humans, and to treat them as a colleague who shares your intuitions and background. This leads to underspecified requests and surprising results.

A more useful mental model: we have found a way to write computer programs in natural language. Your prompt is source code. Like any program, it does what you wrote, not what you meant. Each step from assembly to C to Python/R to English has made programming higher-level and easier to write, but every step is still programming.

Prompts are programs written in English.

You still need to understand code

Using AI for statistics research still requires knowing statistics, or you can't tell a valid result from a confident hallucination. In the same way, using AI for code still requires knowing how code works, or you can't tell a good design from 3,000 lines of spaghetti.

You don't need to write every line yourself, but you do need to know enough to:

• specify the architecture you want,
• read a diff,
• spot a silent bug, and
• say "no, do it this way."

What happens when nobody reviews the agent's work.

Agentic AI is an Iron Man suit

A good way to think about the division of labor is an Iron Man suit:

The pilot (you)	The suit (the agent)
Where to go What "done" means What is correct When to abort Taste and judgment	Speed Stamina (it works 24/7) Breadth of knowledge Boilerplate at scale Tireless retries

A great suit makes a skilled pilot superhuman. It makes an unskilled pilot crash faster.

Best Practices

Saving tokens

Problem	Fix
Long contexts	One task per session. Use /clear between tasks, and /compact with a note on what to keep when you need to continue. A fresh context is cheaper and smarter.
Big outputs	Keep irrelevant data out of the context. Use grep or tail on logs, summarize data with head() or str(), and write results to files instead of printing them. Delegate noisy searches to a subagent that returns only a summary.
Reasoning effort	Match effort to the task: low for renames and typos, high for design decisions and nasty bugs. Use a smaller, faster model for grunt work.
Needless exploring	Say where to look. Point at files (@R/sampler.R) and name functions. Don't make the agent search for things you already know.
Going off track	Interrupt early with Esc. Esc Esc or /rewind rolls back to a checkpoint. Restarting from a good state beats a long correction thread.
Lost threads	Persist state outside the chat: CLAUDE.md for conventions, a PLAN.md checklist for multi-session work, and GitHub issues or markdown files for the backlog.

Saving time

Problem	Fix
Unclear goal	Say what done looks like: the goal, the constraints, and a concrete check ("done when the tests pass and the estimates match lm()"). Tell it to ask follow-up questions before starting.
Wrong direction	Plan first. Use plan mode, or say "propose a plan, don't edit yet." Reviewing a 10-line plan beats reviewing a 600-line diff.
Opaque code	Take small steps and commit each one. Ask for tests first, then the implementation. Ask the agent to explain any change you don't follow.
Repeating yourself	Package recurring workflows as skills or custom slash commands (e.g. /simstudy; see Skills). Use hooks to run formatters or linters automatically after edits.
Babysitting	Use auto mode. Allow-list safe commands (pytest, git diff, Rscript). Run in a sandbox or container if you want to give the agent more autonomy.
Waiting	Run it in the background (e.g. in tmux) and do other work. Run parallel sessions on separate git worktrees for independent tasks.

Prompts as specs

Since prompts are programs, write them like specifications. Compare:

Vague:

> make the simulation faster

The agent reads 40 files, rewrites the random number generator, parallelizes the wrong loop, and breaks reproducibility. An hour later your session usage is gone and the result is unusable.

Structured:

> Goal: speed up simulate_paths() in @src/sim.py; it is 90% of runtime.
  Context: profile output in prof.txt. The inner loop over t is pure Python.
  Constraints: keep the seeded RNG stream identical; no new dependencies
  beyond numba.
  Done when: tests/test_sim.py passes and bench.py shows >=5x speedup.
  First: propose a plan and ask me any questions before editing.

A good prompt has five parts: the goal, the relevant context (and where to find it), the constraints, a checkable definition of done, and an instruction to plan and ask questions first.

Skills

The problem skills solve

After a few weeks with an agent, you will notice that you keep typing the same instructions. Every simulation study needs the same seeding conventions, the same output format, and the same summary table. Every figure should use the same theme and fonts. Every Slurm script needs the same partition and module setup. You have two obvious options, and both are bad:

• Retype the instructions every time. This is tedious, and you will forget something, so results drift from one session to the next.
• Put everything in CLAUDE.md. The memory file is loaded into every session, so a long list of procedures costs tokens on every turn of every task, including the ones that have nothing to do with simulation studies. It also adds to the noise that causes context rot.

Skills are the middle ground: packaged, reusable instructions that are available in every session but loaded only when needed.

How skills work

A skill is a folder containing a SKILL.md file, optionally with supporting files such as templates, reference documents, or scripts. In Claude Code, project skills live in .claude/skills/<skill-name>/ inside the repository (so they are shared with everyone who clones it), and personal skills live in ~/.claude/skills/.

The SKILL.md file starts with a short header giving the skill's name and a description of when to use it:

.claude/skills/simstudy/
├── SKILL.md
├── templates/
│   ├── run_sim.R
│   └── simstudy.slurm
└── scripts/
    └── summarize_results.R

---
name: simstudy
description: Set up, run, and summarize a Monte Carlo simulation study.
  Use when the user asks to design or run a simulation study, compare
  estimators by simulation, or summarize simulation results.
---

# Simulation studies

1. Before writing code, confirm with the user: the data-generating
   process, the estimators being compared, the parameter grid, and the
   number of replications.
2. Copy templates/run_sim.R. Each replication takes (scenario, rep_id)
   and seeds with set.seed(1e5 * scenario + rep_id).
3. Write one .rds file per scenario to results/raw/. Never print raw
   results to the console.
4. Run a 10-replication smoke test locally before submitting anything.
5. Submit with templates/simstudy.slurm as an array job, one task
   per scenario.
6. Summarize with scripts/summarize_results.R: bias, RMSE, coverage,
   and Monte Carlo standard errors for each.

The key mechanism is progressive disclosure. At the start of a session, the agent sees only the name and description of each skill, which costs a few dozen tokens apiece. When a request matches a description, the agent reads the full SKILL.md, and it opens the templates or runs the scripts only when it reaches the step that needs them. You can have dozens of skills installed at almost no standing cost. You can also invoke a skill directly by name, for example by typing /simstudy.

Because skills can bundle scripts, they also let the agent run a deterministic tool instead of regenerating its logic each time. A summary script that always computes Monte Carlo standard errors the same way is more reliable than asking the model to rewrite that code in every session, and it saves tokens too.

Skills, CLAUDE.md, and hooks: Use CLAUDE.md for short facts that are true in every session ("this project uses renv", "never run heavy jobs on the login node"). Use a skill for a procedure that is needed only for certain tasks. Use a hook for something that must always happen after a given action, such as running styler after every edit to an R file. Hooks are enforced by the harness, while skills and memory files are instructions the model may or may not follow perfectly.

Example skills for statistics research

Skill	What it packages
Simulation study	The data-generating process / estimator / replication layout, seeding conventions, file outputs, Slurm array template, and a standard summary of bias, RMSE, coverage, and Monte Carlo error.
Paper figures	Your plotting theme, font sizes matched to the paper's column width, colorblind-safe palette, and export settings (vector PDF, correct dimensions). Can include a script that checks for overlapping labels.
LaTeX tables	Turning a CSV of results into a booktabs table with consistent rounding, bolded best entries, and standard errors in parentheses.
Cluster jobs	The department cluster's partitions, module loads, resource defaults, and etiquette rules, plus how to check on and resubmit failed array tasks.
R package check	Running devtools::document(), devtools::test(), and R CMD check, and fixing NOTEs and WARNINGs in a fixed order.
Reproducibility audit	A checklist for a project before submission or code release: pinned environments (renv / uv), seeds, hard-coded paths, and whether a clean clone reproduces every figure.
Paper reading notes	A fixed template for summarizing a paper: setting, assumptions, main result, proof technique, and how it relates to your work. Useful for building a consistent literature file.
Referee report	Your structure for reviewing a manuscript, from summary through major and minor comments, with a reminder to check derivations and simulation designs.

Tips for writing skills

• The description is what triggers the skill. Write it as "what this does, and when to use it", using the words you would actually type in a request. A skill with a vague description will never be picked up.
• Keep SKILL.md short. Put long reference material in separate files and tell the agent when to open them.
• Write the skill after doing the task by hand once. Do the workflow with the agent interactively, then ask it to turn the session into a skill and edit the result.
• Commit project skills to the repository so collaborators (and their agents) follow the same conventions.
• Only install skills from sources you trust. A skill can contain scripts that the agent will run, so read third-party skills before installing them.
• Check your agent's documentation. The skill format has been adopted by several coding agents besides Claude Code, but locations and details vary between tools and change over time.

Running Agents on a Remote Server

Agents are most useful when they can run unattended, so it pays to run them somewhere that stays on. A simple setup is to run the agent inside a tmux session on an always-on department server, with your laptop (or phone) serving only as a window into it:

• tmux: start a session with tmux new -s claude, detach with Ctrl-b d, and reattach from anywhere with tmux attach -t claude. The agent keeps working when you close your laptop.
• SSH: set up passwordless key-based authentication and put host details in ~/.ssh/config, so the agent can reach the cluster without ever seeing a password prompt.
• Shared server etiquette: the agent edits code and submits jobs; the cluster does the computing. Never let an agent run heavy computations on a shared login or interactive server.
• Git is your undo button: work on branches, protect main, and review every change before it is merged.

For a complete end-to-end workflow built on this setup, in which tasks are written as GitHub issues and the agent plans, implements, runs experiments on the cluster, and opens a pull request for review, see Agentic Workflow via GitHub Issues.

What Goes in CLAUDE.md

CLAUDE.md sits at the root of the repository and is read at the start of every session. Keep it short, specific, and limited to things that are true for every task. An example:

# Project: statbytes-agentic-example

## Cluster
- ssh: user@cluster (keys already set up)
- The cluster runs Slurm. Relevant partitions: ...
- Never run heavy jobs on the department server.
- Do not commit Slurm scripts.

## Conventions
- One branch per task.
- Never force-push; never touch main.
- Don't guess; ask clarifying questions.

Things that belong elsewhere: long procedures (make them skills), anything secret such as passwords or tokens (never put these in the repository), and task-specific plans (put them in a PLAN.md or an issue).

Common Pitfalls

Things to Watch Out For

Merging code you haven't read: The agent's code is your code once it is in your paper. Review every diff, and ask for explanations of anything you don't follow. One endless session: Long sessions get more expensive per turn and worse in quality. Start fresh for each task. Printing everything: Large outputs stay in context for the rest of the session. Write to files and summarize. Vague requests: "Make it better" gets you something different, not necessarily better. Give a goal, constraints, and a definition of done. Letting it run unchecked on shared resources: Allow-list commands deliberately, and keep heavy computation on the cluster's compute nodes. Giving it more access than it needs: Scope tokens (e.g. GitHub) to a single repository, keep secrets out of the repository, and protect your main branch. Bloated memory files: Every line of CLAUDE.md is paid for on every turn. Move procedures into skills.

Note: The tools change quickly, but the principles here don't: keep the context small and focused, write prompts like specifications, plan before building, and stay the pilot. For a full worked example, see Agentic Workflow via GitHub Issues and the demo repository.