Experiment driven development with agents

Aug 6, 2026 11 min read

On several of my side projects I’ve found a great pattern for using agents to autonomously test out new ideas, measure their impact, and track that knowledge for future iterations.

run-experiments is a repeatable skill for an agent to iterate on an experiment or exploration and document the findings in a standardized format. It also helps the agent focus on the right project-specific metrics. This enables the agent to loop for hours and test out many different approaches and optimizations.

run-experiments A reusable agent skill for planning, running, and documenting reproducible experiments with project-owned metrics and conventions. jonniesweb/skills on GitHub

When building novel products, lots of time goes into research and development to figure out what works and what doesn’t. This often means a lot of experimentation and measurement. AI agents are good at implementing ideas, but they need clear goalposts: a metric such as quality of X or accuracy of Y, that lets them compare version A with version B. If they know which metric ultimately matters, agents can be much more successful at delivering higher-quality outcomes over long loops of experimentation and iteration. The goal of optimizing a metric can be put into a prompt, but it’s much more effective if there’s a repeatable process for rapid research and experimentation.

Beyond optimizing a metric, agents can experiment even faster by testing several ideas at once and quickly identifying what worked. Once the agent is done working, potentially for hours, evaluating its results and takeaways can take some serious time if you’re manually reading through the agent’s output. You’re also missing out on all of the knowledge and learnings from that agent’s session in future sessions if none of it is saved anywhere. This is where we can have the agent document its findings in a standardized, human-readable format that retains enough detail for both humans and agents to critique and improve.

Because every project is different, its experiments should be customized accordingly. Over time I find that the domain-specific experimentation knowledge evolves to make running experiments easier and more effective. This can include CLI tools for running simulators, go-to datasets, evaluation methods, and a domain-specific scripting language, all documented in CLAUDE.md. This saves the run-experiments skill from having to come up with one-off harnesses to evaluate each experiment. After running several experiments this way, have your agent review the reports and past agent sessions. It can identify repeated work and common pain points that should become instructions, scripts, or tooling.

Because agents can loop and converge on the initial goal, they can test many more approaches and optimizations than a human driving a single agent ever could. Claude’s /loop or Codex’s /goal are both ways to enable this underrated feature. Looping tells the agent that it can take its time and focus on smaller chunks of the goal over many loop iterations rather than trying to accomplish everything in one single iteration. The result of this is that many more experiments and variations can be run from a single prompt. If you have the tokens for it, why not get even more research from it?

A real example

So what does this process look like in practice? For an audio transcription and processing pipeline project that I’ve been working on, I often want to improve the transcription accuracy by trying out systems such as Parakeet and WhisperX, along with different configurations. I would start by prompting my coding agent to /loop experiment improving transcription accuracy by using Parakeet over WhisperX. Optimize its configuration too. Then the agent does the following:

the agent initializes the experiment by creating a report in the experiments/2026-08-05-parakeet-vs-whisperx/ directory and picking the most relevant metrics to optimize and track (word accuracy, transcription speed, cost, etc.)
the agent performs its first iteration by implementing the new Parakeet model, running it against the project’s existing evals and noting down the results and learnings in the report
the agent continues iterating on the experiment by optimizing configuration options to improve transcription accuracy and noting down the results and learnings in the report
the agent reaches a point where it has tested out several configurations and models and has found the best one compared to the baseline
the agent completes the experiment by concluding the report with the best solution, its takeaways, learnings, gotchas, and future improvements

All of this is possible through my run-experiments skill. An example of the report looks like this:

2026-08-05-parakeet-vs-whisperx.md

Parakeet vs production WhisperX for transcription

Candidate	Type	Dataset / feed	Sample	Primary metric	Guardrails / secondary metrics	Runtime / cost	Biases / caveats	Verdict
Production: WhisperX default pyannote VAD	production baseline	CYVR tower	3 clips, 242 s, ground truth	ATC-WER 0.6504 ↓	naive WER 0.8398; 567 words; 0 empty	CPU int8, 255.6 s ASR	small-n, feed-specific, imperfect-ground-truth, production-path, prior deterministic run	keep
Parakeet v2, full-file bfloat16	ASR model + decode	CYVR tower	1 calibration + 2 evaluation clips	ATC-WER 0.8046 ↓; evaluation 0.8237	naive WER 0.8158; 281 words; 0 empty; severe 1130Z under-generation	Apple Silicon MLX, 5.2 s ASR, local	small-n, feed-specific, same-corpus tuning, imperfect-ground-truth; 1000Z smoke-exposed	rejected for accuracy
Parakeet v3, full-file bfloat16	ASR model	CYVR tower	1 calibration clip, ground truth	ATC-WER 0.8000 ↓	naive WER 0.8155; 125 words	Apple Silicon MLX, 7.9 s ASR	small-n, feed-specific, calibration-only	rejected
Parakeet v2 + Silero, merge gap 3 s	VAD	CYVR tower	1 calibration clip, ground truth	ATC-WER 0.8145 ↓	5 speech regions/116.0 s; 107 words	Apple Silicon MLX, 2.5 s ASR+VAD	small-n, feed-specific, calibration-only	rejected
Parakeet v2, 30 s chunks/5 s overlap	decode/chunking	CYVR tower	1 calibration clip, ground truth	ATC-WER 0.8436 ↓	only 81 words	Apple Silicon MLX, 2.2 s ASR	small-n, feed-specific, calibration-only	rejected

Date(s): 2026-08-05
Status: concluded
Decision: Keep the current WhisperX production backend. Parakeet is dramatically faster on this Mac, but every tested Parakeet configuration is less accurate on the fixed ATC corpus.
Related: 2026-06-21-atc-asr-wer-metric-and-vad.md

Hypothesis & setup

Question: Can NVIDIA Parakeet TDT 0.6B beat the current ATC-fine-tuned WhisperX production path on radio transcription, and which reproducible Parakeet inference configuration works best?

Hypothesis: Parakeet’s newer FastConformer-TDT engine will recover more radio speech than WhisperX while preserving ATC numbers; the English-only v2 model without artificial chunk boundaries will be the strongest starting point.

Method:

Corpus and sampling rule: the existing three afftdn-processed CYVR tower clips in evals.json, totaling about 242 seconds; no clips were added or removed after seeing results.
Tuning/calibration set: 1100Z (122.96 s), selected before the configuration sweep because it is the longest clip and exercises unchunked, chunked, and VAD-segmented inference.
Evaluation set: 1000Z (49.37 s) and 1130Z (69.44 s). The smoke command selected the smallest file, 1000Z, and its score was inspected before configuration selection. That is a protocol deviation: 1000Z is tagged smoke-exposed, though no 1000Z result was used to rank configs. The untouched 1130Z clip is the only strictly held-out sample.
Production baseline: WhisperX default pyannote VAD (vad_onset=0.5, vad_offset=0.363), no_repeat_ngram_size=3, CPU int8. The config was reconfirmed on 2026-08-05; its deterministic saved production-path outputs score 0.6504 ATC-WER.
Primary metric and direction: word-weighted ATC-normalized WER, lower is better. Both ATC and naive WER are printed, but only ATC-WER selects the winner. The scorer’s 11-case self-test passed.
Guardrails / secondary metrics: zero empty outputs; inspect word-count inflation/deflation and per-clip WER; record wall time. Alignment and diarization are excluded because they do not change transcript words.
Confirm if: the selected configuration lowers evaluation ATC-WER by at least 5% relative to the production baseline on the same clips, with no empty output or obvious insertion/omission explosion. Smaller improvements would be promising but insufficient for production replacement.
Falsify if: evaluation ATC-WER is no better than production, or any apparent gain depends on a configuration selected using evaluation clips.
Environment: macOS 26.1 arm64, Apple Silicon Metal, mlx-audio==0.4.6; MLX Community ports of nvidia/parakeet-tdt-0.6b-v2 and nvidia/parakeet-tdt-0.6b-v3; branch codex/parakeet-asr-experiment.
Cost: local execution only; approximately 5 GB of gitignored model weights; no paid API calls.

Rounds

Round 1 — runtime smoke test

Tried: Loaded Parakeet TDT 0.6B v2 through the pinned MLX-Audio runtime and transcribed the shortest reference clip using unchunked bfloat16 inference.

Result:

Candidate	Primary metric	Guardrails / secondary metrics	Runtime / cost	Notes
Production	ATC-WER 0.5833	139 words	63.5 s ASR	saved deterministic baseline
Parakeet v2 full bfloat16	ATC-WER 0.6795	101 words; non-empty	3.5 s first ASR, after 91.6 s initial load/download	runtime valid, accuracy worse

Read: MLX Parakeet runs correctly and is fast, but the first transcript omits and garbles more ATC content than production. The smoke score was inspected even though 1000Z had been designated for evaluation; it was excluded from configuration selection and is explicitly tagged above.

Next: Tune on 1100Z only and freeze the winner before running the final comparison.

Reproduce:

# See experiments/parakeet-vs-whisperx/README.md for pinned commands and cache variables.

Raw outputs: experiments/parakeet-vs-whisperx/

Round 2 — configuration calibration on 1100Z

Tried: Compared English-only v2 vs multilingual v3, bfloat16 vs float32 preprocessing, full file vs fixed 30/60-second overlapping chunks, and three Silero speech-segmentation settings. All candidates used greedy TDT decoding, the only decoder exposed by this pinned MLX implementation.

Result:

Candidate	ATC-WER ↓	Naive WER	Words	Runtime
Production WhisperX	0.6982	0.8240	250	116.4 s
Parakeet v2 full bfloat16	0.7782	0.7811	151	2.7 s
Parakeet v3 full bfloat16	0.8000	0.8155	125	7.9 s
Parakeet v2 full float32	0.8036	0.8326	150	2.8 s
Parakeet v2 + Silero, merge gap 3 s	0.8145	0.8498	107	2.5 s
Parakeet v2 + Silero default	0.8364	0.8369	101	4.2 s
Parakeet v2 + Silero loose	0.8364	0.8369	102	2.9 s
Parakeet v2, 30 s/5 s overlap	0.8436	0.8155	81	2.2 s
Parakeet v2, 30 s/2 s overlap	0.8582	0.8541	101	2.8 s
Parakeet v2, 60 s/2 s overlap	0.8727	0.8584	82	2.1 s

Read: Unchunked English v2 with bfloat16 is the optimized Parakeet configuration. V3 and float32 change the transcript but lower accuracy. Fixed chunking and Silero segmentation discard more words and worsen WER. Even the Parakeet winner is 11.46% relatively worse than production on the calibration clip, although its ASR stage is 43.0 times faster.

Next: Freeze parakeet_v2_full_bf16; run no other candidate on the evaluation clips.

Round 3 — frozen evaluation comparison

Tried: Ran only parakeet_v2_full_bf16 on 1000Z and 1130Z and compared it with the matching production outputs. No configuration changed after these transcripts were scored.

Result:

Candidate	Evaluation ATC-WER ↓	Per-file ATC-WER (1000Z / 1130Z)	Naive WER	Words	Runtime
Production WhisperX	0.6158	0.5833 / 0.6384	0.8516	317	139.2 s
Parakeet v2 full bfloat16	0.8237	0.6795 / 0.9241	0.8419	130	2.5 s

Read: Parakeet is 33.76% relatively worse on evaluation ATC-WER. It is 55.5 times faster in measured ASR time, but that gain comes with severe omission: on 1130Z it emits only 29 words versus production’s 178 and scores 0.9241 WER. The untouched 1130Z result alone therefore falsifies the accuracy hypothesis. Across all three clips, Parakeet is 0.8046 vs production’s 0.6504 ATC-WER (23.71% relatively worse) while running 49.0 times faster.

Next: Do not replace WhisperX for accuracy. Treat Parakeet as a possible future throughput candidate only after domain adaptation and a larger ground-truth corpus.

Conclusions

Keep WhisperX default pyannote VAD in production.
The optimized Parakeet configuration is v2, unchunked, bfloat16. It does not improve accuracy: all-corpus ATC-WER is 0.8046 vs 0.6504, and evaluation ATC-WER is 0.8237 vs 0.6158.
Multilingual v3, float32, fixed overlapping chunks, and Silero VAD all worsen calibration WER.
Parakeet’s real advantage is throughput, approximately 49 times the baseline ASR stage on this Apple Silicon machine. That is insufficient for this accuracy-driven decision.
The corpus is only three CYVR clips, 1000Z was smoke-exposed, and the ground truth is imperfect. The large loss on untouched 1130Z is nevertheless too large to justify a production trial.

Ideas / next time

Add hand-transcribed clips from multiple airports before revisiting the engine choice.
Re-evaluate a domain-fine-tuned Parakeet checkpoint if one becomes available; generic decode and VAD tuning did not close the ATC vocabulary gap.
If throughput becomes the primary constraint, test a two-pass Parakeet/WhisperX routing scheme under an explicit quality guardrail rather than replacing WhisperX globally.

As you can see, the top of the report contains the most important information: a results table with metrics from each iteration, followed by the overall conclusion and recommendations. The rest of the report goes into the experiment hypothesis, method, and details of each round of the experiment. The report ends with additional conclusions, assumptions, and ideas for next time.

This output format removes the need to read the agent conversation. Instead, you fire your experiment off and come back to read the report when it’s done. I often have the agent open a PR when the experiment is done, so I can review the report and its files on GitHub.

Once an agent finishes up its report, I spend most of my time reading the main results table and conclusion sections to gather what worked, what didn’t, and whether there were any gotchas or things to try next. More ideas or tweaks to the experiment naturally develop in my mind at this point, so it’s then time to fire off a few new agents. The cycle continues!

Adding and using the skill

Add the skill to your agent with a few quick commands:

For Claude Code plugins:

claude plugin marketplace add jonniesweb/skills --scope user
claude plugin install run-experiments@jonniesweb-skills --scope user

For Codex plugins:

codex plugin marketplace add jonniesweb/skills --ref main
codex plugin add run-experiments@jonniesweb-skills

On first use, such as when you mention running an experiment or call /run-experiments directly, your agent will configure the skill for your project by determining its metrics and the locations for reports and evidence. You can view or manually configure the overrides added to your CLAUDE.md or AGENTS.md file. A customized template report is also added to your project for the agent to use each time an experiment is run. Feel free to customize the template to your liking.

Claude’s /loop or Codex’s /goal work exceptionally well with this skill. Given an open-ended prompt to broadly explore or try out several ideas, the agent will automatically use the same experiment file and treat each variant as a separate candidate in the report. The agent will keep exploring until it fulfills the initial prompt.

Another benefit of the skill is that your initial prompt, CLAUDE.md, and the template in your repo all take precedence over the skill’s defaults, in that order. This means that you can guide your agent to add new capabilities to your project as it goes instead of keeping all of its changes within the experiment directory.