How To Stop Your Autoresearch Loop From Cheating

How to stop your autoresearch loop from cheating Skip to main content

Cerebras Announces First Quarter 2026 Results >>

Mar 19 2026 How to stop your autoresearch loop from cheating Sarah Chieng Sherif Cherfa

TLDR: We let an AI agent run overnight. By morning, it had abandoned our experiment and started its own. Across 71 experiments on two very different problems: training optimization and model compression, we learned that autoresearch can reliably surface real findings when the loop is tightly scoped. Loosen the guardrails, and the agent drifts within hours. The bottleneck isn't intelligence. It's everything around it. Everything we built/ran is open-source: codex-autoresearch-harness , Bash wrapper that forces Codex into a research loop with built-in A/B testing (Experiment 1) reap-expert-swap , Expert pruning + dynamic swapping to fit Kimi-k2.5 in BF16 (2.5 TB) onto 8× RTX 3090s (Experiment 2)

We left an AI agent running overnight on two research experiments. When we checked in the next morning, it had stopped doing what we asked. Instead of optimizing memory usage, it had gone off on its own side quest investigating how few model weights you actually need to maintain performance. Twelve hours of compute, pointed in the wrong direction.That experience captures both sides of autoresearch right now: it's powerful enough to surface real findings autonomously, and undisciplined enough to waste a full night of GPU time if you're not watching. Andrej Karpathy's Autoresearch Karpathy released autoresearch last week, a framework for letting AI agents run experiments autonomously. Give it a goal, some metrics to track, and a codebase. Let it propose changes, test them, keep or revert, repeat. He ran it for two days on Nanochat, a small model matching GPT-2 performance trained on ~$100 of compute. Codex managed to shave 10% off compute time through its own research. The setup is minimal, just 3 files:

Each iteration, the agent reads the program and results log, proposes a change to train.py , commits it, trains, and evaluates. A change is accepted only if it hits better-or-equal metrics. Everything else gets reverted via git reset. We wanted to know: does this hold up beyond the demo? We ran two experiments pushing autoresearch into different territory, one on training, one on inference, to find out :) What We Ran Experiment 1: Training optimization We wrapped Codex in a bash loop ( codex-autoresearch-harness ) and A/B tested two different LLMs as the "researcher", GPT-5.4 and Codex-Spark, on Karpathy's nanochat model. Codex doesn't natively support looping, so we built a harness to force it. Our question: can autoresearch actually do autonomous research, and how do different models behave/influence the final model? Experiment 2: Inference optimization We pointed autoresearch at a completely different problem, fitting Kimi-k2.5 in BF16, a 2.5 TB model onto $8K of consumer GPUs ( reap-expert-swap ). Our question: can agents figure out how to compress a model without destroying its ability to reason? Here's what happened. Experiment 1: Making Codex Loop Karpathy noted that Codex doesn't really work with autoresearch, codex exec runs once and exits. There's no built-in loop mode. So we built one. The Harness codex-autoresearch-harness puts codex exec inside a bash loop with built-in A/B testing and one experiment per call. Each iteration reads the program and results from disk, proposes a code change, commits, trains for 5 minutes, evaluates, keeps or reverts, logs to TSV, and exits. Bash catches the exit and starts a fresh call. For A/B testing, launch_ab.sh runs Model A for N hours, then Model B for N hours from the same baseline, and compare_results.sh generates a side-by-side analysis. One experiment per call was a deliberate architectural choice: it prevents the context window from overflowing (training output is verbose), gives clean error recovery (a crash just means the next iteration starts fresh), and guarantees state separation through git history and the persistent log. Setup Pitfalls Some annoying things that happened and how you can fix them: Every one of these is the same gap: current sandboxing treats GPU access and long processes as threats. Autoresearch treats them as requirements. Results: Convergent Discovery and Proposal Quality Each model got 6 hours on the same H100, same baseline: Both models independently discovered the same optimization, and proposal quality mattered far more than inference speed. Both models independently converged on learning rate warmdown scheduling as the primary lever. GPT-5.4 systematically hill-climbed the warmdown ratio from 0.5 to 0.95. Spark found the same strategy but with messier proposals. That convergence suggests the search landscape for small training improvements has real structure, different agents find the same peaks. The accept rates tell more. GPT-5.4 accepted 67% of its proposals. Spark accepted 17%. Not because Spark proposed bad ideas, it proposed more ideas, many of them bold. But each rejected proposal still burns 5 minutes of GPU time. Spark's speed advantage (~35 seconds faster per call) let it attempt nearly twice as many proposals, but most were wasted training runs. When each experiment costs real compute, proposal quality dominates total cost. A model that proposes fewer, better-targeted changes outperforms one that explores broadly, at least against a strict gate. Experiment 2: Shrinking a Giant Model onto Consumer GPUs The second experiment ( reap-expert-swap ) tackled a different problem: running a massive AI model on hardware that's way too small for it. The target model needs ~2.5 TB of VRAM. Our hardware: 8× RTX 3090s with 192 GB total. About 13× too small. In dollar terms, ~$300K of GPUs versus ~$8K. Our approach had two phases: first, compress the model permanently to fit on our hardware. Then, see if we could go further by loading and unloading parts of the model dynamically based on what it actually needs. Phase 1: Static Compression, This Worked (not autoresearch) Using REAP (expert pruning from Cerebras) plus INT4 quantization, we compressed the model from 717 GB down to 92 GB. That's 7.8× compression, enough to fit on our consumer GPUs. For context, GLM-4.7 in its original BF16 would require 4 B300s at ~$20/hour. GLM-4.7-Reap fits with full context on just 1 H200, 10× less cost. Static compression is a permanent decision, you pick...