amazon-science/Self-Evolving-Agents-Ratchet

amazon-science/Self-Evolving-Agents-Ratchet

Description: Reference implementation of Ratchet: a minimal hygiene recipe for self-evolving LLM agents (arXiv:2605.22148)

Language: Python

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-07-29T03:12:27Z

Pushed: 2026-07-29T05:20:27Z

Default branch: main

Fork: no

Archived: no

README:

Ratchet

Ratchet is a self-evolving single-agent system that turns its own failures into a governed library of natural-language skills, so a frozen LLM keeps improving on code and patch benchmarks without weight updates. It is the reference implementation of Ratchet: A Minimal Hygiene Recipe for Self-Evolving LLM Agents, whose contribution is the library's lifecycle discipline: skills are synthesised from clustered failures, injected as prompt guidance, and retired once they stop helping, under a bound that keeps the library from drifting.

Ratchet assumes a reliable metric already grades each attempt. Where none exists, Self-Evolving-Agents-Double-Ratchet co-evolves the metric alongside this skill loop and pins a copy of this repository as its skill loop.

Each task: route through an LLM with 0 or 1 skill injected as prompt guidance, log the attempt, judge it with a verbal critic. Each round: synthesise new skills from clustered failures; retire ones that don't help. A separate bank of meta-skills refines *how* skills are written. Nothing is deleted — retired skills keep their evidence for rollback.

Papers

This repository is the reference implementation for two papers:

• Ratchet: A Minimal Hygiene Recipe for Self-Evolving LLM Agents —

arXiv:2605.22148. The technical paper linked above: the full method, the non-divergence guarantee, and the ablations.

• **Library Drift: Diagnosing and Fixing a Silent Failure Mode in

Self-Evolving LLM Skill Libraries** — arXiv:2605.19576. Accepted to the ICML 2026 Workshop on Failure Modes in Agentic AI (FAGEN@ICML 2026), Seoul, South Korea.

See [Reproducibility](#reproducibility) for the reported configuration and how to rerun it.

Quickstart

Prerequisites

• Python 3.10+
• An AWS account with Bedrock access enabled in your region for:
• anthropic.claude-opus-4-7 (and claude-sonnet-4-6 as fallback)
• cohere.embed-v4
• (SWE-Bench only) Docker daemon running

1. Install

git clone https://github.com/amazon-science/Self-Evolving-Agents-Ratchet
cd Self-Evolving-Agents-Ratchet
pip install -e '.[dev,bedrock,mbpp]'
# add ',swebench' if you plan to use the SWE-Bench Verified suite

2. Set credentials

export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
# Optional but recommended (avoids HuggingFace rate limits):
export HF_TOKEN=...
# Optional model overrides:
# export SKILLEVO_BEDROCK_MODEL_ID=global.anthropic.claude-opus-4-7
# export SKILLEVO_BEDROCK_FALLBACK_MODEL_ID=global.anthropic.claude-sonnet-4-6
# export SKILLEVO_BEDROCK_EMBED_MODEL_ID=global.cohere.embed-v4:0

3. Verify the environment

skillevo doctor # _hard_t_e_seed.json`, then:

MBPP+ hard-100 (60 train / 40 eval)

python scripts/run_skill_loop.py mbpp --rounds 100 \ --subset results/subsets/mbpp_hard100_t60_e40_seed42.json

SWE-Bench Verified hard-150 (90 train / 60 eval) — same pattern

Selection is deterministic given the same base model, seed, and probe
count, so the same command reproduces the same subset.

### 5. Inspect

RUN=$(ls -td results/mbpp/mbpp_r* | head -1) python scripts/plot_skill_loop.py "$RUN/loop.json" skillevo --db "$RUN" skill list skillevo --db "$RUN" audit --limit 20

### Bringing your own skill bank

If you already have skills authored some other way, import them
as YAML matching `skillevo/skill/schema.py` (id / intent / signals_match /
guidance.{applies_when, key_insight, common_pitfalls, verify_before_returning}).
Drop the files into the run dir's `skills/` folder, then load:

skillevo --db results// skill load

Two flags support this case:

- **`do_not_retire: true`** in any skill YAML — exempts that skill
from contribution-score retirement *and* bank-cap eviction. Use for
compliance / business-required skills you must keep regardless of
measured contribution. The curator still emits a report with the
live score so you can review unlock candidates.
- **`--no-synth`** — disables per-round skill synthesis. Capsules,
verdicts, stats, and curator still run, so you get a full audit of
the existing bank without any new skills being authored.

Recommended phased rollout:

1. **Audit (read-only)**: `--no-synth` + every skill `do_not_retire: true`.
You get per-skill contribution-score reports without changing the bank.
2. **Curated retirement**: drop `--no-synth` (still no new skills) but
selectively unlock skills the audit flagged as low-performing.
3. **Full self-evolution**: drop the locks. Synth + curator manage the
bank end-to-end.

**Missing fields are tolerated.** `signals_match`, `tags`, `preconditions`
all default to empty lists; the retriever drops empty fields from its
scoring surface and still matches on `intent`, `name`, and `description`.
You should still set **`intent`** (one short imperative sentence:
"Sum a list of numbers") and **`description`** (one line describing
when the skill applies) — without those two, the retriever has nothing
to score against and the skill becomes effectively un-routable.

### Bringing your own task

Custom suites plug in via `skillevo.adapters` — no fork. See
[`examples/custom_task/`](examples/custom_task/) for a worked example
(CSV/JSON record extraction, ~150 lines).

from skillevo.adapters import BenchResult, register

class MyAdapter: name = "my_suite" def load(self, n_train, n_eval, seed, kw): ... # → (train, eval) lists def run(self, engine, capsules, tasks, kw): ... # → BenchResult def pack_cli_kwargs(self, args): return {} # forward CLI flags

register("my_suite", MyAdapter)

export SKILLEVO_ADAPTERS=path/to/my_adapter.py python scripts/run_skill_loop.py my_suite --rounds 5

Set `tags=["my_suite", "python_code"]` on each task to reuse the
code-gen prompt. For non-code outputs, add a template to
`skillevo/llm/prompts.py` and a branch in `skillevo/engine/solver.py`.

---

## Architecture at a glance