amazon-science/Self-Evolving-Agents-Double-Ratchet

amazon-science/Self-Evolving-Agents-Double-Ratchet

Description: Reference implementation of Double Ratchet: co-evolving an inspectable evaluation metric with a lifecycle-managed skill library for self-improving LLM agents (arXiv:2607.12790)

Language: Python

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-07-29T03:15:00Z

Pushed: 2026-07-29T05:20:33Z

Default branch: main

Fork: no

Archived: no

README:

Double Ratchet

Double Ratchet co-evolves an inspectable evaluation metric with a lifecycle-managed skill library, so a self-improving LLM agent can be trained on tasks where no reliable metric exists to grade it. It is the reference implementation of Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents: the metric loop (evalratchet/) evolves a pool of drawback detectors and the expression that combines them, while the skill loop (vendor/skillevo/) evolves the agent's natural-language skills, both driven by a frozen LLM with no weight updates.

That skill loop is Ratchet, released on its own at Self-Evolving-Agents-Ratchet, with a copy pinned under vendor/ here so this repository runs standalone against one fixed version of it. Read that repository for the skill loop's own design and ablations, which assume a trustworthy metric is already given; read this one for the metric loop and the co-evolution driver that removes that assumption.

See [Reproducibility](#reproducibility) for what is and is not included here, and [Citation](#citation) for the BibTeX entry.

Layout

evalratchet/ the metric loop: op pool, expression language, anchored
selection, lifecycle (synthesis / birth gate / retirement),
and the co-evolution driver
ops/ per-task drawback-detector packs (mbpp, spider_sql, report_gen)
synthesizer/ op + expression synthesis
evolve.py op lifecycle: birth gate, shadow tier, merit retirement
storage/ metric/op stores (JSON/SQLite result files)
vendor/ the skill loop (Ratchet), pinned
skillevo/ skill bank, router, capsules, curator, engine
scripts/run_skill_loop.py the skill-loop driver the runners call
scripts/ experiment entry points (one per loop x task) + suite adapters
config/config.py single source of truth: models, roles, paths
datasets/ deterministic split builders (no data files included)
LICENSE Apache-2.0 (covers this repo, including vendor/)

Requirements

• Python >= 3.11; pip install pydantic pyyaml boto3
• Access to the Claude models named in config/config.py, served through

Amazon Bedrock. The runtime uses the standard AWS SDK credential chain, so export AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY, set AWS_PROFILE, or run on an instance with an IAM role. Set MODEL_API_REGION to override the default region.

• Spider 2.0-Snow only: pip install snowflake-connector-python plus

credentialed access to the Spider 2.0 hosted Snowflake warehouse. Place the official Spider2 repo (with its snowflake_credential.json) at ~/Spider2 or pass --spider2-repo / set SPIDER2_REPO. No credentials are stored in this repository; the connection factory reads that file at runtime and returns nothing if it is absent.

• Report generation only: this task needs report-generation data (tasks,

evidence cards, golden sections) plus its RAQS evaluator, which are not included in this repository. Point the runner at your own copy via --report-skills or REPORT_SKILLS. The MBPP+ and Spider 2.0 tasks below run without it.

Run everything from this directory; the scripts put vendor/ on the path themselves.

Datasets

No data files are included. Splits are built ONCE and stored; every experiment then reads the same stored files. The pipeline per task:

1. Screen (once): run the frozen solver 5x per task, grade with the official anchor, store per-task pass counts.

python scripts/screen_mbpp.py baseline && python scripts/screen_mbpp.py subset
python scripts/screen_spider2_snow.py # needs warehouse access

(Report generation needs no screening; its tasks derive from the golden demo reports in your report-generation data. Its dev soft labels come from scripts/label_report_gen_softlabels.py.)

2. Build the stored split (once): select the hard subset from the screen and freeze train / dev (eval_dev) / test (eval_locked) to datasets//splits/.json (deterministic, seed 42). Each builder reads the screening output from step 1, so run these in order.

For MBPP the two step-1 subcommands do this in sequence: baseline writes the per-task pass counts and subset selects the 60/40 hard split both loops share. build_splits.py then reads both files and adds the eval_dev anchor.

python datasets/mbpp/build_splits.py
python datasets/spider2/build_splits.py
python datasets/report_gen/build_splits.py

3. Run experiments: every run script loads the stored split file. --seed varies only the loop's stochasticity (LLM sampling, synthesis order), never the task selection, so all runs and all seeds of a task share the exact same train/dev/test tasks. If the stored split is missing the loader falls back to a random split and prints a WARNING; that fallback does not reproduce the paper.

Reproducing the paper's experiments

Every run writes results//_/ with round-by-round loop.json, a final result.json, and sqlite stores; every number in the paper is computed from these. The paper uses three seeds per run (--seed).

Config defaults already pin every loop role to Claude Opus 4.7 as in the paper (the report final judge is Opus 4.8, set in final_judge_report_gen.py); the $M flag below makes that pinning explicit per run. Shared flags: M="--single-model global.anthropic.claude-opus-4-7", CURR="eval:15,skill:25,eval:8,skill:25,eval:5,skill:25,eval:2,skill:25".

Metric loop (Figure 2, Table 3):

python scripts/run_metric_evo.py $M --rounds 100 --patience 100 --min-delta 0 --seed 42 --tag anchored
python scripts/run_metric_evo_spider2.py $M --rounds 100 --patience 100 --min-delta 0 --seed 42 --tag anchored
python scripts/run_metric_evo_report.py $M --rounds 100 --patience 100 --min-delta 0 --seed 42 --tag anchored

Ablation arms (Table 3): add --naive (anchor guards off) or --no-lifecycle (birth gate + retirement off; report script).

Reference skill loop (Figure 3 / Table 1, "skill" rows):

python scripts/run_skill_evo.py $M --rounds 100...