novitalabs/novita-gym

novitalabs/novita-gym

Description: Verifiable task environments on Novita Sandbox — convert SWE benchmark images (SWE-bench, SWE-rebench, Scale-SWE, ...) into instant sandbox templates, then evaluate with official-parity grading. No local Docker.

Language: Python

License: MIT

Stars: 1

Forks: 0

Open issues: 4

Created: 2026-07-27T08:42:19Z

Pushed: 2026-07-30T04:06:30Z

Default branch: main

Fork: no

Archived: no

README:

---

Highlights

• 🏗️ One-time conversion — each dataset's per-instance Docker image becomes a Novita template, built server-side; every attempt after that boots in seconds
• ✅ Official-parity grading — SWE-bench via the canonical swebench package; SWE-rebench-V2 mirrors the official eval.py byte-for-byte (same apply flags, same strict resolved rule)
• 📊 Proven at full-dataset scale — the whole of SWE-bench Verified has been built and gold-verified on Novita (fleet being rebuilt on 2c4g; manifests land in templates/)
• 🔁 Resumable everything — batched builds with a persistent ledger, crash-safe verdicts, automatic failure logs, 429/SSL retry
• 🧭 Dataset quality built in — curated leaderboard sets, upstream is_lite filters, rotted-task exclusion

uv venv && uv pip install -e . && export NOVITA_API_KEY=...

novita-gym smoke swebench # prove the loop on 1 instance (~3 min)
novita-gym verify swebench --all # build + gold-verify every environment
novita-gym eval swebench --predictions preds.jsonl # score your model

What a smoke test looks like

$ novita-gym smoke swerebench_v2 wtforms__wtforms-614
Ensuring template (converting the prebuilt image if needed) ...
template: swerebench-v2-wtforms-wtforms-614 (built, 7s)
Verifying the environment inside a fresh sandbox ...
/wtforms HEAD: 848d28d67e45 (base_commit 848d28d67e45 ancestor? yes) OK
GOLD patch (expect resolved=True) ...
resolved=True applied=True
EMPTY patch (expect resolved=False) ...
resolved=False

PASS ✅ harness round-trips correctly

The problem it solves

Modern LLM engineering runs on *verifiable tasks*: an environment, an objective, and held-out checks that decide success. The SWE benchmark family (a real repository + an issue + tests) is the largest and best-standardized case, and is where novita-gym starts. Two workflows consume these tasks —

• Evaluation (what novita-gym does today): score a model's patches

against a benchmark, with results comparable to the official leaderboards.

• RL data generation: agents attempt tasks at scale and the test outcome

becomes the reward signal. novita-gym doesn't run this loop, but provides its two hardest ingredients — reproducible environments and a leaderboard-parity reward function.

Both need the same expensive thing: thousands of isolated, reproducible execution environments. The official harnesses assume a local Docker daemon; that stops scaling exactly when fleet-scale workloads begin.

novita-gym moves the whole loop onto Novita sandboxes: each dataset's published per-instance Docker image is converted (server-side) into a Novita template once, and every attempt thereafter is a fresh sandbox that boots ready-to-test. Grading stays byte-compatible with each dataset's official harness — so a score from novita-gym is reportable, and anyone using the grade as a reward signal gets the *same number* the leaderboard would produce.

Boundary: novita-gym grades patches; it does not train models and is not an agent scaffold. It is the environment + reward layer that training pipelines and eval pipelines share.

> Why "gym"? In RL, a *gym* (after OpenAI Gym) is a standardized, > interactive environment an agent can act in and get a reward from. That is > what this provides for SWE tasks — spawn, attempt, get graded. Evaluation > is the same environment exercised once with a fixed patch; the SWE training > ecosystem already uses the term (SWE-Gym, R2E-Gym).

Datasets

novita-gym datasets prints this live. Every command takes the dataset name as its first argument; results are kept apart under results//.

| name | source | notes | | --- | --- | --- | | swebench | princeton-nlp/SWE-bench_Verified (or Lite/Full via env) | the standard benchmark; canonical swebench grader; full Verified set previously built & gold-verified (rebuild in progress) | | swerebench_lb | nebius/SWE-rebench-leaderboard | 860 curated instances + monthly decontamination splits (2025_01…2026_03) — use for formal SWE-rebench evaluation | | swerebench | nebius/SWE-rebench | raw 21k mined set; meta.is_lite quality filter on by default (~1.1k kept) | | swerebench_v2 | nebius/SWE-rebench-V2 | 32k multi-language task release (default filter: python); grading mirrors the official eval.py | | scale_swe | AweAI-Team/Scale-SWE | 20k Python instances; f2p_script-based tests; grading mirrors the official AweAgent evaluator (merged F2P+P2P pytest run) | | r2e_gym | R2E-Gym/R2E-Gym-Lite (or Subset/V1 via env) | 4.5k instances, 10 repos; synthetic tests baked into the image; grading mirrors the official _calculate_reward_r2e (exact status-map match) |

Planned: SWE-Lego · SWE-smith · OpenSWE · Multi-SWE-RL.

Quality note: raw mined sets contain *rotted* tasks — e.g. SathyaBhat__spotify-dl-311's tests download from YouTube and call the live Spotify API during the test run, which now fails in every harness (bot checks / 403). The curated sets exclude them upstream; the raw adapter's default filter excludes them here.

Adding a dataset — including your private one is one adapter file answering four questions (load / image / exercise / grade). Full guide with the onboarding loop, grading-rule choices, and private-registry notes: [docs/adapters.md](docs/adapters.md).

Released templates (the artifact)

Datasets we have already converted and gold-verified ship as a static artifact: [templates/](templates/) lists every released dataset, the deterministic template-naming rule, and per-instance manifests (template name + source image + verify result). (Fleet is being rebuilt on the new 2c4g resource profile — manifests land back here as each dataset's verify completes.)

Commands

novita-gym datasets list supported datasets
novita-gym smoke [instance_id] 1 instance end-to-end: convert image,
gold patch must resolve, empty must not
novita-gym build convert images -> Novita templates
novita-gym verify build + gold-verify, resumable ledger
novita-gym status [dataset]...