AI21Labs/lift-bench

AI21Labs/lift-bench

Language: Python

License: Apache-2.0

Stars: 1

Forks: 0

Open issues: 0

Created: 2026-10-18T08:26:02Z

Pushed: 2026-10-18T20:04:02Z

Default branch: master

Fork: no

Archived: no

README:

LIFT-Bench

Instruction-following benchmark, training set and synthesis pipeline for document-drafting tasks whose constraints are scoped to individual sections.

Localized Instruction-Following Tasks (LIFT-Bench) is a benchmark of synthetic document-drafting tasks. A task gives the model a topic and an outline — a nested tree of headers whose leaves hold paragraphs, code blocks, tables and lists — and attaches constraints to individual elements of it.

Every constraint is *scoped*: it is scored only against the element it is attached to. That distinguishes "a paragraph in French somewhere" from "section 2.1 in French", and makes constraint density a generation parameter rather than an afterthought.

Each task mixes *verifiable* checks, run in code, with *judged* content requirements, run by an LLM judge, in a single weighted score. A response that meets every formatting constraint but skips the content still fails every judged constraint.

This repository releases:

• LIFT-Bench — 784 tasks over 10 splits, factored by document size, constraint density and scoping, with the

scoring code that computes the benchmark metric.

• LIFT-Train — a 34,258-task training counterpart. In our experiments, training on it was less prone to reward

hacking than training on lexical-only instruction-following rewards.

• The synthesis pipeline that generated both, so new splits can be produced for other domains, densities or

outline shapes. Tasks are emitted in the format consumed natively by PrimeIntellect's `verifiers`, ready for evaluation and RL training.

Quickstart

Requirements: uv (it installs Python 3.12 and the locked dependencies and runs vf-eval's harness) and an OpenAI-compatible endpoint.

uv sync --locked # Python, dependencies and .venv

export OPENAI_BASE_URL=http://localhost:8000/v1 # e.g. a local vLLM server
export OPENAI_API_KEY=...

Check the install with the smoke test. One self-contained task, no dataset:

./run_eval.sh

It runs environments/drafting_constraint_dummy/, a single task with an emoji-count, a lexical-match and a case-type constraint. An answer such as hello there, hope you're having a wonderful day! 😊🌟 meets all three: the trace comes back with reward 1.0 and info.agg_eval_scores [1.0, 1.0, 1.0].

Then run a real split:

VLLM_MODEL= NUM_EXAMPLES=100 ENV_ID=drafting_constraint \
./run_eval.sh --env.taskset.dataset-path /test.jsonl

Results land in .results/----null--/traces.jsonl, one episode per line. Each trace's info keeps the per-constraint scores and ids (agg_eval_scores, agg_eval_ids).

> `run_eval.sh` keeps the run local. It passes --env.agent.harness.id null, > --env.agent.runtime.type subprocess and --no-push. Keep those three if you call vf-eval directly: > its defaults are the bash harness (a shell for the model), Prime Intellect's cloud sandboxes, and > uploading the run to the Prime Intellect platform.

The judge

Judged constraints are scored by an LLM. Four things about it matter for a run:

• Which model. google/gemma-4-31B-it for every released split. The name is written into each task

(critic_model_name in the constraint's metric_args), not chosen at eval time.

• How to serve it. Through OPENAI_BASE_URL / OPENAI_API_KEY — by default the same endpoint as the model

under test. Set VLLM_BASE_URL and VLLM_API_KEY when the model under test lives somewhere else, and the judge keeps using OPENAI_*.

• How to point it elsewhere. Serve a different model at that endpoint under the same name, or rewrite

critic_model_name in the tasks.

• Comparability. Scores are only comparable across runs that used the same judge; report which judge was

served alongside the score.

Evaluating a model

The drafting_constraint taskset (environments/drafting_constraint) loads a split and scores each answer with constraint_reward. Beyond the quickstart command, --env.taskset.system-prompt adds a system prompt to every task. Instead of a local file (--env.taskset.dataset-path), --env.taskset.dataset-config --env.taskset.dataset-split reads a split straight from the Hub dataset, and --env.taskset.dataset-revision pins which version of it.

Both tasksets are packages in this repo's uv workspace, so uv sync installs them. To use one from another environment, such as a trainer's, install it from this checkout, which pulls in lift-bench with it:

uv pip install -e environments/drafting_constraint

> Reasoning traces are not stripped. Checks read the answer's message content as-is, assuming reasoning > arrived separately (e.g. your server's reasoning parser moved it to reasoning_content). A … > block left inline is scored as part of the answer.

Benchmark data

The splits are on the Hugging Face Hub as `ai21labs/LIFTBench`, one config per split with a train and a test file each. Download them into data/splits/, where the commands in this README expect them:

hf download ai21labs/LIFTBench --repo-type dataset --local-dir data/splits

To work with the data outside the taskset, load it with datasets (4.7 or later; lift-bench does not install it) instead:

from datasets import load_dataset

bench = load_dataset("ai21labs/LIFTBench", "short_doc_few_constraints", split="test")

datasets re-encodes the rows and rounds constraint weights to 10 decimal places, which moves a task's score by less than 1e-9; the JSONL files hold the exact values, and the taskset reads those directly.

Ten splits, from three generation runs — short outlines, long outlines, and flat. Within a run every split shares the same topics and outlines; only the number of constraints attached to them changes. The config behind each one is in configs/published_splits/.

| Split | Config | Outline | Sections with constraints | Verifiable / judged per section | | --- | --- | --- | --- | --- | | short/few | structured_l3_d2_sh3_sv2_snv2.yaml | ≤3 children, 2 levels | ≤3 | ≤2 / ≤2 | | short/some | structured_l3_d2_sh5_sv3_snv2.yaml | ≤3 children, 2 levels | ≤5 |...