novitalabs/tau2-bench

novitalabs/tau2-bench

Description: τ-Bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

License: MIT

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-08-12T05:53:34Z

Pushed: 2026-08-12T05:57:00Z

Default branch: main

Fork: yes

Parent repository: sierra-research/tau2-bench

Archived: no

README:

$\tau$-Bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

> How do you say $\tau^3$-bench? We just say "tau three," but you do you!

What's New in $\tau^3$-bench

> 📢 July 2026 — v1.0.1 grading update: This release fixes a couple of banking_knowledge task errors. Scores on that domain change as a result — results produced with tau2-bench = 1.0.1, and affected leaderboard submissions have been re-graded. Old results files can be re-scored with tau2 evaluate-trajs --fresh-tasks; to reproduce pre-fix behavior, pin the `pre-v1.0.1` tag. Details in the [changelog](CHANGELOG.md) and [release notes](RELEASE_NOTES.md). Other domains are unaffected.

• Knowledge Domain (`banking_knowledge`) — A knowledge-retrieval-based customer service domain with configurable RAG pipelines, document search, embeddings, and agentic shell-based search. [Learn more →](src/tau2/knowledge/README.md)
• Voice Full-Duplex (Audio Native) — End-to-end voice evaluation with realtime providers (OpenAI, Gemini, xAI). [Learn more →](src/tau2/voice/README.md)
• Task Quality (75+ fixes) — Removed incorrect expected actions, clarified ambiguous instructions, fixed impossible constraints, and added missing fallback behaviors across airline, retail, and banking domains. Based on analysis from SABER (Cuadron et al., 2025). Learn more →
• Updated Leaderboard — Now includes voice and knowledge results. Compare model performance at taubench.com. [Submit your results →](docs/leaderboard-submission.md)

See [CHANGELOG.md](CHANGELOG.md) for the full version history.

> Backward compatibility note: If you are evaluating an agent (not training), use the base task split to evaluate on the complete task set that matches the original τ-bench structure. This is the default.

> Upgrading from $\tau^2$-bench? Installation now uses uv instead of pip install -e ., and Python >=3.12, =3.10). Some internal APIs have been refactored — see [CHANGELOG.md](CHANGELOG.md) for details.

Overview

$\tau$-bench is a simulation framework for evaluating customer service agents across multiple domains. It supports text-based half-duplex (turn-based) evaluation and voice full-duplex (simultaneous) evaluation using real-time audio APIs.

Each domain specifies:

• A policy that the agent must follow
• A set of tools that the agent can use
• A set of tasks to evaluate the agent's performance
• Optionally: a set of user tools for the user simulator

Available domains: mock · airline · retail · telecom · banking_knowledge

| Mode | Description | |------|-------------| | Text (half-duplex) | Turn-based chat with tool use | | Voice (full-duplex) | End-to-end audio via realtime providers (OpenAI, Gemini, xAI) |

Quick Start

1. Install

git clone https://github.com/sierra-research/tau2-bench
cd tau2-bench
uv sync # core only (text-mode: airline, retail, telecom, mock)

Optional extras (install what you need):

uv sync --extra voice # + voice/audio-native features
uv sync --extra knowledge # + banking_knowledge domain (retrieval pipeline)
uv sync --extra gym # + gymnasium RL interface
uv sync --extra dev # + pytest, ruff, pre-commit (required for contributing)
uv sync --all-extras # everything

This requires uv. Voice features also need system dependencies (brew install portaudio ffmpeg on macOS). See the [full installation guide](docs/getting-started.md) for details.

2. Set up API keys

cp .env.example .env
# Edit .env with your API keys (uses LiteLLM — any supported provider works)

3. Run an evaluation

tau2 run --domain airline --agent-llm gpt-4.1 --user-llm gpt-4.1 \
--num-trials 1 --num-tasks 5

Results are saved to data/simulations/. Use tau2 view to browse them.

> Tip: Run tau2 intro for an overview of available domains, commands, and examples.

Documentation

Getting Started

| Document | Description | |----------|-------------| | [Getting Started](docs/getting-started.md) | Installation, API keys, first run, output structure, configuration | | [CLI Reference](docs/cli-reference.md) | All tau2 commands and options |

Core Concepts

| Document | Description | |----------|-------------| | [Agent Developer Guide](src/tau2/agent/README.md) | Build and evaluate your own agent | | [Domains](src/tau2/domains/README.md) | Domain structure, data format, and available domains | | [Orchestrator & Communication Modes](src/tau2/orchestrator/README.md) | Half-duplex and full-duplex orchestration | | [Task Schema & Evaluation](docs/evaluation.md) | What evaluation_criteria.actions means, how reward_basis gates the reward, and how to inspect action correctness |

Knowledge Retrieval

| Document | Description | |----------|-------------| | [Knowledge Retrieval](src/tau2/knowledge/README.md) | Retrieval pipeline configs, embeddings, RAG, and sandbox setup for the banking_knowledge domain |

Voice & Audio

| Document | Description | |----------|-------------| | [Voice (Full-Duplex)](src/tau2/voice/README.md) | Providers, speech complexity, CLI options, and output structure for voice evaluation | | [Audio Native Architecture](src/tau2/voice/audio_native/README.md) | Internal architecture for adding or modifying realtime provider adapters |

RL & Training

| Document | Description | |----------|-------------| | [Gym Interface](src/tau2/gym/README.md) | Gymnasium-compatible environment, play mode, train/test splits |

Leaderboard & Experiments

| Document | Description | |----------|-------------| | [Leaderboard Submission](docs/leaderboard-submission.md) | How to submit results to taubench.com | | [Experiments](src/experiments/README.md) | Experimental features and research code |

Project

| Document | Description | |----------|-------------| |...