amazon-science/PatientAgentBench

amazon-science/PatientAgentBench

Description: PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents

Language: Python

License: NOASSERTION

Stars: 0

Forks: 0

Open issues: 1

Created: 2026-07-10T16:52:57Z

Pushed: 2026-07-29T04:49:15Z

Default branch: main

Fork: no

Archived: no

README:

PatientAgentBench

This repository hosts the reference implementation for the paper > [PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents](https://arxiv.org/abs/2607.25485)

PatientAgentBench evaluates patient-facing health AI agents through realistic, multi-turn conversations in primary care contexts. A simulated patient converses with the system under evaluation — itself an agent, a base model wrapped in a harness — which reasons over the patient's health record and acts through a stateful sandbox of healthcare tools to carry out real workflows (scheduling, prescriptions, telehealth, escalation to clinical staff). Each conversation is scored by an LLM-as-a-Jury against a reusable suite of over a hundred clinician-grounded, conversation-agnostic criteria across six dimensions.

Scenarios are generated fresh on demand from a configurable seed distribution — the framework ships no fixed dataset, so there is no answer key to memorize and it extends to new models and clinical domains without extra annotation. All patient profiles, records, and conversations are fully synthetic; no real patient health information is used at any stage. For experimental results and findings, see the paper.

Leaderboard

Per-agent results as reported in the paper, sorted by aggregate score. Each rubric cell shows the mean score (1–5 scale) and the pass rate — the percentage of conversations scoring ≥ 3 (Adequate or better) — as mean [pass%]. See the paper for methodology, confidence intervals, and additional detail.

| # | Agent | Aggregate | Task Completion | Clinical Safety | Workflow Accuracy | Triage Quality | Clinical Helpfulness | Conversational Quality | |---|-------|:---------:|:---:|:---:|:---:|:---:|:---:|:---:| | 1 | Claude Opus 4.8 | 4.25 | 4.84 [100%] | 4.18 [98%] | 4.41 [98%] | 3.70 [88%] | 4.23 [100%] | 4.40 [100%] | | 2 | GPT-5.5 | 4.22 | 4.79 [100%] | 4.16 [99%] | 4.32 [98%] | 3.49 [76%] | 4.19 [99%] | 4.69 [100%] | | 3 | Claude Sonnet 5 | 4.20 | 4.86 [100%] | 4.04 [98%] | 4.28 [97%] | 3.77 [86%] | 4.14 [100%] | 4.42 [100%] | | 4 | GPT-5.4 | 4.16 | 4.85 [100%] | 4.02 [98%] | 4.44 [98%] | 3.45 [82%] | 4.04 [99%] | 4.56 [100%] | | 5 | Gemini 3 Flash | 3.78 | 4.69 [100%] | 3.57 [91%] | 3.72 [89%] | 3.03 [60%] | 3.82 [98%] | 4.49 [100%] | | 6 | Gemini 3.1 Pro | 3.71 | 4.65 [100%] | 3.52 [94%] | 3.95 [95%] | 2.66 [36%] | 3.66 [97%] | 4.39 [100%] | | 7 | Claude Haiku 4.5 | 3.63 | 4.52 [99%] | 3.43 [91%] | 4.02 [93%] | 2.80 [47%] | 3.63 [96%] | 3.65 [97%] | | 8 | GPT-OSS-120B | 3.45 | 4.56 [100%] | 3.22 [80%] | 3.36 [73%] | 2.78 [46%] | 3.63 [93%] | 3.62 [90%] | | 9 | Qwen3-235B | 3.40 | 4.49 [99%] | 3.18 [79%] | 3.30 [72%] | 2.55 [32%] | 3.68 [94%] | 3.79 [93%] | | 10 | Qwen3-Next-80B | 3.10 | 4.17 [93%] | 2.68 [56%] | 2.98 [58%] | 2.56 [32%] | 3.57 [89%] | 3.16 [69%] |

Architecture

PatientAgentBench is a three-phase pipeline: a benchmark case is generated from a sampled seed (patient profile + scenario + initialized sandbox), a multi-turn conversation is run between the simulated patient and the assistant agent, and the resulting conversation trace is scored by the evaluation framework.


The conversation loop is a dual-agent exchange: the assistant is a LangGraph ReAct agent that reasons and acts against the healthcare sandbox, while the simulated patient responds in persona. Every turn is captured in a structured conversation trace (messages, tool calls, and tool results).


Overview

PatientAgentBench has four main components:

1. Assistant Agent — the health AI system under evaluation: a base model wrapped in a LangGraph ReAct harness that reasons over the patient's context and acts through the sandbox tools 2. User Agent — a simulated patient that converses in persona, driven by a synthetic health record and clinical scenario 3. Healthcare Sandbox — stateful tools across four categories: appointments, prescriptions, telehealth, and profile management 4. Evaluation Rubrics — LLM-as-a-Jury scoring across six clinician-grounded dimensions, using conversation-agnostic criteria that generalize across scenarios

Installation

PatientAgentBench targets Python 3.11+. Clone the repository, then use a virtual environment:

git clone https://github.com/amazon-science/PatientAgentBench.git
cd PatientAgentBench

python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate

pip install -e . # runtime install
pip install -e ".[dev]" # with test/lint/type-check tooling

Dependencies are declared in pyproject.toml; the commands above install everything needed. For exact reproducibility, requirements.txt pins the specific versions used during development — install it with pip install -r requirements.txt if you need to reproduce that environment precisely (otherwise it is not required).

Model Providers

PatientAgentBench supports three model access channels, and any role (assistant, simulated patient, evaluator, sandbox, seed generator, analyzer) can use any of them:

• AWS Bedrock ("provider": "bedrock") — Bedrock-hosted models via

ChatBedrockConverse.

• OpenAI-protocol API ("provider": "openai-protocol-api") — any

OpenAI-compatible endpoint via ChatOpenAI (the OpenAI API, a LiteLLM proxy, or the AWS Bedrock Mantle gateway).

• Anthropic-protocol API ("provider": "anthropic-protocol-api") — the

Anthropic API (or the Bedrock Mantle gateway) via ChatAnthropic.

The channel is selected per model in the config. Bedrock is the default when provider is omitted; registry keys encode the channel in their -bedrock/-api/-mantle suffix, so they need no separate provider. Copy .env.example to .env and fill in credentials for the channels you use:

cp .env.example .env

AWS Credentials

For Bedrock, credentials are resolved through the standard AWS credential chain (boto3), in this order:

1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY,...