A new benchmark for evaluating patient-facing health AI agents
Captured source
source ↗A new benchmark for evaluating patient-facing health AI agents - Amazon Science
Close
Close
Social
bluesky
threads
youtube
github
rss
Menu
Research
Research areas
Automated reasoning
Cloud and systems
Computer vision
Conversational AI
Economics
Information and knowledge management
Machine learning
Operations research and optimization
Quantum technologies
Robotics
Search and information retrieval
Security, privacy, and abuse prevention
Sustainability
Our scientific contributions
Publications
Research from our scientists and collaborators.
Conferences
Our experts present and discuss cutting-edge research at scientific meetings globally.
Research areas
Automated reasoning
Cloud and systems
Computer vision
Conversational AI
Economics
Information and knowledge management
Machine learning
Operations research and optimization
Quantum technologies
Robotics
Search and information retrieval
Security, privacy, and abuse prevention
Sustainability
Our scientific contributions
Publications
Research from our scientists and collaborators.
Conferences
Our experts present and discuss cutting-edge research at scientific meetings globally.
News & blog
The latest from Amazon researchers
Amazon Science Blog
Technical deep-dives and perspectives from our scientists.
News
Research milestones and recent achievements.
The latest from Amazon researchers
Amazon Science Blog
Technical deep-dives and perspectives from our scientists.
News
Research milestones and recent achievements.
Collaborations
Amazon Research Awards
Overview
Call for proposals
Latest news
Research stories
Recipients
Amazon Nova AI Challenge
Overview
Rules
FAQs
Teams
Research collaborations
Overview
Carnegie Mellon University
Columbia University
Hampton University
Howard University
IIT Bombay
Johns Hopkins University
Max Planck Society
MIT
Tennessee State University
University of California, Los Angeles
University of Illinois Urbana-Champaign
University of Southern California
University of Texas at Austin
Virginia Tech
University of Washington
Amazon Research Awards
Overview
Call for proposals
Latest news
Research stories
Recipients
Amazon Nova AI Challenge
Overview
Rules
FAQs
Teams
Research collaborations
Overview
Carnegie Mellon University
Columbia University
Hampton University
Howard University
IIT Bombay
Johns Hopkins University
Max Planck Society
MIT
Tennessee State University
University of California, Los Angeles
University of Illinois Urbana-Champaign
University of Southern California
University of Texas at Austin
Virginia Tech
University of Washington
Resources
Code and datasets
Amazon Nova
Try Amazon’s frontier foundation models.
Code and datasets
Amazon Nova
Try Amazon’s frontier foundation models.
Careers
Careers
Explore our open roles.
Amazon Scholars
Faculty research opportunities on industry-scale technical challenges.
Postdoctoral Science Program
Early-career research opportunities alongside experienced industry scientists.
Careers
Explore our open roles.
Amazon Scholars
Faculty research opportunities on industry-scale technical challenges.
Postdoctoral Science Program
Early-career research opportunities alongside experienced industry scientists.
Search
Submit Search
Conversational AI
A new benchmark for evaluating patient-facing health AI agents
PatientAgentBench generates a synthetic patient health record, a realistic clinical vignette, and a patient agent that converses with the AI system under evaluation, to capture what a patient-facing agent actually has to do.
By Korosh Vatanparvar , Ashutosh Joshi
July 29, 2026
6 min read
Share
Share
Copy link
X
Line
QZone
Sina Weibo
分享到微信
x
Key takeaways
As healthcare AI shifts from answering questions to completing tasks on behalf of patients, we need benchmarks that measure whether these systems keep patients safe, accurately reason over their health records, and determine the appropriate level of care across multiturn conversations. PatientAgentBench is a reproducible, clinician-vetted evaluation standard built specifically for patient-facing AI agents in healthcare. It generates a synthetic patient record, a clinical vignette, and a patient agent that converses with the system under evaluation. An LLM-as-a-jury panel scores each conversation against reusable criteria. Benchmarking families of language models, we identified recurring patterns of clinical shortcomings: omitting crisis resources in an emergency, or claiming actions that were never executed. More capable models narrowed these gaps but did not close them. The framework enables evaluating systems at scale to identify clinical safety gaps and target improvements.
Was this answer helpful?
When a virtual patient in a benchmark scenario tells an AI agent they're feeling feverish, or hopeless, the difference between a safe response and an unsafe one often comes down to clinical design — not just medical knowledge. As healthcare AI shifts from answering questions to completing tasks on behalf of patients — scheduling visits, managing prescriptions, triaging symptoms — we need benchmarks that measure what actually matters: whether these systems keep patients safe, follow clinical workflows, and help patients accomplish their goals across realistic, multiturn conversations. Primary care already guards against diagnostic errors, unsafe medication use, and gaps in follow-up; an agent acting on a patient's behalf should be evaluated on how it handles the same risks. Today we're sharing PatientAgentBench, a reproducible, clinician-vetted evaluation standard built specifically for patient-facing AI agents in healthcare. We’re also releasing key findings about the ways in which even capable foundation models fall short, and we show how the findings can guide the design of safer agents.
The three-phase PatientAgentBench pipeline: scenario generation, dual-agent conversation, and LLM-as-a-jury evaluation.
Why existing benchmarks don't measure patient-facing agentic AI
Most healthcare AI benchmarks fall into two camps. Those in the first camp test medical knowledge in a static form, such as answers to medical-exam questions or responses to single-turn or short clinician-facing exchanges. Those in the second test agentic, tool-using agents, but on technical tasks done for providers...
Excerpt shown — open the source for the full document.