Circle A Framework For Evaluating Ai From A Real World Lens 2026 03 03

CIRCLE: A Framework for Evaluating AI from a Real-World Lens North Mini Code. Cohere's first model for developers. Learn more

Mar 03, 2026 CIRCLE: A Framework for Evaluating AI from a Real-World Lens CIRCLE is a six-stage framework designed to bridge the gap between AI model performance and real-world outcomes by translating stakeholder concerns into measurable, context-sensitive metrics through coordinated methods like field testing and red teaming.

Read the paper

Authors

Reva Schwartz, Carina Westling, Morgan Briggs, Marzieh Fadaee, Isar Nejadgholi, Matthew Holmes, Fariza Rashid, Maya Carlyle, Afaf Taïk, Kyra Wilson, Peter Douglas, Theodora Skeadas, Gabriella Waters, Rumman Chowdhury, Thiago Lacerda

Abstract

This paper proposes CIRCLE, a six-stage, lifecycle-based framework to bridge the reality gap between model-centric performance metrics and AI's materialized outcomes in deployment. While existing frameworks like MLOps focus on system stability and benchmarks measure abstract capabilities, decision-makers outside the AI stack lack systematic evidence about the behavior of AI technologies under real-world user variability and constraints. CIRCLE operationalizes the Validation phase of TEVV (Test, Evaluation, Verification, and Validation) by formalizing the translation of stakeholder concerns outside the stack into measurable signals. Unlike participatory design, which often remains localized, or algorithmic audits, which are often retrospective, CIRCLE provides a structured, prospective protocol for linking context-sensitive qualitative insights to scalable quantitative metrics. By integrating methods such as field testing, red teaming, and longitudinal studies into a coordinated pipeline, CIRCLE produces systematic knowledge: evidence that is comparable across sites yet sensitive to local context. This can enable governance based on materialized downstream effects rather than theoretical capabilities.

Evaluation Frameworks Policy Optimization

Related works

Research Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects

Read

Research RewardBench 2: Advancing Reward Model Evaluation

Read

Research Findings of the WMT25 Multilingual Instruction Shared Task: Persistent Hurdles in Reasoning, Generation, and Evaluation

Read