Rcp Ndcg

RCP-nDCG@10: A New Approach to Enterprise Retrieval Quality | Cohere Skip to content Embed 5 is here: state-of-the-art retrieval - now available in Pro and Fast tiers. Learn more

Products

Solutions

Resources

Blog

Research

Company

Sign in

Request a demo

Platform North

Enterprise-ready AI for business

Compass

Intelligent search and discovery

Models Command

Generative language models

Transcribe New

Speech recognition model

North Small Translate NEW

Machine translation model

Parse New

Document parsing model

Embed

Semantic representation model

Rerank

Retrieval optimization model

Models Overview

Product Products Overview

Total Cost of AI Ownership

Pricing

Featured Command: High-performance generative AI models for real-world applications

Deploy Model Vault

Dedicated model inference platform

Private Deployments

On-prem or isolated VPCs

Security

Protect your data at every stage

See deployment options

By Industry Financial Services

Public Sector

Technology

Telecommunications

Energy and Utilities

Healthcare and Life Sciences

Manufacturing

Featured Model Vault provides fully-isolated, performant inference with Saas simplicity

Insights Customer Stories

For Developers Developers

Models Overview

Docs

Discord

LLM University

Connect Partners

Events

Webinars

Merch Store

Featured How CoreWeave used Cohere North to transform its customer support in 90 days

Blog

The latest news, launches, and insights

Read more

Introducing Embed 5—A new family of frontier embedding models

Cohere and Aleph Alpha sign agreement to become the first transatlantic sovereign AI solution

Who Gets to Define the Rules for AI?

Research Cohere Labs

Cohere’s ML research lab

Explorations Future(s) of Work

How will AI change the way we work?

Aya Models

Multilingual AI at scale

All Papers

Initiatives Research Scholars

Finding the new generation of ML talent

Open Science Community

Championing global, open science

Catalyst Grants

Supporting impactful ML endeavors

Resources Blog

Hugging Face

Events

Featured The future of work debate has an evidence problem

About

Careers

Newsroom

Sep 30, 2026

7 minute read

RCP-nDCG@10: A more complete way to measure retrieval relevance Our new standard for measuring enterprise retrieval quality, validated against human judgment.

How well does nDCG capture the quality of modern retrieval systems?

nDCG is the established yardstick for measuring how well search systems rank results, and is widely used across benchmarks such as MTEB and BEIR . It is preferred because it reflects both relevance and position, giving more weight to useful results that appear near the top. However, it relies on pre-existing relevance labels, which, if incomplete, mean other relevant results can be missed or scored incorrectly.

To address this, we are introducing Rubric-Calibrated Preferences nDCG@10 (RCP-nDCG@10), an evaluation methodology that grades each retrieved document against the same explicit relevance criteria for every query, using a calibrated AI judge. Designed to better reflect real-world enterprise retrieval quality, RCP-nDCG@10 is the metric against which we have optimized our next generation search models. Here, we explain how it works, what's different from the status quo, and how we validated it against human judgment.

nDCG is only as good as its labels In enterprise search, model performance is typically compared using a portfolio of established offline retrieval metrics. The most common are Recall@k, Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (nDCG@k). Each captures a different aspect of retrieval quality.

Recall@10 measures how much relevant material appears within the top ten results. It is useful when missing information is costly, but it pays no attention to ordering: a relevant result at position one is the same as one at position ten. MRR, by contrast, measures how high the first relevant result appears, but ignores everything that follows. It is therefore most useful when a single correct result is enough, such as FAQ search or simple question answering.

nDCG became widely adopted because it improved on simpler metrics by accounting for both degree of relevance and ranking position. More relevant documents contribute more to the score, and those near the top are weighted more heavily. Over time, this made nDCG a standard across search benchmarks and leaderboards.

Because each metric rewards something different, the three can reach different verdicts on the same results (Figure 1). Figure 1. But nDCG does not judge relevance itself. It compares a model’s results against an existing set of relevance judgments, or qrels , typically created by human assessors reviewing a subset of documents surfaced by earlier retrieval systems. Exhaustively judging a much broader candidate set of query-document pairs would be prohibitively expensive, so this pooling approach is a reasonable practical compromise — and one that is widely called upon by the industry today.

Figure 2 shows this on a real query from ViDoRe v3 , asking for the color code used to mark nitrogen tanks. The answer key grades only two documents, and only one of them appears in the system's top five. The document at rank 3 quotes the answer directly, but because it was never judged, it is not considered relevant. The resulting nDCG@5 of 0.760 reflects just one of the five results. Figure 2. The trade-off is a growing coverage gap: a newer model may retrieve a genuinely relevant document that was never included in the original judgment pool, yet conventional nDCG will not recognize it. As retrieval systems become more capable and explore parts of a corpus that earlier systems did not, benchmarks can increasingly reflect just what previous generations of systems were able to find.

In our own 46-person human annotation study, these results were borne out: Qrels are only modestly better than chance at predicting blind human relevance judgments . As a predictor of a human-assigned grade, the qrel achieves an AUC — a measure of how well a score separates two classes, e.g., relevant vs irrelevant — of just 0.65. As we explore below, this shoots to 0.91 using the LLM's calibrated relevance score. Missed relevance is the largest source of error . Among documents labeled irrelevant by the benchmark, 28% are judged useful by humans. The reverse error is much less common: only 15% of documents labeled relevant fail even the...