Introducing RTEB: A New Standard for Retrieval Evaluation

TL;DR – We’re excited to introduce the beta version of the Retrieval Embedding Benchmark (RTEB), a new benchmark designed to reliably evaluate the retrieval accuracy of embedding models for real-world applications. Existing benchmarks struggle to measure true generalization, while RTEB addresses this with a hybrid strategy of open and private datasets. Its goal is simple: to create a fair, transparent, and application-focused standard for measuring how models perform on data they haven’t seen before.

The performance of many AI applications, from RAG and agents to recommendation systems, is fundamentally limited by the quality of search and retrieval. As such, accurately measuring the retrieval quality of embedding models is a common pain point for developers. How do you really know how well a model will perform in the wild?

This is where things get tricky. The current standard for evaluation often relies on a model's "zero-shot" performance on public benchmarks. However, this is, at best, an approximation of a model's true generalization capabilities. When models are repeatedly evaluated against the same public datasets, a gap emerges between their reported scores and their actual performance on new, unseen data.

To address these challenges, we developed RTEB, a benchmark built to provide a reliable standard for evaluating retrieval models.

The Challenges of Current Benchmarks

While the underlying evaluation methodology and metrics (such as NDCG@10) are well-known and robust, the integrity of existing benchmarks is often set back by the following issues:

The Generalization Gap. The current benchmark ecosystem inadvertently encourages "teaching to the test." When training data sources overlap with evaluation datasets, a model's score can become inflated, undermining a benchmark's integrity. This creates a feedback loop where models are rewarded for memorizing test data rather than developing robust, generalizable capabilities.

Misalignment with Today’s AI Applications. Many benchmarks are poorly aligned with the enterprise use cases that developers are building today. They often rely on academic datasets or on retrieval tasks derived from QA datasets, which were not designed to evaluate retrieval and can fail to capture the complexities encountered in real-world scenarios.

Introducing RTEB

To combat benchmark overfitting, RTEB implements a hybrid strategy using both open and private datasets:

• Open Datasets: The corpus, queries, and relevance labels are fully public. This ensures transparency and allows any user to reproduce the results.
• Private Datasets: These datasets are kept private, and evaluation is handled by the MTEB maintainers to ensure impartiality. This setup provides a clear, unbiased measure of a model’s ability to generalize to unseen data.

Designed for Enterprise Use Cases

• Multilingual in Nature: The benchmark datasets cover 20 languages, from common ones like English to rarer languages such as Bengali or Finnish.
• Domain-Specific Focus: The benchmark includes datasets from critical enterprise domains like law, healthcare, code, and finance.
• Efficient Dataset Sizes: Datasets are large enough to be meaningful without being so large that they make evaluation time-consuming.
• Retrieval-First Metric: The default leaderboard metric is NDCG@10, a gold-standard measure for the quality of ranked search results.