BEIR
Heterogeneous zero-shot information-retrieval benchmark bundling 15+ diverse IR datasets behind one evaluation API.
BEIR
BEIR (Benchmarking IR) is the standard for measuring how well a retriever generalizes zero-shot across domains it was never tuned on. Instead of overfitting to a single collection, it aggregates 15+ heterogeneous datasets — spanning fact-checking, question answering, bio-medical, scientific, financial, duplicate-detection, and news retrieval — into a common format with unified corpus/queries/qrels loaders and evaluation.
Key features
- 15+ ready-to-use retrieval datasets in one consistent schema
- Standardized nDCG@k, MAP, Recall, and Precision evaluation out of the box
- Compare BM25, dense bi-encoders, ColBERT, rerankers, and hybrid systems apples-to-apples
- Focus on zero-shot generalization, exposing where dense models quietly underperform lexical baselines
- Widely cited reference used to report embedding and retriever quality
Use it to sanity-check a new embedding model or reranker before shipping it into a RAG stack, so you know it holds up beyond your own domain.
Curated mirror of the open-source BEIR (Apache-2.0). Get it from the source.
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
The same gate an agent runs before installing (POST /api/v1/trust/beir-ir-benchmark/check). Click a policy:
Consume BEIR programmatically. Authenticate with an API key or session — see Authorize an agent.
# Agents: CHECK BEFORE YOU INSTALL (no auth) — score, grade, level, capability manifest
curl https://ai-supply.store/api/v1/trust/beir-ir-benchmark
# Gate against your org policy (returns { pass, violations })
curl -X POST https://ai-supply.store/api/v1/trust/beir-ir-benchmark/check \
-H "Content-Type: application/json" \
-d '{"minGrade":"B","denyPermissions":["shell"],"denyUnknownEgress":true}'
# CLI
npx ai-supply add beir-ir-benchmark
# REST (install → download)
curl -X POST https://ai-supply.store/api/v1/listings/beir-ir-benchmark/install \
-H "Authorization: Bearer $AIM_KEY"
# MCP tool
install_listing({ "slug": "beir-ir-benchmark" })OpenAPI spec →Curated mirror — latest upstream source. See the repository for tagged releases.