TruthfulQA
Benchmark of 817 questions across 38 categories measuring whether LLMs avoid imitative falsehoods — a direct probe of factual hallucination.
TruthfulQA — benchmark for truthfulness & imitative falsehoods
TruthfulQA measures whether a language model gives truthful answers to questions that humans often answer falsely because of common misconceptions — a direct probe of imitative hallucination rather than raw knowledge.
Key features
- 817 questions across 38 categories (health, law, finance, politics, conspiracies, and more) crafted to elicit false beliefs
- Two evaluation tracks: generation (free-form) and multiple-choice (MC1 / MC2)
- Truthfulness and informativeness scoring via the "GPT-judge" protocol and BLEURT/reference metrics
- Curated reference true/false answer sets for automated grading
- A widely cited yardstick for catching factuality regressions when fine-tuning or prompting
Because the questions are adversarially chosen to trap models that parrot popular falsehoods, TruthfulQA is a sharp regression test for hallucination — ideal for gating a model release on measured truthfulness.
Curated mirror of the open-source TruthfulQA (Apache-2.0). Get it from the source.
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
The same gate an agent runs before installing (POST /api/v1/trust/truthfulqa-hallucination-benchmark/check). Click a policy:
Consume TruthfulQA programmatically. Authenticate with an API key or session — see Authorize an agent.
# Agents: CHECK BEFORE YOU INSTALL (no auth) — score, grade, level, capability manifest
curl https://ai-supply.store/api/v1/trust/truthfulqa-hallucination-benchmark
# Gate against your org policy (returns { pass, violations })
curl -X POST https://ai-supply.store/api/v1/trust/truthfulqa-hallucination-benchmark/check \
-H "Content-Type: application/json" \
-d '{"minGrade":"B","denyPermissions":["shell"],"denyUnknownEgress":true}'
# CLI
npx ai-supply add truthfulqa-hallucination-benchmark
# REST (install → download)
curl -X POST https://ai-supply.store/api/v1/listings/truthfulqa-hallucination-benchmark/install \
-H "Authorization: Bearer $AIM_KEY"
# MCP tool
install_listing({ "slug": "truthfulqa-hallucination-benchmark" })OpenAPI spec →Curated mirror — latest upstream source. See the repository for tagged releases.