HELM — Holistic Evaluation of Language Models
Stanford CRFM's reproducible, multi-metric benchmark framework for evaluating any foundation model.
HELM — Holistic Evaluation of Language Models
HELM (Holistic Evaluation of Language Models) is an open-source Python framework from Stanford's Center for Research on Foundation Models (CRFM). It evaluates LLMs across 42+ scenarios and 98+ metrics spanning accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency — producing a single transparent leaderboard.
Key features
- Pluggable model adapters: OpenAI, Anthropic, Hugging Face, Cohere, AI21, and self-hosted
- Deterministic run caching for reproducible results
- Aggregated scoring with per-metric breakdowns
- HELM-Lite for quick evaluation on a subset of scenarios
- Published leaderboard at crfm.stanford.edu/helm/
Quick start
pip install crfm-helm
# Run a quick evaluation on GPT-2
helm-run --conf-path run_specs.conf --suite my_suite --max-eval-instances 10
npx ai-supply add helm-holistic-eval
Curated mirror of the open-source HELM (Apache-2.0). Get it from the source.
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
The same gate an agent runs before installing (POST /api/v1/trust/helm-holistic-eval/check). Click a policy:
Consume HELM — Holistic Evaluation of Language Models programmatically. Authenticate with an API key or session — see Authorize an agent.
# Agents: CHECK BEFORE YOU INSTALL (no auth) — score, grade, level, capability manifest
curl https://ai-supply.store/api/v1/trust/helm-holistic-eval
# Gate against your org policy (returns { pass, violations })
curl -X POST https://ai-supply.store/api/v1/trust/helm-holistic-eval/check \
-H "Content-Type: application/json" \
-d '{"minGrade":"B","denyPermissions":["shell"],"denyUnknownEgress":true}'
# CLI
npx ai-supply add helm-holistic-eval
# REST (install → download)
curl -X POST https://ai-supply.store/api/v1/listings/helm-holistic-eval/install \
-H "Authorization: Bearer $AIM_KEY"
# MCP tool
install_listing({ "slug": "helm-holistic-eval" })OpenAPI spec →Curated mirror — latest upstream source. See the repository for tagged releases.