HELM — Holistic Evaluation of Language Models

HELM — Holistic Evaluation of Language Models

HELM (Holistic Evaluation of Language Models) is an open-source Python framework from Stanford's Center for Research on Foundation Models (CRFM). It evaluates LLMs across 42+ scenarios and 98+ metrics spanning accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency — producing a single transparent leaderboard.

Key features

• Pluggable model adapters: OpenAI, Anthropic, Hugging Face, Cohere, AI21, and self-hosted
• Deterministic run caching for reproducible results
• Aggregated scoring with per-metric breakdowns
• HELM-Lite for quick evaluation on a subset of scenarios
• Published leaderboard at crfm.stanford.edu/helm/

Quick start

pip install crfm-helm
# Run a quick evaluation on GPT-2
helm-run --conf-path run_specs.conf --suite my_suite --max-eval-instances 10

npx ai-supply add helm-holistic-eval

Curated mirror of the open-source HELM (Apache-2.0). Get it from the source.