CaseHOLD — Legal Holdings Benchmark Dataset

CaseHOLD — Legal Holdings Benchmark Dataset

CaseHOLD is an expert-curated dataset of 53,000+ multiple-choice questions derived from US case law. Each question asks a model to identify the correct legal holding of a cited case from five candidate holdings, testing genuine legal reasoning rather than surface pattern matching.

Key features

• 53,137 multiple-choice questions from federal court opinions (2010–2019)
• Five candidate holdings per question — designed to require legal inference
• Balanced across federal circuit courts (1st–11th plus DC)
• Hosted on Hugging Face Datasets for one-line download
• Reference fine-tuned models: custom-legalbert, legalbert-large available

Quick start

pip install datasets transformers

from datasets import load_dataset

ds = load_dataset("casehold/casehold", "all")
print(ds["train"][0])  # {citing_prompt, holding_0..4, label}

# Fine-tune your own legal LLM
from transformers import AutoTokenizer, AutoModelForMultipleChoice
tokenizer = AutoTokenizer.from_pretrained("nlpaueb/legal-bert-base-uncased")

npx ai-supply add casehold-legal-benchmark

Curated mirror of the open-source CaseHOLD (Apache-2.0). Get it from the source.