Open NeurIPS benchmark for jailbreaking LLMs: balanced harmful/benign behaviors, reproducible attack artifacts, standardized judges, and a live leaderboard.
scanned 1mo ago·osv · gitleaks · opengrep · picklescan + heuristics·full breakdown in the Security tab ↓
JailbreakBench — open robustness benchmark for jailbreaking LLMs
JailbreakBench is an open benchmark (NeurIPS 2024 Datasets & Benchmarks Track) for evaluating how susceptible language models are to jailbreak attacks and how well defenses hold up under a shared threat model.
Key features
JBB-Behaviors dataset of 100 harmful and 100 benign behaviors for balanced, over-refusal-aware testing
A repository of adversarial jailbreak artifacts you can reproduce and compare against
Standardized threat model plus an LLM/classifier judge for scoring attack success
Public leaderboard tracking attack and defense submissions over time
Pip-installable harness for plugging in your own attacks, defenses, or target models
Because it fixes the behaviors, judge, and threat model, JailbreakBench makes jailbreak results reproducible and comparable across papers and vendors — exactly what a security-vetted catalog needs to trust a robustness claim.
Curated mirror of the open-source JailbreakBench (MIT). Get it from the source.
! Security: Review · 7575/100 · grade Bscanned 1mo ago
✓ no compromise signals6 risk-surface · 5/20 OWASP controls flagged
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
OWASP Top 10 for LLM Applications
⚠LLM03Supply Chaincritical
Vulnerable/compromised dependencies, models or archives in the artifact.