Discover Categories Leaderboards Community Agent API FAQ

catalog / Coding / HumanEval

▣DatasetCodingFree

HumanEval

Name: HumanEval
Availability: InStock
Author: ai-supply

OpenAI's 164-problem Python benchmark for evaluating code-generation correctness (pass@k).

@ai-supply

Installs3.6k

↗ Source repository

← More Coding Coding leaderboard →How we grade security →Source ↗

HumanEval

OpenAI's HumanEval benchmark — 164 hand-written Python programming problems, each with a function signature, docstring, and unit tests — used to evaluate the functional correctness of code-generation models (the pass@k metric).

The repository includes the problem dataset and an execution harness that runs generated solutions against the hidden tests.

MIT licensed and tiny; a standard reference dataset for measuring code-synthesis capability.

Rating rank

of 27 in Coding

Install rank

#23

of 27 in Coding

Security score

88/100 · B

review

Security rank

of 27 in Coding

Installs

3.6k

cat avg 157k

This listing vs category average

Installs

this

cat avg

Security (of 100)

this

cat avg

Adoption trend

See the Coding leaderboard →

! Security: Review · 8888/100 · grade Bscanned 14h ago

✓ no compromise signals3 risk-surface · 5/20 OWASP controls flagged

Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.

Data card · high confidence (static)

jsonltxtlicense: detected

12 files

Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not.

OWASP Top 10 for LLM Applications

⚠LLM03Supply Chainhigh

Vulnerable/compromised dependencies, models or archives in the artifact.

•Vulnerable dependencies — 3 known vulnerabilities in: tqdm@4.9.0 (CWE-1395)known CVE · -12 pts

⚠LLM05Improper Output Handlingmedium

Code that pipes model/user output into shell, eval, SQL or paths unsafely.

•Suspicious code patterns — OS command execution · openai-human-eval-6d43fb9/data/example_samples.jsonl (CWE-78)risk surface

•Suspicious code patterns — dynamic code execution · openai-human-eval-6d43fb9/human_eval/execution.py (CWE-95)risk surface

§LLM09MisinformationGovernance

Artifacts designed to produce false/deceptive output.

Detectable only by runtime behavioral evaluation; addressed via responsible-use attestation.

◷LLM10Unbounded ConsumptionRuntime-enforced

Unbounded loops/recursion causing DoS or runaway cost.

Enforced at runtime by the gateway (rate limits + spend caps + size caps); static check flags unbounded loops.

✓LLM01Prompt InjectionPassed

✓LLM02Sensitive Information DisclosurePassed

✓LLM04Data and Model PoisoningPassed

Backdoors/poisoning in training data or serialized models.

Behavioral poisoning needs model execution; static check covers unsafe serialization + dataset skew only.

✓LLM06Excessive AgencyPassed

✓LLM07System Prompt LeakagePassed

✓LLM08Vector and Embedding WeaknessesPassed

PII or plaintext source leakage in embedding/vector exports.

Embedding inversion/poisoning is largely runtime; static check covers PII in vector exports.

OWASP Machine Learning Security Top 10

⚠ML06AI Supply Chainhigh

Compromised PyPI/npm packages, typosquats, unsafe serialized models.

•Vulnerable dependencies — 3 known vulnerabilities in: tqdm@4.9.0 (CWE-1395)known CVE · -12 pts

⚠ML09Output Integritymedium

Middleware tampering with model outputs in transit.

Gateway enforces TLS + response integrity; static check flags output-rewriting code.

•Suspicious code patterns — OS command execution · openai-human-eval-6d43fb9/data/example_samples.jsonl (CWE-78)risk surface

•Suspicious code patterns — dynamic code execution · openai-human-eval-6d43fb9/human_eval/execution.py (CWE-95)risk surface

⚠ML05Model Theftlow

Unlicensed re-distribution / license-incompatible derivatives.

Static check verifies license declaration; extraction throttling is runtime.

•No license signal — no SPDX id or license keyword found · openai-human-eval-6d43fb9/README.mdrisk surface

§ML01Input Manipulation (Adversarial)Governance

Models vulnerable to adversarial perturbations.

Requires runtime robustness evaluation; addressed via publisher robustness attestation.

§ML03Model InversionGovernance

Training data reconstructable from a model's outputs.

Runtime/evaluation property; addressed via model-card data-provenance + DP attestation.

§ML04Membership InferenceGovernance

Determining whether a record was in the training set.

Runtime/evaluation property; addressed via overfitting disclosure + DP attestation.

§ML08Model SkewingGovernance

Models trained on skewed data producing biased output.

Requires fairness evaluation; addressed via model-card bias/limitations disclosure.

✓ML02Data PoisoningPassed

Poisoned training datasets with triggers or anomalous distributions.

Static check covers trigger phrasing, PII and label skew; full poisoning detection is runtime.

✓ML07Transfer Learning AttackPassed

Backdoored base models / LoRA adapters propagating to derivatives.

Backdoor detection needs behavioral probing; static check covers unsafe serialization + provenance.

✓ML10Model Poisoning (Weights)Passed

Tampered model weight files; integrity must be verifiable.

Static check enforces safe formats + records a content hash for downstream verification.

Other findings (2) · hygiene / uncategorized

•Unrecognized file type — '.?' is not on the allowlist · openai-human-eval-6d43fb9/LICENSErisk surface

•Unrecognized file type — '.jsonl' is not on the allowlist · openai-human-eval-6d43fb9/data/example_problem.jsonlrisk surface

✔ verified source · pinned openai-human-eval-6d43fb9

Check against a policy

The same gate an agent runs before installing (POST /api/v1/trust/openai-humaneval/check). Click a policy:

Consume HumanEval programmatically. Authenticate with an API key or session — see Authorize an agent.

# Agents: CHECK BEFORE YOU INSTALL (no auth) — score, grade, level, capability manifest
curl https://ai-supply.store/api/v1/trust/openai-humaneval

# Gate against your org policy (returns { pass, violations })
curl -X POST https://ai-supply.store/api/v1/trust/openai-humaneval/check \
  -H "Content-Type: application/json" \
  -d '{"minGrade":"B","denyPermissions":["shell"],"denyUnknownEgress":true}'

# CLI
npx ai-supply add openai-humaneval

# REST (install → download)
curl -X POST https://ai-supply.store/api/v1/listings/openai-humaneval/install \
  -H "Authorization: Bearer $AIM_KEY"

# MCP tool
install_listing({ "slug": "openai-humaneval" })

OpenAPI spec →

vlatest

! Security: Review · 8815h ago

Curated mirror — latest upstream source. See the repository for tagged releases.

HumanEval

HumanEval

More from @ai-supply