catalog / Coding / HumanEval

▣DatasetCodingFree

HumanEval

Name: HumanEval
Availability: InStock
Author: ai-supply

OpenAI's 164-problem Python benchmark for evaluating code-generation correctness (pass@k).

@ai-supply

安装量3.6k

⟳ upstream master@6d43fb9 · updated 1y ago

↗ 源代码仓库

← More Coding Coding leaderboard →How we grade security →Source ↗

! Grade B · 88/100 · ReviewSecurity assessment

✓No compromise signals2capabilities surfaced1known CVE9of 20 OWASP controls clear

Suspicious code patternsSuspicious code patternsVulnerable dependencies

scanned 1mo agoosv · gitleaks · opengrep · picklescan + heuristicsfull breakdown in the Security tab ↓

HumanEval

OpenAI's HumanEval benchmark — 164 hand-written Python programming problems, each with a function signature, docstring, and unit tests — used to evaluate the functional correctness of code-generation models (the pass@k metric).

The repository includes the problem dataset and an execution harness that runs generated solutions against the hidden tests.

MIT licensed and tiny; a standard reference dataset for measuring code-synthesis capability.

! Security: Review · 8888/100 · grade Bscanned 1mo ago

✓ no compromise signals3 risk-surface · 5/20 OWASP controls flagged

Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.

Data card · high confidence (static)

jsonltxtlicense: detected

12 files

Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).

OWASP Top 10 for LLM Applications

⚠LLM03Supply Chainhigh

Vulnerable/compromised dependencies, models or archives in the artifact.

•Vulnerable dependencies — 4 known vulnerabilities in: tqdm@4.9.0 (CWE-1395)known CVE · -12 pts

⚠LLM05Improper Output Handlingmedium

Code that pipes model/user output into shell, eval, SQL or paths unsafely.

•Suspicious code patterns — OS command execution · openai-human-eval-6d43fb9/data/example_samples.jsonl (CWE-78)risk surface

•Suspicious code patterns — dynamic code execution · openai-human-eval-6d43fb9/human_eval/execution.py (CWE-95)risk surface

§LLM09MisinformationGovernance

Artifacts designed to produce false/deceptive output.

Detectable only by runtime behavioral evaluation; addressed via responsible-use attestation.

◷LLM10Unbounded ConsumptionRuntime-enforced

Unbounded loops/recursion causing DoS or runaway cost.

Enforced at runtime by the gateway (rate limits + spend caps + size caps); static check flags unbounded loops.

✓LLM01Prompt InjectionPassed

✓LLM02Sensitive Information DisclosurePassed

✓LLM04Data and Model PoisoningPassed

Backdoors/poisoning in training data or serialized models.

Behavioral poisoning needs model execution; static check covers unsafe serialization + dataset skew only.

✓LLM06Excessive AgencyPassed

✓LLM07System Prompt LeakagePassed

✓LLM08Vector and Embedding WeaknessesPassed

PII or plaintext source leakage in embedding/vector exports.

Embedding inversion/poisoning is largely runtime; static check covers PII in vector exports.

OWASP Machine Learning Security Top 10

⚠ML06AI Supply Chainhigh

Compromised PyPI/npm packages, typosquats, unsafe serialized models.

•Vulnerable dependencies — 4 known vulnerabilities in: tqdm@4.9.0 (CWE-1395)known CVE · -12 pts

⚠ML09Output Integritymedium

Middleware tampering with model outputs in transit.

Gateway enforces TLS + response integrity; static check flags output-rewriting code.

•Suspicious code patterns — OS command execution · openai-human-eval-6d43fb9/data/example_samples.jsonl (CWE-78)risk surface

•Suspicious code patterns — dynamic code execution · openai-human-eval-6d43fb9/human_eval/execution.py (CWE-95)risk surface

⚠ML05Model Theftlow

Unlicensed re-distribution / license-incompatible derivatives.

Static check verifies license declaration; extraction throttling is runtime.

•No license signal — no SPDX id or license keyword found · openai-human-eval-6d43fb9/README.mdrisk surface

§ML01Input Manipulation (Adversarial)Governance

Models vulnerable to adversarial perturbations.

Requires runtime robustness evaluation; addressed via publisher robustness attestation.

§ML03Model InversionGovernance

Training data reconstructable from a model's outputs.

Runtime/evaluation property; addressed via model-card data-provenance + DP attestation.

§ML04Membership InferenceGovernance

Determining whether a record was in the training set.

Runtime/evaluation property; addressed via overfitting disclosure + DP attestation.

§ML08Model SkewingGovernance

Models trained on skewed data producing biased output.

Requires fairness evaluation; addressed via model-card bias/limitations disclosure.

✓ML02Data PoisoningPassed

Poisoned training datasets with triggers or anomalous distributions.

Static check covers trigger phrasing, PII and label skew; full poisoning detection is runtime.

✓ML07Transfer Learning AttackPassed

Backdoored base models / LoRA adapters propagating to derivatives.

Backdoor detection needs behavioral probing; static check covers unsafe serialization + provenance.

✓ML10Model Poisoning (Weights)Passed

Tampered model weight files; integrity must be verifiable.

Static check enforces safe formats + records a content hash for downstream verification.

Other findings (2) · hygiene / uncategorized

•Unrecognized file type — '.?' is not on the allowlist · openai-human-eval-6d43fb9/LICENSErisk surface

•Unrecognized file type — '.jsonl' is not on the allowlist · openai-human-eval-6d43fb9/data/example_problem.jsonlrisk surface

✔ verified source · pinned openai-human-eval-6d43fb9

Check against a policy

The same gate an agent runs before installing (POST /api/v1/trust/openai-humaneval/check). Click a policy:

Consume HumanEval programmatically. Authenticate with an API key or session — see Authorize an agent.

# Agents: CHECK BEFORE YOU INSTALL (no auth) — score, grade, level, capability manifest
curl https://ai-supply.store/api/v1/trust/openai-humaneval

# Gate against your org policy (returns { pass, violations })
curl -X POST https://ai-supply.store/api/v1/trust/openai-humaneval/check \
  -H "Content-Type: application/json" \
  -d '{"minGrade":"B","denyPermissions":["shell"],"denyUnknownEgress":true}'

# CLI
npx ai-supply add openai-humaneval

# REST (install → download)
curl -X POST https://ai-supply.store/api/v1/listings/openai-humaneval/install \
  -H "Authorization: Bearer $AIM_KEY"

# MCP tool
install_listing({ "slug": "openai-humaneval" })

OpenAPI spec →