How security scanning works (and why it matters)
How security scanning works (and why it matters)
AI capabilities run with real permissions inside real systems. A compromised MCP server can exfiltrate data; a malicious prompt can jailbreak a production agent. ai-supply.store scans every artifact on every version before a listing becomes installable.
What gets scanned
The pipeline runs eight distinct checks in parallel:
| Check | What it catches |
|---|---|
| Malware | Known malicious payloads, shellcode, obfuscated scripts |
| Secrets | API keys, tokens, private keys, credentials hardcoded in artifacts |
| Dangerous code | Shell injections, eval() abuse, network backdoors |
| PII | Names, email addresses, phone numbers, SSNs in datasets or prompts |
| License | GPL/AGPL contamination in commercial or permissive-only listings |
| Dependency CVEs | Known vulnerabilities in package.json, requirements.txt, etc. |
| Model format | Pickle exploits, malformed GGUF/safetensors, hidden execution layers |
| Prompt injection | Instructions designed to hijack downstream agent behaviour |
The OWASP-AI checklist
On top of the automated pipeline, each listing is evaluated against the OWASP AI Security Top 10 (LLM01–LLM10) and the OWASP ML Top 10 (ML01–ML10). These cover risks such as:
- LLM01: Prompt injection
- LLM02: Insecure output handling
- LLM06: Sensitive information disclosure
- ML03: Model inversion attack surface
- ML07: Transfer learning attack
For a full breakdown of the report format, see reading a listing's OWASP-AI security report.
The score, grade, and safety level
After scanning, every listing receives:
- Score — 0 (worst) to 100 (best)
- Grade — A, B, C, or D
- Safety level — one of three outcomes:
SAFE → installable immediately
REVIEW → installable only after the buyer explicitly acknowledges the risks
QUARANTINE → blocked; listing is not visible or installable
A quarantined listing cannot be acknowledged away — the provider must fix the issue and submit a new version.
The "Most secure" leaderboard
Listings with grade A and score ≥ 90 appear on the Most secure leaderboard, visible on the benchmarks page. This is a meaningful trust signal for enterprise buyers.
Re-scanning on new versions
Every new version upload triggers a fresh scan. If a dependency introduces a new CVE after initial publication, the provider should re-upload — the score will update automatically.
What providers should do
- Never hardcode secrets — use environment variable references instead.
- Pin dependency versions and audit them with
npm audit/pip-auditbefore upload. - Strip PII from datasets and prompts before submitting.
- Test for prompt injection by red-teaming your own prompts.
- Use safetensors format for model weights instead of pickle.
Following these practices almost guarantees a grade A on first submission. For MCP-specific advice, see writing a safe MCP server for the registry.