scanned 18d ago · partial·osv · gitleaks · opengrep · picklescan + heuristics·full breakdown in the Security tab ↓
Counterfit — ML Model Security Testing CLI
Counterfit is a Microsoft Azure open-source CLI that provides a generic automation layer for assessing AI/ML model security. It wraps popular attack libraries (ART, TextAttack, Augly) behind a single interface so red teams can probe any model — whether it's a REST endpoint, a local PyTorch model, or a cloud API.
Key features
Unified interface over 20+ attack algorithms (evasion, poisoning, model extraction, inference)
Works against black-box REST endpoints — no source code required
Replay attack logs for reproducible security reports
Out-of-the-box target adapters for image classifiers and NLP models
Built-in logging to Azure Monitor or local JSON
Quick start
pip install counterfit
cf # launch interactive CLI
# Inside the CLI:
list targets
set_target common-cartpole
list attacks
set_attack art-boundary
run
npx ai-supply add counterfit-ml-security-cli
Curated mirror of the open-source Counterfit (MIT). Get it from the source.
! Security: Review · 7575/100 · grade Bscanned 18d ago
✓ no compromise signals13 risk-surface · 6/20 OWASP controls flagged
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
What this capability can do · med confidence (static)
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
OWASP Top 10 for LLM Applications
⚠LLM03Supply Chaincritical
Vulnerable/compromised dependencies, models or archives in the artifact.