Evidently
Open-source ML and LLM observability framework for evaluating, monitoring, and testing AI system quality.
Evidently
Evidently is an open-source Python library for evaluating, testing, and monitoring ML models and LLM-powered applications. It provides 100+ built-in metrics covering data quality, data drift, model performance, and LLM output quality.
Key Features
- LLM evaluation: Assess text quality, hallucination, toxicity, semantic similarity, and custom criteria
- Data drift detection: Statistical tests (KS, PSI, Wasserstein) to detect distribution shifts in features
- Column-level reports: Generate interactive HTML reports for any dataset or prediction batch
- Test suites: Codify quality expectations as pass/fail tests for CI/CD integration
- Monitoring platform: Evidently Cloud or self-hosted for continuous production monitoring
- Integrations: Works with MLflow, Airflow, Prefect, Dagster, and any Python-based pipeline
Quick Start
pip install evidently
from evidently import Dataset, DataDefinition
from evidently.presets import DataDriftPreset
from evidently import Report
report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=reference_df, current_data=current_df)
report.save_html("drift_report.html")
Add to ai-supply
npx ai-supply add evidently-ml-monitoring
Curated mirror of the open-source Evidently (Apache-2.0). Get it from the source.
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
The same gate an agent runs before installing (POST /api/v1/trust/evidently-ml-monitoring/check). Click a policy:
Consume Evidently programmatically. Authenticate with an API key or session — see Authorize an agent.
# Agents: CHECK BEFORE YOU INSTALL (no auth) — score, grade, level, capability manifest
curl https://ai-supply.store/api/v1/trust/evidently-ml-monitoring
# Gate against your org policy (returns { pass, violations })
curl -X POST https://ai-supply.store/api/v1/trust/evidently-ml-monitoring/check \
-H "Content-Type: application/json" \
-d '{"minGrade":"B","denyPermissions":["shell"],"denyUnknownEgress":true}'
# CLI
npx ai-supply add evidently-ml-monitoring
# REST (install → download)
curl -X POST https://ai-supply.store/api/v1/listings/evidently-ml-monitoring/install \
-H "Authorization: Bearer $AIM_KEY"
# MCP tool
install_listing({ "slug": "evidently-ml-monitoring" })OpenAPI spec →Curated mirror — latest upstream source. See the repository for tagged releases.