LangKit
Open-source toolkit that extracts safety and quality signals — injection, PII, toxicity, sentiment, relevance — from LLM prompts and responses.
LangKit — safety & quality metrics for LLM prompts and responses
LangKit is an open-source text-metrics toolkit that extracts safety, quality, and relevance signals from LLM prompts and responses, so you can monitor and guardrail models in production rather than trusting them blind.
Key features
- Prompt-injection and jailbreak similarity scoring against known-attack themes
- PII pattern detection, toxicity, and sentiment analysis on both inputs and outputs
- Text-quality/readability metrics and prompt-response relevance via semantic similarity
- Consistency and refusal signals to surface likely hallucinations or off-policy replies
- Emits metrics compatible with whylogs for drift monitoring, dashboards, and alerting
Unlike a single classifier, LangKit produces a bundle of interpretable signals you can threshold and combine into your own guardrail policy, making it a practical observability layer for LLM safety and security.
Curated mirror of the open-source LangKit (Apache-2.0). Get it from the source.
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
The same gate an agent runs before installing (POST /api/v1/trust/langkit-llm-safety-metrics/check). Click a policy:
Consume LangKit programmatically. Authenticate with an API key or session — see Authorize an agent.
# Agents: CHECK BEFORE YOU INSTALL (no auth) — score, grade, level, capability manifest
curl https://ai-supply.store/api/v1/trust/langkit-llm-safety-metrics
# Gate against your org policy (returns { pass, violations })
curl -X POST https://ai-supply.store/api/v1/trust/langkit-llm-safety-metrics/check \
-H "Content-Type: application/json" \
-d '{"minGrade":"B","denyPermissions":["shell"],"denyUnknownEgress":true}'
# CLI
npx ai-supply add langkit-llm-safety-metrics
# REST (install → download)
curl -X POST https://ai-supply.store/api/v1/listings/langkit-llm-safety-metrics/install \
-H "Authorization: Bearer $AIM_KEY"
# MCP tool
install_listing({ "slug": "langkit-llm-safety-metrics" })OpenAPI spec →Curated mirror — latest upstream source. See the repository for tagged releases.