OpenLLMetry
OpenTelemetry-based observability SDK for LLM applications — traces, metrics, and logs for any AI framework.
OpenLLMetry
OpenLLMetry is an open-source observability SDK for LLM applications built on OpenTelemetry standards. Add one line of code to get automatic traces for LangChain, LlamaIndex, OpenAI, Anthropic, Bedrock, VertexAI, and 20+ other frameworks — without vendor lock-in.
Key Features
- One-line instrumentation: Auto-trace any supported LLM framework without modifying application code
- OpenTelemetry native: Sends standard OTLP traces to any backend (Jaeger, Grafana, Datadog, New Relic, Traceloop)
- Prompt & completion capture: Records full prompts, completions, token counts, and model parameters
- Vector DB tracing: Instruments Chroma, Qdrant, Pinecone, Weaviate, and Milvus calls
- 20+ integrations: LangChain, LlamaIndex, Haystack, CrewAI, AutoGen, Transformers, and more
- Privacy controls: Configurable suppression of prompt/completion content for sensitive environments
Quick Start
pip install opentelemetry-sdk traceloop-sdk
from traceloop.sdk import Traceloop
Traceloop.init(app_name="my-llm-app")
# All subsequent OpenAI / LangChain / etc. calls are now traced automatically
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello!"}]
)
Add to ai-supply
npx ai-supply add openllmetry-llm-observability
Curated mirror of the open-source OpenLLMetry (Apache-2.0). Get it from the source.
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
The same gate an agent runs before installing (POST /api/v1/trust/openllmetry-llm-observability/check). Click a policy:
Consume OpenLLMetry programmatically. Authenticate with an API key or session — see Authorize an agent.
# Agents: CHECK BEFORE YOU INSTALL (no auth) — score, grade, level, capability manifest
curl https://ai-supply.store/api/v1/trust/openllmetry-llm-observability
# Gate against your org policy (returns { pass, violations })
curl -X POST https://ai-supply.store/api/v1/trust/openllmetry-llm-observability/check \
-H "Content-Type: application/json" \
-d '{"minGrade":"B","denyPermissions":["shell"],"denyUnknownEgress":true}'
# CLI
npx ai-supply add openllmetry-llm-observability
# REST (install → download)
curl -X POST https://ai-supply.store/api/v1/listings/openllmetry-llm-observability/install \
-H "Authorization: Bearer $AIM_KEY"
# MCP tool
install_listing({ "slug": "openllmetry-llm-observability" })OpenAPI spec →Curated mirror — latest upstream source. See the repository for tagged releases.