AgentOps — AI Agent Observability & Monitoring SDK
Python SDK for AI agent monitoring, LLM cost tracking, session replay, and benchmarking across all major agent frameworks.
AgentOps
AgentOps gives every AI agent a flight recorder. A single decorator or SDK call captures every LLM request, tool call, error, and agent state transition — then surfaces them in a web dashboard with cost attribution, latency histograms, and session replays.
Key Features
- One-line integration:
agentops.init(api_key)wraps any LLM call automatically - Session replay: step-through timeline of every agent action
- Cost tracking: per-model, per-session, per-run token and dollar spend
- Error detection: captures exceptions, hallucinations, and infinite loops
- Benchmarking: compare agent versions side-by-side
- Native integrations: CrewAI, AutoGen, LangChain, OpenAI Agents SDK, Agno, CamelAI
Quick Start
import agentops
from openai import OpenAI
agentops.init("<AGENTOPS_API_KEY>")
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "What is 2 + 2?"}],
)
print(response.choices[0].message.content)
agentops.end_session("Success")
Install via ai-supply
npx ai-supply add agentops-agent-observability-sdk
Curated mirror of the open-source AgentOps (MIT). Get it from the source.
Compromise signals — malicious or tampered code (leaked secrets, backdoors, a dropped executable) — reduce the score, and known dependency CVEs carry a bounded penalty (they warrant review but never QUARANTINE — update the dependency to clear). Other dangerous-by-capability traits are risk surface, expected for some capabilities. Every finding is mapped to its OWASP control below.
Findings mapped to the OWASP Top 10 for LLM Applications (2025) and the OWASP Machine Learning Security Top 10. Expand any flagged control for the exact findings — compromise reduces the score; expected/risk-surface do not, except a known CVE, which carries a small bounded penalty (high/critical → Review).
The same gate an agent runs before installing (POST /api/v1/trust/agentops-agent-observability-sdk/check). Click a policy:
Consume AgentOps — AI Agent Observability & Monitoring SDK programmatically. Authenticate with an API key or session — see Authorize an agent.
# Agents: CHECK BEFORE YOU INSTALL (no auth) — score, grade, level, capability manifest
curl https://ai-supply.store/api/v1/trust/agentops-agent-observability-sdk
# Gate against your org policy (returns { pass, violations })
curl -X POST https://ai-supply.store/api/v1/trust/agentops-agent-observability-sdk/check \
-H "Content-Type: application/json" \
-d '{"minGrade":"B","denyPermissions":["shell"],"denyUnknownEgress":true}'
# CLI
npx ai-supply add agentops-agent-observability-sdk
# REST (install → download)
curl -X POST https://ai-supply.store/api/v1/listings/agentops-agent-observability-sdk/install \
-H "Authorization: Bearer $AIM_KEY"
# MCP tool
install_listing({ "slug": "agentops-agent-observability-sdk" })OpenAPI spec →Curated mirror — latest upstream source. See the repository for tagged releases.