promptfoo

promptfoo is a developer-first LLM evaluation and red-teaming framework. Write test cases with assertions, run them against any model or prompt variant, compare results side-by-side, and integrate automated evals into your CI pipeline — before bad outputs reach production.

Key features

• Declarative test cases — define expected outputs with string matchers, regex, JSON schema, semantic similarity, or LLM-as-judge
• Multi-provider comparison — benchmark the same prompts across OpenAI, Anthropic, Groq, Mistral, and local models simultaneously
• Red-teaming — automated adversarial probing for jailbreaks, prompt injection, PII leakage, and harmful content
• CI integration — promptfoo eval exits non-zero on failures; ships a GitHub Actions example
• Web UI — browser-based results viewer and diff tool
• Caching — reuse LLM responses to speed up iterative prompt development

Quick start

npx ai-supply add promptfoo-llm-eval

# Or install directly
npm install -g promptfoo

# Initialize a config
promptfoo init

# promptfooconfig.yaml
prompts:
  - "Summarize the following: {{text}}"

providers:
  - openai:gpt-4o
  - anthropic:claude-opus-4-5

tests:
  - vars:
      text: "The quick brown fox jumps over the lazy dog."
    assert:
      - type: contains
        value: fox
      - type: llm-rubric
        value: "The summary is concise and accurate."

promptfoo eval
promptfoo view

Curated mirror of the open-source promptfoo project (MIT). Install upstream from the repository.