What is Promptfoo? A Practical Guide to Testing and Evaluating AI Prompts

What is Promptfoo?

Promptfoo is an open-source framework for evaluating and testing LLM applications, especially prompts, so you can compare outputs, catch regressions, and improve quality in a repeatable way

With Promptfoo you can:

• run multiple prompt variants against the same inputs
• compare outputs across models side by side
• score responses automatically with deterministic checks and model-graded rubrics
• speed up evaluation with caching and concurrency
• use it from CLI and in CI/CD workflows

Think of it as unit testing for prompts, with assertions that can be both rule-based and LLM-assisted

Why Prompt Testing Matters

Even a small prompt change or model upgrade can change outputs in ways that are hard to predict by intuition

Without testing:

• regressions can slip in unnoticed
• behavior becomes inconsistent across prompts, inputs, and models
• quality is difficult to measure objectively

With Promptfoo:

• experiments are repeatable
• quality is measurable
• prompt changes are safer to ship

How Promptfoo Works

Promptfoo evaluations usually rely on three building blocks: prompts, test cases, and assertions or metrics

1. Prompts

Prompt A: Explain this like I'm five
Prompt B: Provide a concise technical explanation

2. Test Cases

Input: "What is blockchain?"
Input: "Explain photosynthesis"

3. Assertions and metrics

Examples include:

• deterministic checks such as contains and equals
• weighted scoring and thresholds
• model-graded evaluation with llm-rubric

Installation

A fast way to start is an example template:

npx promptfoo@latest init --example getting-started

You can also install globally:

npm install -g promptfoo

Or use Homebrew:

brew install promptfoo

Most providers require credentials. For OpenAI:

export OPENAI_API_KEY=sk-...

Basic Configuration

A typical setup lives in promptfooconfig.yaml:

prompts:
  - "Explain {{topic}} in simple terms."
  - "Provide a concise technical explanation of {{topic}}."

providers:
  - openai:gpt-4.1-mini

tests:
  - vars:
      topic: "quantum computing"
    assert:
      - type: contains
        value: "quantum"
      - type: llm-rubric
        value: "Is the explanation accurate and appropriate for the requested style?"
        provider: openai:gpt-4.1-mini
        threshold: 0.7

  - vars:
      topic: "machine learning"
    assert:
      - type: llm-rubric
        value: "Does the answer avoid hallucinations and keep a clear structure?"
        provider: openai:gpt-4.1-mini

You can also load tests from YAML, JSON, JSONL, CSV, TypeScript, JavaScript, and external datasets such as Google Sheets

Viewing Results

After an evaluation, open interactive browser results:

npx promptfoo@latest view

If you prefer guided setup:

npx promptfoo@latest eval setup

Useful additions in practice:

• validate configuration with promptfoo validate
• treat custom assertions and plugins like local Node.js code with full local permissions

Conclusion

Prompt engineering without testing is mostly guesswork

Promptfoo turns it into a structured, repeatable, and measurable process