Did your prompt improve? Build a small regression set with promptfoo
Compare prompt changes with repeatable checks and a little human review.
promptfoo provides tools for evaluating language-model applications, prompts and model outputs. A fixed set of examples can make comparisons more informative than repeatedly trying one favorite question.
A starter evaluation Collect a small set of representative tasks, including ordinary inputs, edge cases and questions without enough evidence. Define the expected behavior for each: required fields, faithful citations or an explicit statement of uncertainty.
Editorial recommendation Compare two prompt versions on the same inputs. Track format compliance separately from factual quality, then manually inspect failures. If another model grades the answers, check whether its criteria reward confidence or verbosity over correctness.
A narrow evaluation set cannot establish reliability across every real task. Also account for inference costs. Any security testing should stay within systems you are authorized to assess.
Which regression would hurt your application most: invented facts, unstable formatting or unsupported answers?
Primary source promptfoo/promptfoo official repository ↗ — checked on November 11, 2026.
An AI-assisted editorial guide from the site team. Exercises are suggestions, not reported benchmarks. Check current project documentation for features, licenses and pricing.
Discussion 0
Oldest firstSign in to join the discussion.
No public comments yet. Start the discussion.