Building a promptfoo regression set that supports an actual release decision
Connect representative tasks, evidence checks and human review to the consequences of a model or prompt change.
Decide what a successful application does
promptfoo provides tools for evaluating language-model applications, including comparisons involving prompts and models. These capabilities become more useful when the test set reflects an application's real responsibilities. This article proposes a regression process for a hypothetical assistant that summarizes public product announcements. It supplies an editorial evaluation design, without claiming measured improvements, a completed red-team assessment or a particular level of reliability for any model or application.
Write an application contract before collecting examples. Specify the permitted sources, the required output, the audience and the mistakes that would cause the most harm. A summary may need to preserve dates and conditions while explicitly marking unavailable information. Another application may prioritize a strict machine-readable structure. Define these requirements separately so that a release decision is grounded in the service people need rather than a general preference for fluent text.
Collect cases that represent meaningful differences
Begin with examples from the intended task, using public or appropriately authorized material. Include clear announcements, long inputs, revised statements and requests whose answer is absent from the source. Record why each case belongs in the set. A small collection can still be informative when its cases cover distinct failure mechanisms and reviewers understand what it leaves untested.
Avoid building the entire set from failures discovered during one recent debugging session. Those cases matter, but a narrow collection can overemphasize one problem while forgetting ordinary work. Keep categories for routine tasks, known regressions, difficult boundaries and unsupported requests. If examples come from different user groups or languages, make that coverage visible. A result from one group should not quietly become a conclusion about all intended users.
Write expectations before viewing candidate outputs
For every case, list required facts, relevant qualifications, acceptable uncertainty and prohibited unsupported additions. When the source permits several correct expressions, record the meaning that must be preserved rather than only one ideal sentence. Keep expected evidence with the case so a reviewer can distinguish factual disagreement from a preference about wording. This also helps future maintainers revise a test when the source itself changes.
Consider a hypothetical announcement stating that a feature is available only to a limited group. A response naming the feature and omitting that condition should receive a specific failure label. Add a paired case asking about availability beyond the announced scope; the expected response should explain that the source does not establish it. Such pairs reveal whether a prompt preserves boundaries instead of rewarding an answer merely because it mentions the right product name.
Separate structure, evidence and usefulness
Different evaluation checks answer different questions. Structural checks can establish whether required fields exist or whether an output follows an agreed format. Evidence checks ask whether claims are supported and conditions are preserved. Human usefulness includes clarity, relevance and whether the result helps the intended reader complete a task. Keep these categories visible instead of assuming a response that passes one category has passed all the others.
Where an automated assertion is appropriate, explain what it measures and what it cannot establish. A required phrase might be present in an otherwise misleading answer. A correctly shaped object might contain invented facts. Choose checks that correspond to actual requirements, then review representative failures and successes against the source. The particular configuration should follow the current promptfoo documentation and the application's chosen providers and operating environment.
Treat model-based grading as a component to evaluate
If another model grades answers, specify its rubric and inspect how it applies that rubric. Include examples with polished unsupported statements and less elegant but faithful answers. Ask whether the grader rewards confidence, length or stylistic similarity at the expense of evidence. A grading model's output is another judgment in the evaluation system, so preserve enough information to investigate disagreements with human reviewers.
Use a small calibration exercise before relying heavily on automated judgments. Have reviewers assess the same examples, compare reasons and clarify ambiguous criteria. Keep a record of unresolved disagreements rather than forcing every case into a simple consensus score. This procedure is an editorial recommendation, not a claim that it eliminates subjectivity. Its value is making the source of a release decision more visible and easier to challenge constructively.
Compare changes under recorded conditions
Record the prompt version, model identity, relevant generation settings, source revision and evaluation configuration for each run. Change one important factor at a time when the purpose is to understand its effect. If a release requires several changes together, describe the combined change honestly and avoid attributing its outcome to one component without additional evidence. Preserve individual outputs so a summary score can be traced to concrete cases.
Generated responses may vary, so choose an observation strategy that reflects the application and available budget. Do not imply that one run establishes stable behavior under every condition. Record any repeated runs and how their results were combined. Include actual usage and timing measurements only when collected through a documented method, and keep provider pricing assumptions separate from observed consumption. This prevents convenient estimates from becoming unsupported operating claims.
Define release gates around consequences
Before examining the candidate's results, write down what would block release. A critical invented fact or access failure may deserve a different response from a minor formatting issue. Define any thresholds as application decisions, with reasons tied to the cost of error. A combined average can help summarize progress, while reviewers still need to inspect the categories and cases that carry the most significant consequences.
Prepare a decision note explaining improvements, regressions, unresolved cases and accepted limitations. If a candidate improves ordinary summaries while worsening unsupported questions, make that tradeoff explicit. The team may decide to revise the change, add a review step or restrict its scope. These are product and operating judgments informed by evaluation evidence; the test framework does not make the responsibility for them disappear.
Keep the set useful after release
Add newly observed failures with their source context and explanation. Retire or update examples when documents change, while preserving enough history to understand earlier decisions. Keep some evaluation material outside the immediate prompt-tuning loop so that repeated editing does not simply optimize for memorized examples. Describe that separation in the team's process rather than assuming the presence of a test file automatically creates independent validation.
The final evaluation package should let another reviewer reconstruct what changed, what was tested and why the release was accepted. Include the application contract, cases, evidence, configuration records and decision note. A useful regression set then becomes part of everyday maintenance: it makes tradeoffs visible, preserves lessons from mistakes and gives future changes a consistent point of comparison without overstating the coverage of a limited collection of examples.
Primary sources: promptfoo/promptfoo official repository ↗. Checked on November 13, 2026.
Editorial note: An original COTCC site-team guide, written with AI assistance and checked against primary sources. Scenarios and acceptance criteria are editorial proposals, not reported deployments or benchmark results. Verify current project documentation for features, licensing and costs.
Discussion 0
Oldest firstSign in to join the discussion.
No public comments yet. Start the discussion.