Community post

A RAGFlow acceptance plan that separates evidence retrieval from answer quality

Build a small evidence-led evaluation set, diagnose each stage, and make unsupported answers visible.

Begin with an answerable question

RAGFlow describes a platform combining retrieval-augmented generation, document processing and agent capabilities. These features provide useful building blocks for applications that answer questions from a document collection. A dependable application still needs an acceptance plan tied to its intended use. The following plan is an editorial proposal for a public product-documentation assistant. It reports no measured accuracy, deployment outcome or comparison between products. Its purpose is to make failures understandable enough that a small team can investigate them.

Write a short service description before collecting documents. Specify the intended readers, the subjects covered, the acceptable age of information and the behavior expected when evidence is missing. For example, an assistant might explain the documented behavior of a particular product release while leaving purchasing recommendations to a human. This boundary makes it possible to identify a correct response without requiring the system to answer every imaginable question about the product.

Create a document register

Give every source a stable identifier and record its title, publisher, version, capture date and access conditions. Keep an original copy or a reliable route back to the source. If two documents describe different releases, preserve that distinction throughout processing. A filename alone may be insufficient because people often rename files or reuse names when publishing revisions. The register should make those changes visible to anyone reviewing an answer later.

Start with a collection small enough for a reviewer to understand. Include an ordinary guide, a document containing tables, an update notice and a page containing an explicit exception. Record why each item belongs in the collection. This selection creates opportunities to test relationships between conditions and claims. Expand the collection after the evaluation process becomes usable, rather than allowing a large import to obscure basic questions about what the application is supposed to know.

Define expected evidence before expected wording

For each evaluation question, identify the passages that support an answer. Record essential facts, relevant qualifications and information that remains unavailable. Several answers may express the same correct meaning, so a single polished reference paragraph should not become the only acceptable wording. Evidence annotations give reviewers a more stable basis for judgment and make it easier to distinguish a stylistic preference from a factual or interpretive error.

Consider a hypothetical question about whether a feature works without additional configuration. One release guide describes the feature, while a later notice introduces a required setting. The expected evidence includes both documents and their relationship. An answer that names the feature while omitting the setting would be incomplete. Also include a question about a feature absent from the collection; its expected outcome should explicitly acknowledge that the available material cannot establish an answer.

Inspect parsing independently

Before running answer evaluations, inspect the material produced from representative source pages. Check reading order, table headers, units, headings and references to exceptions. Preserve page or section information wherever the chosen processing path supports it. If the text entering retrieval already reverses a condition or separates a value from its label, later improvements to generation may conceal the problem without repairing the underlying evidence.

Maintain a small parsing defect register with the source identifier, location, observed result and likely downstream effect. A duplicated footer and a missing negation deserve different priorities. After changing a parser or its settings, revisit the same cases and record improvements and regressions. This procedure is an application-level recommendation; the exact inspection interfaces and available metadata depend on the RAGFlow version and deployment being evaluated.

Evaluate retrieved material before generated prose

For each question, inspect the returned passages without first reading the generated answer. Ask whether the necessary evidence appears, whether its conditions remain intact and whether irrelevant versions dominate the candidate set. Record missing evidence separately from distracting evidence. The two problems can require different changes, and combining them into a single score may hide the reason an answer failed.

Change one retrieval-related factor at a time and keep the questions constant. Possible experiments include document preparation, passage boundaries or the retrieval configuration available in the deployment. Record the exact change and the questions affected. If the correct passage appears among candidates but loses its context, simply returning more text may increase noise. Use observations from the actual candidate passages to decide the next experiment instead of adjusting several settings together.

Judge answers against the retrieved evidence

Once useful evidence is available, review each material claim in the answer. Check whether the response preserved version limits, exceptions and uncertainty. Citation presence and citation relevance should receive separate checks. A link to a real document can still accompany a statement that the document does not support. Ask reviewers to identify the passage that supports each important statement, particularly any statement that could change a reader's next action.

Separate direct facts, reasonable synthesis and editorial advice in the presentation. If the application offers a recommendation, make its assumptions visible. Questions with insufficient evidence should remain in the regular evaluation set, because fluent unsupported answers may otherwise disappear from the reported results. Reviewers should also watch for answers that combine individually true passages into a conclusion that none of the passages actually establishes.

Exercise access and failure boundaries

Use harmless test documents and separate test identities to check the intended access model. Define which documents each identity may retrieve, then inspect search results, answer citations and direct links. Include a permission change in the exercise. A document disappearing from a navigation list does not establish that every retrieval or cached-answer path now respects the change. Any unverified access path should be recorded as an unresolved acceptance item.

Test unavailable model services, empty retrieval results and malformed source material. The user-facing response should explain the operational failure clearly enough to support recovery. A failed retrieval request should not silently turn into an apparently sourced answer based on unrelated knowledge. Keep diagnostic records proportionate: request identifiers, stage outcomes and error categories often help investigation without requiring the full source text to appear in ordinary logs.

Turn acceptance into a maintenance practice

Keep the evaluation questions, evidence annotations, configuration records and reviewer decisions together. Revisit them when documents change, when models change and when processing settings change. Report outcomes by failure category before relying on a combined number. A modest collection of carefully understood cases can guide improvements, while its limited coverage should remain visible. It cannot establish universal reliability for every document or user question.

Release decisions should explain what improved, what remains uncertain and how a problematic change can be reversed. Assign responsibility for updating the document register and retiring obsolete evidence. The useful outcome is an assistant whose answers can be investigated and corrected through a clear chain of material and decisions. That makes the evaluation set an operating tool for the team, rather than a collection of attractive examples saved only for a demonstration.


Primary sources: infiniflow/ragflow official repository ↗. Checked on November 13, 2026.

Editorial note: An original COTCC site-team guide, written with AI assistance and checked against primary sources. Scenarios and acceptance criteria are editorial proposals, not reported deployments or benchmark results. Verify current project documentation for features, licensing and costs.

Discussion 0

Oldest first

No public comments yet. Start the discussion.

Publish