Community post

Dify Document Q&A Layered Error Diagnosis: Separating Parsing, Retrieval, and Generation

Use a clear set of small questions with evidence to identify errors in the knowledge base, avoiding repetitive attempts on prompt words.

Reconstructing from wrong answers often leads to missing upstream issues

When the knowledge base application provides an incorrect answer, people usually modify the prompt or switch to a different model. However, the response is just the final step in the process: the original text may have been misinterpreted, relevant sections might not be found, or evidence could be lost during generation. The official Dify documentation covers workflows, knowledge base retrieval, model connections, and runtime observation capabilities, which are suitable for organizing phased checks.

Below, using a public product manual Q&A scenario as an example, we present a repeatable error‑correction method. This is an experimental scheme designed by editors, and its success rate has not been tested with real projects. First, select a small amount of stable, manually readable material, save the complete original text and version information, then evaluate the Q&A results. This usually makes it easier to explain the issues compared to importing a large number of documents from the start.

First, outline expected evidence for each question

The set of questions should include at least several categories: direct answers, combined multi‑segment information, version conflicts, and missing material. For each question, manually record the chapter that supports the answer, key sentences, and necessary conditions. Do not write a single ideal answer, as different expressions may be correct; more importantly, identify which facts must appear and which inferences cannot go beyond the existing evidence.

For example, the old manual describes a feature that is defaultly enabled, while the new manual adds activation conditions. A valid answer should not only mention the feature name but also indicate the applicable version and conditions. Prepare another question about a feature not written in the manual, and expect the response to explain the lack of evidence. Such counterexamples can test whether the system is willing to retain uncertainty rather than generating definite answers for every question.

First‑level check: Is the text after import still conveying the original meaning?

Randomly check titles, lists, tables, and cross‑column order in the parsing results page by page. Pay special attention to whether exception conditions have separated from the main description, whether units still correspond to values, and whether headers/footers have been mixed into the body. If the negation word “not supported” in the original text is omitted, no accurate retrieval later will be able to restore the lost meaning.

You can organize the problem analysis into a separate defect list, recording the original file, location, parsing result, and impact. After fixing, re‑import the affected material and save the processed version. Do not use a generative model to fill in missing content from the original text before treating it as raw facts, otherwise it will be difficult to distinguish between the author’s statement, the parser’s output, and the model’s additions during error correction.

Second‑level check: Does the retrieval provide sufficient material?

Ignore the final answer for now and only examine what was retrieved for each question. Manually determine whether the returned segments are relevant, contain complete conditions, and include materials with the same name but different versions. Adjust keywords, splitting granularity, and retrieval strategies one factor at a time, then re‑check with the same question.

If the correct chapter appears in the candidate results but at the end, the handling method differs from a complete failure; if a table is split into segments with only values and no header, increasing the number of returned items may not solve the problem. Recording these details helps avoid attributing all issues to the model.For problems that require multiple sections to be combined, it is also necessary to check whether the evidence can be correctly connected through document identities and chapter relationships.

Third-level check: Ensure no evidence exceeds its boundary

After confirming that the retrieval results are usable, compare each key conclusion in the answer with the evidence. Check if ‘possible’ is changed to ‘guaranteed’, specific environments are expanded to all scenarios, or paragraphs adjacent to but not supporting the conclusion are cited. Proper formatting and a large number of references cannot replace item-by-item verification.

The prompt can explicitly require distinguishing original facts, summaries, and unverifiable information, but acceptance should not be limited to writing requirements. Continue observing the actual output in counter-examples, especially when there is insufficient material. If the model indeed needs to provide suggestions, treat them as a separate section and explain that they are editorial judgments based on existing materials, not the original manual text.

Maintain a small and stable regression set after release

Add the identified issues to a fixed test set and run it repeatedly when changing models, updating documents, or adjusting parsing configurations. Count parsing defects, retrieval omissions, missing conditions, and unfounded supplements separately. There is no need to rush to combine them into a overall score. When there are few issues, recording the reasons item by item usually provides more guidance for fixing than looking at a single percentage.

Before launch, also arrange permission and failure scenarios: materials without access rights should not be leaked due to retrieval, failure of the model service should be clearly explained, and long-term use of old versions should be avoided after material updates. Each stage has observable inputs and outputs, and error troubleshooting will shift from guessing to evidence-based identification. What truly deserves to be retained is a reference for the team when the same problem occurs next time.


Original source: langgenius/dify official repository ↗. Verification date: November 13, 2026.

Editor’s note: This article is original material organized by COTCC’s official side, written with AI assistance and verified against official sources. Solutions, examples, and acceptance suggestions are the editor’s design; they do not claim completion of deployment or performance testing. Specific features, licenses, and costs shall be based on the current project description.

Discussion 0

Oldest first

No public comments yet. Start the discussion.

Publish