Corrective RAG (CRAG): How to Fix Hallucinations and Irrelevant Retrieval in RAG Systems

Retrieval-Augmented Generation (RAG) has revolutionized how businesses deploy Large Language Models (LLMs) on their proprietary data. However, traditional RAG systems suffer from a major structural flaw: blind trust in vector search results. When vector databases return irrelevant or incomplete document chunks, standard RAG pipelines force the LLM to generate answers from bad context—leading to misinformation and hallucinations.

Corrective RAG (CRAG) fixes this fundamental problem. By introducing an automated Retrieval Evaluator, dynamic Knowledge Refinement, and adaptive Web Search Fallbacks, CRAG transforms passive retrieval into an intelligent, self-correcting system.

1. Introduction #

In a standard RAG pipeline, a user’s query is converted into an embedding vector, searched against a vector database, and the top-matching document chunks are passed straight into the LLM prompt. This works well when the vector store contains exact, relevant documents.

• However, vector databases operate on mathematical similarity, not absolute truth. If a database lacks the answer to a question (or contains only tangential information), it will still return the closest matching vectors.
• A traditional RAG system uncritically accepts these chunks and tells the LLM to answer based only on the provided context.
• The result? The LLM either struggles to force an answer out of unrelated text or falls back on its pre-training memory, creating confident hallucinations.
• In enterprise applications—such as HR policy lookup, medical guidance, or legal research—generating answers from wrong context can lead to costly mistakes. Corrective RAG (CRAG) was developed to solve this exact bottleneck.

2. What Is Corrective RAG (CRAG)? #

Corrective RAG (CRAG) is an advanced RAG architecture designed to detect and recover from poor document retrieval before answer generation occurs.

Instead of assuming that vector search always succeeds, CRAG places an independent Retrieval Evaluator between the database search and the LLM generator. The evaluator analyzes the relevance of the retrieved document chunks relative to the user query and assigns confidence scores.

Depending on whether the retrieved context is evaluated as Correct, Incorrect, or Ambiguous, CRAG dynamically routes the request down different execution paths:

• Refining internal knowledge when the retrieval is accurate.
• Falling back to external web search when internal documents are missing.
• Combining internal and external sources when retrieval is partially relevant.

3. Key Concepts #

To understand how CRAG operates, it helps to break down its core structural components:

• Retrieval Evaluator: A lightweight evaluation model or prompt chain that inspects each retrieved chunk and assigns a numerical relevance score.
• Confidence Thresholds: Upper (e.g., 0.7) and lower (e.g., 0.3) confidence boundaries used to categorize retrieval quality into three distinct states.
• Knowledge Refinement: A three-stage process (Decomposition, Filtering, and Re-combination) that strips away unnecessary filler text from retrieved chunks.
• Query Rewriting: An automated optimization step that converts vague human questions into keyword-dense, search-engine-friendly queries.
• Internal vs. External Knowledge: Internal knowledge refers to private documents stored in local vector databases, while external knowledge represents dynamic search results fetched from live web search APIs.

4. Detailed Explanation #

CRAG replaces static document passing with dynamic, state-aware decision-making. Below is a deep dive into its two primary operational pillars: Verdict Classification and Knowledge Refinement.

The 3 Retrieval Verdict States #

When a query retrieves document chunks from a vector database, the Retrieval Evaluator scores each chunk between 0.0 (completely irrelevant) and 1.0 (perfect match). Based on these scores, CRAG classifies the retrieval into one of three execution paths:

1. Correct State (High Relevance) #

• Condition: At least one retrieved document chunk scores above the upper threshold (0.7).
• System Action: The internal retrieval is deemed reliable. The system discards any chunks scoring below the lower threshold (< 0.3), applies Knowledge Refinement to the remaining “good documents,” and passes the clean context to the generator LLM.

2. Incorrect State (Low Relevance) #

• Condition: All retrieved document chunks score below the lower threshold ($< 0.3$).
• System Action: The internal database lacks relevant context. The system discards all internal chunks, rewrites the user query for web search compatibility, queries an external search API (such as Tavily), refines the web search results, and generates the final answer using external knowledge.

3. Ambiguous State (Partial Relevance) #

• Condition: No single document chunk scores above the upper threshold (0.7), but at least one chunk scores above the lower threshold (0.3).
• System Action: The internal retrieval is incomplete. CRAG preserves the partially relevant internal chunks and simultaneously triggers an external web search. The refined internal and external contexts are combined to give the LLM a comprehensive context payload.

The Knowledge Refinement Process #

In standard RAG, text chunks are created using fixed window sizes (e.g., 900 characters with 150-character overlap). As a result, a single retrieved chunk often contains a few sentences answering the user’s question surrounded by irrelevant background text or adjacent topics. Passing unrefined chunks wastes token budget and introduces context noise.

CRAG fixes this through Knowledge Refinement:

1. Decomposition: The raw chunk is broken down into fine-grained atomic units (sentence-level strips: S_1, S_2, ….., S_n).
2. Filtering: Each sentence strip is evaluated individually against the query. Irrelevant strips are stripped away.
3. Re-combination: The remaining high-relevance strips are concatenated back together into a compact, noise-free context string.

5. How It Works: Step-by-Step CRAG Workflow #

Here is the exact step-by-step execution path of a complete CRAG pipeline:

1. User Query Ingestion: The system receives a natural language query from the user (e.g., “What are the key updates in AI news from last month?”).
2. Vector Database Retrieval: The query is embedded into a vector space and similarity-searched against internal documents to return the top-$k$ candidate chunks.
3. Retrieval Evaluation: The Retrieval Evaluator processes each chunk against the query, assigning numerical relevance scores and textual explanations.
4. State Classification & Routing:
   • If Correct: Route directly to internal Knowledge Refinement.
   • If Incorrect or Ambiguous: Route to the Query Rewriter.
5. Query Rewriting (Web Path): An LLM converts conversational human queries into search-engine-optimized strings (e.g., adding temporal constraints like “last 30 days” or stripping conversational clutter).
6. External Web Search Execution: The rewritten query is sent to a search engine API to pull top web pages.
7. Refinement Execution:
   • Incorrect path: Refine external web pages into clean external context.
   • Ambiguous path: Refine both internal “good docs” and external web pages, then merge them into a single context payload.
8. Final Answer Generation: The LLM generator receives the user’s original question alongside the refined context payload to produce a grounded, highly accurate response.

6. Examples and Real-World Scenarios #

Scenario 1: The Out-of-Domain Technical Query (Incorrect Path) #

• User Query: “What is a Transformer architecture in deep learning?”
• Database Context: A vector database built exclusively from three classic machine learning textbooks (covering Random Forests, Support Vector Machines, and Convolutional Neural Networks).
• Traditional RAG Behavior: The vector database matches vectors for CNNs or MLPs because it must return something. The LLM receives CNN text, gets confused, and either attempts to explain Transformers using CNN concepts or hallucinates wildly.
• CRAG Behavior: The Retrieval Evaluator scores all returned chunks below 0.3 (Verdict: Incorrect). CRAG rewrites the query to “Transformer architecture deep learning overview”, queries Tavily for live web documentation, refines the search results, and delivers a precise explanation of self-attention mechanisms.

Scenario 2: The Multi-Part Question (Ambiguous Path) #

• User Query: “How does Batch Normalization compare with Layer Normalization?”
• Database Context: The database contains extensive chapters on Batch Normalization, but lacks documentation on Layer Normalization.
• CRAG Behavior: Chunks explaining Batch Normalization score 0.5 (above lower threshold 0.3, but below upper threshold 0.7). The verdict is Ambiguous. CRAG retains the Batch Normalization chunks, rewrites the query to pull Layer Normalization details from the web, combines both sources into a unified context, and provides a clear comparative response.

7. Comparison Table: Traditional RAG vs. Corrective RAG (CRAG) #

8. Advantages and Limitations #

Advantages #

• Significant Hallucination Reduction: By blocking low-relevance chunks from reaching the generator, CRAG prevents forced hallucinations.
• High Operational Resilience: Automatically bridges internal knowledge gaps by pulling live, verified web data.
• Token Efficiency: Sentence-level filtering removes context noise, saving prompt tokens and keeping the LLM focused on core facts.
• Search Engine Optimization: Query rewriting ensures search engine APIs return crisp, time-bounded, and keyword-dense web results.

Limitations #

• Increased Latency: Adding evaluation, rewriting, filtering, and optional web search nodes increases total response time per query.
• Higher API Costs: Running evaluator chains and making external search engine API calls incurs additional operational expense.
• External Dependency: Fallback execution paths depend on internet connectivity and third-party search API availability.

9. Real-World Applications #

• Enterprise HR & Policy Bots: Prevents outdated local PDF documents from giving employees incorrect policy or compliance advice by verifying context before answering.
• Healthcare & Medical Search: Ensures clinical decision tools do not output wrong treatment recommendations when exact clinical guidelines are missing from local databases.
• Legal Research Platforms: Guarantees that legal assistants cross-reference live statutory updates on the web when local case databases lack recent precedent.
• Customer Support Automation: Bypasses missing helpdesk articles by fetching verified resolution steps directly from public developer documentation.

10. Important Points for Revision #

As we wrap up our deep dive into Corrective Retrieval-Augmented Generation (CRAG) , here is a quick-reference guide to the most critical takeaways. These points are essential for understanding how CRAG moves beyond traditional RAG limitations.

1. Core Purpose: Eliminating Blind Trust #

The fundamental goal of CRAG is to eliminate blind trust in vector search. Traditional RAG systems assume that whatever is retrieved is relevant. CRAG changes this by evaluating the quality of the retrieved context before the generation phase begins.

2. The Retrieval Evaluator #

At the heart of the system is a Retrieval Evaluator. This component acts as a gatekeeper, assigning a confidence score (ranging from 0.00.0 to 1.01.0) to every individual chunk of retrieved data.

3. Default Thresholds #

To make decisions, the system relies on two default confidence thresholds:

• Upper Threshold (≥0.7≥0.7): Indicates high confidence in the retrieved information.
• Lower Threshold (<0.3<0.3): Indicates low confidence (likely irrelevant or incorrect).

4. Three Dynamic Pathways #

Based on the evaluator’s score, the system takes one of three distinct actions:

• Correct (High Confidence):
  • Action: The internal chunks are refined and used directly to generate the answer.
• Incorrect (Low Confidence):
  • Action: The system discards the internal data. It rewrites the query, performs a Web search, refines the web documents, and then generates the answer.
• Ambiguous (Mid Confidence):
  • Action: A hybrid approach. It rewrites the query and performs a web search, then refines and merges both internal and web documents before generating the answer.

5. Knowledge Refinement Stages #

When refinement occurs, it follows a specific three-step pipeline:

1. Decomposition: Splitting retrieved documents into smaller “sentence strips.”
2. Filtering: Scoring these strips to remove irrelevant information.
3. Re-combination: Rebuilding a clean, concise context from the surviving strips.

6. Model Efficiency in Production #

CRAG is designed for real-world performance. In production environments, you don’t need massive models for evaluation. Lightweight models (such as fine-tuned T5-Large models with 770M parameters) can be used as evaluators to minimize latency and inference costs.

11. Interview / Exam Questions #

Question 1: Why does traditional RAG fail when a vector database returns low-relevance documents? #

Answer: Traditional RAG blindly trusts vector search results. When a database lacks exact answers, vector search returns the closest mathematical matches, even if they are topically unrelated. Because traditional RAG forces the LLM to answer using this bad context, the LLM either distorts the provided text or hallucinates answers using pre-training memory.

Question 2: What are the three retrieval evaluation outcomes in CRAG, and how are they triggered? #

Answer:

1. Correct: Triggered when at least one retrieved chunk scores at or above the upper confidence threshold ($\ge 0.7$).
2. Incorrect: Triggered when all retrieved chunks score below the lower confidence threshold ($< 0.3$).
3. Ambiguous: Triggered when no chunk scores above $0.7$, but at least one chunk scores above $0.3$.

Question 3: Describe the three steps of Knowledge Refinement in CRAG. #

Answer:

1. Decomposition: Splitting raw text chunks into individual sentence-level strips.
2. Filtering: Evaluating each sentence strip against the user query and dropping irrelevant strips.
3. Re-combination: Joining the remaining relevant strips into a clean, noise-free context payload.

Question 4: What is the role of Query Rewriting in a CRAG pipeline? #

Answer: Raw user queries are often conversational, vague, or missing temporal boundaries. Query Rewriting uses an LLM to transform human input into an optimized search engine query (adding keywords, time constraints like “last 30 days”, or specific search flags) to maximize web search accuracy.

Question 5: How does CRAG handle the “Ambiguous” retrieval state differently from the “Incorrect” state? #

Answer: In the Incorrect state, internal chunks are completely discarded, and the answer is generated using external web search data alone. In the Ambiguous state, internal “good documents” are preserved and merged with external web search results, creating a hybrid context that covers both internal and external facts.

12. Quick Revision #

Corrective RAG (CRAG) transforms standard RAG from a passive document-retrieval pipeline into an active, self-correcting AI system. By evaluating retrieved chunks, stripping out background noise through Knowledge Refinement, and dynamically triggering Web Search with Query Rewriting, CRAG ensures Large Language Models receive only verified, high-quality context—eliminating hallucinations and bringing enterprise reliability to AI applications.

Corrective RAG Quiz #