Selecting the Right LLM and Evaluating RAG Applications

Choosing an LLM is not simply about selecting the model with the highest benchmark score. Engineers must balance
accuracy, latency, reliability, context-window requirements and operating cost.

At the same time, a RAG application must be continuously tested to determine whether it retrieves the right
information, generates faithful answers, protects sensitive information and maintains its performance after
code or prompt changes.

This is why production AI engineering requires a structured evaluation strategy instead of informal
“vibe testing”.

What Is AI Application Evaluation? #

AI Application Evaluation is the systematic practice of measuring, benchmarking and monitoring an
LLM-powered product throughout its lifecycle.

Traditional software is largely deterministic: the same input normally produces the same output.
LLM applications are different. They are probabilistic systems, meaning the same prompt can produce
different wording, structures or even different reasoning paths across multiple runs.

Evaluation provides a bridge between this probabilistic behavior and the reliability expected from
production software.

Key Concepts in LLM and RAG Evaluation #

Custom Model Evaluations vs. Public Benchmarks #

Public benchmarks such as MMLU, GSM8K and HumanEval are useful for understanding the general capabilities
of foundation models. They provide broad directional information about model intelligence.

However, public benchmarks cannot fully predict how a model will perform on your own database schema,
business terminology, prompts or proprietary workflows.

This is where Custom Model Evals become important.

Custom evaluations run candidate models against a domain-specific
Golden Dataset containing representative inputs and verified target outputs.

The RAG Triad #

RAG applications require evaluation of the relationship between the user’s question, the retrieved context
and the final generated answer.

The three major relationships are:

• Context Relevance:
  Does the retriever fetch context that is actually related to the user’s question?
• Faithfulness:
  Is the generated answer supported by the retrieved context without introducing unsupported facts?
• Answer Relevance:
  Does the final response directly answer what the user asked?

Three Levels of AI System Evaluation #

A production AI application can be evaluated at three different levels.

• Component Level:
  Individual components such as the retriever or generator are tested independently.
• Pipeline Level:
  The interaction between the components is evaluated as a complete pipeline.
• Application Level:
  The complete user experience is evaluated, including correctness, safety, latency and cost.

Selecting the Right LLM with Custom Model Evals #

When starting an AI project, choosing a model simply because it is popular or ranks highly on a public
leaderboard can lead to poor production decisions.

A larger model may provide excellent general performance while being unnecessarily expensive or slow for
a specific business task.

A better approach is to combine business requirements, candidate filtering and custom evaluations.

Start with Business and Technical Requirements #

Before testing models, define the operational boundaries of the application.

• Task Definition:
  Clearly define the input and output transformation.
• Cost Ceiling:
  Determine the maximum monthly operating budget.
• Token Usage:
  Estimate average input and output tokens per request.
• Prompt Caching:
  Identify static prompts or schemas that can benefit from caching.
• Latency:
  Define the maximum acceptable response time.
• Context Window:
  Determine whether the application requires short or long context.
• Correctness:
  Define the acceptable error rate according to the business impact.

Shortlist Candidates Using Leaderboards #

Once the requirements are clear, public model leaderboards can be used as a filtering mechanism rather than
as the final decision-maker.

• Collect performance, cost and speed information.
• Remove models that exceed the projected budget.
• Calculate a composite score for remaining candidates.
• Give higher weight to task capability and accuracy.
• Shortlist several proprietary and open-weight models.

Run Custom Evaluations on a Golden Dataset #

The shortlisted models should then be evaluated using a representative Golden Dataset.
The dataset should contain both common requests and difficult edge cases.

For structured tasks such as Text-to-SQL, the evaluation should focus on
functional correctness rather than merely comparing generated strings.

Why Result Tables Matter in Text-to-SQL Evaluation #

Two SQL queries can look completely different while producing exactly the same result.
For example, one query may use explicit JOINs while another may use subqueries.

Therefore, comparing the generated SQL string with a reference SQL string can incorrectly mark a valid query
as wrong.

A stronger evaluation strategy executes both queries against the test database and compares their resulting
tables after normalizing formats and handling row-order rules.

Case Study: Text-to-SQL for Sports Analytics #

Imagine a high-traffic sports media platform that wants to allow fans to ask natural-language questions
during live matches.

During major matches, thousands of users may ask questions simultaneously. Manually writing SQL queries
for every request is not scalable.

Operational Requirements #

Custom Evaluation Results #

How to Evaluate a RAG Application #

After selecting the base model, the next challenge is evaluating the complete RAG application.

A RAG system contains multiple components, meaning a bad answer may be caused by poor retrieval,
generation problems or an interaction between both.

Evaluating the Retriever #

The retriever is responsible for finding relevant information from the knowledge base.

Evaluation should determine whether the correct chunks are being retrieved and whether unnecessary noise
is appearing among the top results.

Two important metrics are Context Recall and Context Precision.

Evaluating the Generator #

The generator receives the retrieved context and produces the final answer.

A generator can be tested independently by providing it with carefully curated context and measuring
whether its answer remains grounded in that context.

Important metrics include Faithfulness, Answer Relevance and
Citation Accuracy.

Evaluating the Complete RAG Pipeline #

Once the retriever and generator have been tested independently, they can be connected and evaluated as
an end-to-end system.

Application-Level Evaluation #

A production RAG application needs more than retrieval and generation metrics.
The complete application should also be tested for correctness, completeness, style, security and
operational performance.

• Correctness
• Completeness
• Style Consistency
• Toxicity
• PII Leakage
• Jailbreak Resistance
• Prompt Injection Resistance
• Latency
• Cost per Query
• Token Efficiency

Case Study: CampusX Doubt Solver RAG Chatbot #

Consider an educational RAG chatbot designed to answer student technical questions using transcripts
from an LLM course.

Instead of writing isolated evaluation scripts, the system can use an organized evaluation suite.

project_root/
├── src/
│   ├── retriever.py
│   ├── generator.py
│   └── pipeline.py
│
├── evals/
│   ├── eval_retriever.py
│   ├── eval_generator.py
│   ├── eval_pipeline.py
│   └── eval_safety.py
│
└── run_evals.py

This structure separates application code from evaluation code and makes it easier to run the complete
test suite repeatedly.

Regression Testing and CI/CD for AI Applications #

Building an evaluation suite is only the beginning. AI applications change frequently.
Developers may modify prompts, chunk sizes, embedding models, model versions or generation parameters.

Every such change can potentially affect application quality.

Regression Testing #

When developers change system prompts, vector chunking strategies or embedding models, the evaluation
suite can compare the new results with the previous baseline.

Experiment Tracking #

Important experiment parameters such as temperature, model version, chunk overlap and retrieval settings
can be logged using experiment-tracking platforms.

CI/CD Quality Gates #

Evaluation can become an automated deployment gate.
For example, a team may define minimum quality requirements such as:

If the new version fails these requirements, the deployment pipeline can automatically block the release.

Online Evaluation and Production Observability #

Offline evaluation is essential before deployment, but it cannot capture every problem that appears in
real-world production.

Real users may submit unexpected questions, typos, mixed-language queries or adversarial prompts.
Production traffic can also reveal latency and scaling problems that are difficult to reproduce locally.

Another important challenge is data and concept drift. Information that was correct when the
Golden Dataset was created may become outdated later.

Building a Self-Improving Feedback Loop #

Production observability can turn real-world failures into future evaluation cases.

• Telemetry Logging:
  Track latency, token usage and user feedback.
• Stratified Sampling:
  Select high-risk or low-rated sessions for deeper evaluation.
• Drift Detection:
  Monitor quality metrics over rolling time windows.
• Feedback Loop:
  Sanitize real production failures and add them to the offline Golden Dataset.

This creates a continuous improvement cycle where the application becomes better at handling the same types
of failures that previously occurred in production.

Model Evaluation vs. Application Evaluation #

Offline Evaluation vs. Online Evaluation #

Advantages and Limitations #

Why Custom Model Evaluations Matter #

Why RAG Evaluation and CI/CD Gating Matter #

Real-World Applications of AI Evaluation #

Enterprise Search and Documentation #

Organizations can evaluate internal HR, legal and documentation assistants to ensure responses remain grounded
in approved company documents.

Financial Data Extraction #

Text-to-SQL and document-processing systems can be evaluated to ensure financial values are extracted
without calculation or syntax errors.

Customer Support Automation #

Online monitoring can track customer sentiment, resolution rates and policy compliance in real time.

Autonomous AI Agents #

Agentic workflows can be evaluated for multi-step tool calls, API parameters and error-recovery behavior
before being deployed to users.

Important Points to Remember #

Practice and Interview Questions #

Final Takeaway #

Building a production-ready AI application requires much more than selecting a powerful LLM.
The complete engineering process combines model selection, custom evaluation, RAG testing, safety validation,
CI/CD regression testing and continuous production monitoring.

Remember:

The goal of AI evaluation is not simply to determine whether a model can generate a good answer.
The goal is to build an AI system that remains accurate, reliable, safe, cost-effective and measurable
as it evolves from development to production.