--- title: "LLM Model Evals vs Application Evals: The Complete Guide to Offline and Online AI Testing" description: "Deploying Large Language Models (LLMs) into real-world software applications is vastly different from building a simple prototype. While generating text with an API call takes only a few minutes, ensuring that an AI application is accurate, cost-effective, safe, and performant…" url: "https://csestudy247.com/docs/llm-model-evals-vs-application-evals-the-complete-guide-to-offline-and-online-ai-testing/" updated: "2026-10-05" category: LLM Evaluations --- # LLM Model Evals vs Application Evals: The Complete Guide to Offline and Online AI Testing Deploying Large Language Models (LLMs) into real-world software applications is vastly different from building a simple prototype. While generating text with an API call takes only a few minutes, ensuring that an AI application is accurate, cost-effective, safe, and performant requires a rigorous testing methodology. Relying on casual testing—asking a model a handful of questions and judging it by "vibe"—creates massive risks in production, including hallucinated facts, data leakage, latency spikes, and runaway operational costs. To build production-grade AI systems, engineers must master two core evaluation domains: - **Model Evaluations (Model Evals):** Measuring and selecting the right underlying LLM based on its core capabilities and benchmarks. - **Application Evaluations:** Testing the full application workflow before launch and continuously monitoring live production traffic after deployment. This guide provides a comprehensive breakdown of model capabilities, offline test pipelines, online telemetry, and the architecture required to build self-improving AI systems. ## Moving Beyond Vibe Testing In traditional software development, code is deterministic. An input yields a predictable, repeatable output, and unit tests pass or fail based on exact matches. LLM-powered applications are probabilistic. The same user prompt can produce varying phrasing, tone, or structure across runs. Because of this non-deterministic nature, developers often fall back on **vibe testing**—informally trying five to ten prompts, feeling satisfied with the output, and shipping the feature to users. Vibe testing fails in production because it is: ### Subjective Judgments are based on general impressions rather than quantifiable metrics. ### Non-Repeatable You cannot objectively measure whether updating a prompt or changing a database improved performance across edge cases. ### Blind to Scale It misses latency bottlenecks, cost inflation, and emergent errors that only appear when thousands of users interact with the system simultaneously. ### Systematic Evaluation Systematic evaluation replaces vibe testing with repeatable, data-driven frameworks that test AI systems across their entire lifecycle. ## What Are Model Evals, Offline Evals, and Online Evals? Understanding LLM evaluation requires distinguishing between what is being tested and when the test occurs. flowchart TD A\["LLM Evaluation"\] A --> B\["Model Evals"\] A --> C\["Application Evals"\] B --> B1\["Raw Model Capabilities"\] B1 --> B2\["Reasoning"\] B2 --> B3\["Model Selection"\] C --> D\["Offline Application Evals"\] D --> D1\["Pre-Deployment"\] D1 --> D2\["CI/CD"\] D2 --> D3\["Golden Dataset"\] D3 --> D4\["Regression Testing"\] C --> E\["Online Application Evals"\] E --> E1\["Production Traffic"\] E1 --> E2\["Telemetry"\] E2 --> E3\["Monitoring"\] E3 --> E4\["Drift Detection"\] %% ROOT classDef root fill:#2563eb,color:#ffffff,stroke:#1d4ed8,stroke-width:4px,font-size:24px,font-weight:bold; %% MAIN classDef main fill:#7c3aed,color:#ffffff,stroke:#6d28d9,stroke-width:3px,font-size:21px,font-weight:bold; %% MODEL classDef model fill:#dbeafe,color:#1e3a8a,stroke:#3b82f6,stroke-width:3px,font-size:19px,font-weight:bold; %% OFFLINE classDef offline fill:#16a34a,color:#ffffff,stroke:#15803d,stroke-width:3px,font-size:20px,font-weight:bold; classDef offlineDetail fill:#dcfce7,color:#166534,stroke:#22c55e,stroke-width:3px,font-size:19px,font-weight:bold; %% ONLINE classDef online fill:#ea580c,color:#ffffff,stroke:#c2410c,stroke-width:3px,font-size:20px,font-weight:bold; classDef onlineDetail fill:#ffedd5,color:#9a3412,stroke:#f97316,stroke-width:3px,font-size:19px,font-weight:bold; class A root; class B,C main; class B1,B2,B3 model; class D offline; class D1,D2,D3,D4 offlineDetail; class E online; class E1,E2,E3,E4 onlineDetail; ## LLM Model Evals Model Evals measure the raw capabilities, reasoning powers, and behavioral traits of an underlying base model under controlled conditions. **Core question:** Which foundation model possesses the best core skills for our workload? Model evaluations are particularly useful when comparing foundation models such as proprietary cloud models and self-hosted open-source models. ## Offline Application Evals Offline Evals take place before deployment during development and CI/CD pipelines. They evaluate the entire application—including prompts, vector databases, search retrievers, and guardrails—against a pre-curated test dataset containing known inputs and target answers. **Offline evaluation focuses on:** correctness, regression prevention, release confidence, and controlled comparison of system changes. ### Release Gating Automated thresholds can be integrated into deployment pipelines. If a code push yields an offline evaluation score below a defined threshold, the build can automatically be blocked from release. ### Version Comparison Teams can evaluate changes across system prompts, embedding models, vector databases, retrieval strategies, or retrieval chunk sizes using a fixed benchmark dataset. ### Regression Testing Regression testing ensures that fixes made to resolve one specific user issue do not silently break existing capabilities across other query types. ## Online Application Evals Online Evals run continuously after deployment on live user traffic. Because real-world queries arrive without pre-written answer keys, online evaluations analyze operational metrics, user feedback, and semantic health signals to ensure that the application continues running normally in production. **Online evaluation focuses on:** operational health, unexpected behavior, drift, user feedback, and production anomalies. ### Unanticipated Inputs Real users interact with systems in unexpected ways—using mixed languages, incomplete sentences, emotional requests, complex instructions, or prompt injection attempts that were never included in the offline test set. ### Emergent and Scale Failures Some system issues only manifest under concurrent user load, such as server latency spikes or subtle biases visible only across thousands of conversations. ### Data and Concept Drift Business policies, product pricing, and documentation can change over time. An offline test set created months earlier can become obsolete, creating a disconnect between offline evaluation scores and real user satisfaction. ## Model Selection and Engineering Tradeoffs Choosing an LLM for an application is rarely about picking the single most intelligent model on the market. AI engineers must evaluate tradeoffs between: ### Intelligence How accurately the model performs the target task. ### Cost The operational cost associated with inference and token usage. ### Latency How quickly the model can return useful results. ### Deployment Architecture Proprietary cloud APIs versus self-hosted open-source models. ## The 8 Core LLM Capabilities Model evaluations judge LLMs across eight fundamental capability domains. ### Knowledge & Reasoning Factual recall across diverse fields and multi-step logical deduction. ### Coding & Software Engineering Generating functional code, fixing bugs, refactoring multi-file codebases, and executing CLI commands. ### Mathematics Solving grade-school word problems, competition-level mathematics, and research-grade symbolic reasoning. ### Long Context Retaining, searching, and summarizing information across very large context windows without significant performance degradation. ### Vision & Multimodal Processing and reasoning over images, diagrams, documents, and video inputs alongside text. ### Agentic & Tool Use Executing structured function calls, navigating APIs, browsing the web, and interacting with external environments. ### Safety & Alignment Resisting adversarial prompt injections and jailbreaks, avoiding harmful content, and maintaining truthfulness. ### Instruction Following Adhering to structural constraints, formatting requirements, length limits, and negative constraints. ## Standardized Benchmarks vs. Custom Evaluation Sets Public benchmarks provide a general indication of model intelligence, but they should not be treated as the only source of evidence when selecting a model for a production application. | Feature | Standardized Benchmarks | Custom Evaluation Sets | | --- | --- | --- | | Primary Goal | Measure general model intelligence and reasoning. | Measure performance on a specific business task. | | Dataset Source | Public academic or industry datasets such as MMLU or SWE-bench. | Proprietary, curated production data. | | Target Audience | Frontier AI labs and model researchers. | AI product engineers and application developers. | | Cost & Latency Focus | Often secondary or omitted. | Primary evaluation criteria alongside accuracy. | | Product Relevance | Provides general directional guidance. | Provides direct evidence of product readiness. | ## Reference-Based vs. Reference-Free Evals ### Reference-Based Evals Testing output against a ground-truth "answer key". These are primarily useful in offline evaluation where expected answers are available. ### Reference-Free Evals Judging output quality without a known answer key by using heuristic checks, user behavior signals, or an LLM-as-a-Judge analyzing context alignment. ## Custom Model Evaluation Workflow Public benchmarks can give a general sense of model intelligence, but running a custom evaluation on task-specific data reveals the true performance-to-cost ratio for a specific use case. flowchart TD A\["Custom Evaluation Workflow"\] A --> B\["Model A Large / Top Model"\] A --> C\["Model B Small / Fast Model"\] B --> B1\["High Benchmark Rank"\] B --> B2\["Higher Cost"\] B --> B3\["Higher Latency"\] C --> C1\["Mid-Tier Benchmark"\] C --> C2\["Lower Cost"\] C --> C3\["Lower Latency"\] B1 --> D\["Golden Test Dataset"\] B2 --> D B3 --> D C1 --> D C2 --> D C3 --> D D --> E\["Evaluation Results"\] E --> F\["Model A 94% Accuracy"\] E --> G\["Model B 91% Accuracy"\] F --> H\["Business Decision"\] G --> H H --> I\["Select Model B Lower Cost + Lower Latency with Small Accuracy Tradeoff"\] classDef title fill:#2563eb,color:#fff,stroke:#1d4ed8,stroke-width:3px,font-size:20px,font-weight:bold; classDef modelA fill:#dbeafe,color:#1e3a8a,stroke:#3b82f6,stroke-width:2.5px,font-size:17px; classDef modelB fill:#dcfce7,color:#166534,stroke:#22c55e,stroke-width:2.5px,font-size:17px; classDef metricA fill:#eff6ff,color:#1e40af,stroke:#60a5fa,stroke-width:2px,font-size:16px; classDef metricB fill:#f0fdf4,color:#166534,stroke:#4ade80,stroke-width:2px,font-size:16px; classDef dataset fill:#fef3c7,color:#92400e,stroke:#f59e0b,stroke-width:3px,font-size:18px,font-weight:bold; classDef result fill:#f3e8ff,color:#6b21a8,stroke:#a855f7,stroke-width:2.5px,font-size:17px; classDef decision fill:#ffedd5,color:#9a3412,stroke:#f97316,stroke-width:3px,font-size:18px,font-weight:bold; class A title; class B modelA; class C modelB; class B1,B2,B3 metricA; class C1,C2,C3 metricB; class D dataset; class E,F,G result; class H,I decision; ## Offline Evals vs. Online Evals flowchart LR A\["LLM Application"\] A --> B\["Offline Evals"\] A --> C\["Online Evals"\] B --> B1\["Pre-Deployment"\] B --> B2\["Golden Dataset"\] B --> B3\["Ground Truth"\] B --> B4\["Regression Testing"\] B --> B5\["CI/CD Release Gates"\] C --> C1\["Post-Deployment"\] C --> C2\["Live User Traffic"\] C --> C3\["Telemetry"\] C --> C4\["Monitoring"\] C --> C5\["Drift Detection"\] classDef root fill:#2563eb,color:#fff,stroke:#1d4ed8,stroke-width:3px,font-size:20px,font-weight:bold; classDef offline fill:#16a34a,color:#fff,stroke:#15803d,stroke-width:3px,font-size:18px,font-weight:bold; classDef online fill:#ea580c,color:#fff,stroke:#c2410c,stroke-width:3px,font-size:18px,font-weight:bold; classDef offlineDetail fill:#dcfce7,color:#166534,stroke:#4ade80,stroke-width:2px,font-size:16px; classDef onlineDetail fill:#ffedd5,color:#9a3412,stroke:#fb923c,stroke-width:2px,font-size:16px; class A root; class B offline; class C online; class B1,B2,B3,B4,B5 offlineDetail; class C1,C2,C3,C4,C5 onlineDetail; | Dimension | Offline Evals | Online Evals | | --- | --- | --- | | Execution Phase | Development and CI/CD testing pipelines. | Continuous live production monitoring. | | Primary Objective | Verify functional correctness and prevent regressions. | Monitor operational health, drift, and unexpected failures. | | Dataset Type | Static, pre-curated Golden Dataset. | Dynamic, live production user traffic. | | Answer Key | Ground-truth reference answer key. | Generally operates without a ground-truth answer key. | | Primary Metrics | Factual accuracy, instruction following, and context recall. | Latency, token cost, toxicity, drift, and score distributions. | | Key Mechanism | Automated tests and release gates. | Telemetry logging, sampling, dashboards, and alerts. | ## Correctness vs. Normality In an offline setting, evaluation measures correctness by comparing generated outputs to known ground-truth answers. In an online setting, live user queries generally have no pre-labeled answers. Therefore, online evaluation shifts from measuring exact correctness to measuring **normality**—verifying whether the application is performing consistently relative to established baseline behavior. flowchart TD A\["Production Behavior"\] A --> B\["Measure Normality"\] B --> C\["Historical Baseline"\] C --> D\["Week 1 Normal Distribution"\] C --> E\["Week 2 Normal Distribution"\] C --> F\["Week 3 Distribution Shift"\] D --> G\["Compare Metrics"\] E --> G F --> G G --> H{"Significant Deviation?"} H -->|No| I\["Normal Operation"\] H -->|Yes| J\["Anomaly Detected"\] J --> K\["Alert Engineering Team"\] K --> L\["Investigate Root Cause"\] classDef main fill:#2563eb,color:#fff,stroke:#1d4ed8,stroke-width:3px,font-size:19px,font-weight:bold; classDef normal fill:#dcfce7,color:#166534,stroke:#22c55e,stroke-width:2.5px,font-size:17px; classDef warning fill:#fef3c7,color:#92400e,stroke:#f59e0b,stroke-width:2.5px,font-size:17px; classDef alert fill:#fee2e2,color:#991b1b,stroke:#ef4444,stroke-width:3px,font-size:18px,font-weight:bold; class A,B,C,G,H main; class D,E,I normal; class F warning; class J,K,L alert; **Production anomaly example:** A system may suddenly produce an unusual distribution of outputs even though individual live responses cannot be directly compared with a known answer key. Detecting this behavioral shift can trigger an engineering investigation. ## How Model Evaluation Works Every model evaluation follows a structured four-step methodology. flowchart LR A\["Select Target Capability"\] --> B\["Select or Build Test"\] --> C\["Execute Under Fixed Protocols"\] --> D\["Score & Interpret Results"\] --> E\["Model Selection Decision"\] classDef step fill:#dbeafe,color:#1e3a8a,stroke:#3b82f6,stroke-width:3px,font-size:18px,font-weight:bold; classDef final fill:#dcfce7,color:#166534,stroke:#22c55e,stroke-width:3px,font-size:18px,font-weight:bold; class A,B,C,D step; class E final; ### Select the Target Capability Decide precisely which skill to test, such as mathematical reasoning, instruction following, or code generation. ### Select or Build the Test Choose a recognized public benchmark or construct a custom dataset representing target production tasks. ### Execute Under Fixed Protocols Run the target models through the dataset under standardized, repeatable conditions. Parameters such as temperature, system context, and prompt structure should be controlled where appropriate. ### Score and Interpret Results Compute aggregate metrics, evaluate tradeoffs against cost and speed, and document the model selection rationale. ## End-to-End Online Evaluation Pipeline A production online evaluation pipeline consists of six interconnected stages. %%{init: { 'theme': 'base', 'themeVariables': { 'fontSize': '8px', 'fontFamily': 'Arial' }, 'flowchart': { 'nodeSpacing': 6, 'rankSpacing': 10, 'padding': 2, 'curve': 'basis', 'useMaxWidth': true } }}%% flowchart TD A\["Production Traffic"\]:::traffic B\["Telemetry Logging"\]:::pipeline C\["Signal Extraction"\]:::pipeline D\["Stratified Sampling"\]:::pipeline E\["Metric Computation"\]:::pipeline F\["Dashboard & Alerts"\]:::monitoring G\["Close the Loop"\]:::monitoring H\["Golden Dataset"\]:::feedback I\["Offline Eval"\]:::feedback A --> B --> C --> D --> E --> F --> G --> H --> I --> A classDef traffic fill:#2563eb,color:#ffffff,stroke:#1d4ed8,stroke-width:1px,font-size:8px classDef pipeline fill:#dbeafe,color:#1e3a8a,stroke:#3b82f6,stroke-width:1px,font-size:8px classDef monitoring fill:#ffedd5,color:#9a3412,stroke:#f97316,stroke-width:1px,font-size:8px classDef feedback fill:#dcfce7,color:#166534,stroke:#22c55e,stroke-width:1px,font-size:8px ### Structured Telemetry Logging Every conversation turn is recorded asynchronously without adding user-facing latency. Each log entry can capture: - **Identifiers:** Conversation ID, Turn ID, User ID, and Timestamp. - **Content:** User prompt, retrieved context documents, and generated response. - **Operational Telemetry:** Response latency, prompt tokens, completion tokens, total cost, and HTTP status codes. - **User Feedback Signals:** Explicit thumbs-up/thumbs-down, copy actions, or escalation requests. - **Privacy Controls:** Automatic masking or redaction of Personally Identifiable Information before storage. ### Signal Extraction and Classification Production signals are divided into two distinct processing streams. ### Captured Signals Data points available directly from system logs without extra computation, such as latency, token counts, error rates, and thumbs-up/down feedback. ### Computed Signals Quality metrics generated by passing logs through secondary evaluation models, such as faithfulness, answer relevance, toxicity, or hallucination checks. ### Stratified Sampling Evaluating 100% of production traffic with a secondary LLM-as-a-Judge can significantly increase operating costs. Instead, applications can use stratified sampling to select specific, high-risk conversations for detailed evaluation. - Queries flagged with negative user ratings. - Conversations involving high-risk topics such as refunds, billing, or account security. - Session traces where users repeatedly reframed their query or abruptly ended the interaction. ### Metric Aggregation Across Time Windows Rather than judging individual interactions in isolation, telemetry data can be aggregated across time windows such as 1-hour, 24-hour, and 7-day rolling windows. This reveals broader trends, system-level degradation, and performance drift. ### Dashboard Visualization and Automated Alerting Aggregated metrics feed into monitoring dashboards. When an operational or quality metric breaches a defined threshold, an automated alert can notify the engineering team through Slack, email, or an incident management system. ### Closing the Self-Improving Feedback Loop Production failures flagged by online evaluations can be annotated and imported back into the offline Golden Dataset. Future system updates can then be tested against these real-world failure cases before release, creating a continuous self-improving development cycle. ## Real-World Examples ### Custom Model Evaluation for Email Classification Imagine a company that needs to route incoming customer emails into three departments: **Billing**, **Technical Support**, or **General Inquiry**. ### Option A — Top-Tier Foundation Model - Ranked highly on public benchmarks - Higher token cost - Average latency: 4.1 seconds ### Option B — Compact Open-Source Model - Mid-tier benchmark ranking - Much lower token cost - Average latency: 0.9 seconds ### The Test Setup The engineering team creates a custom evaluation dataset of 300 real, anonymized past customer emails with verified category labels. Both models are evaluated on exactly the same dataset. ### The Results | Model | Accuracy | Latency | Cost | | --- | --- | --- | --- | | Option A | 94% | 4.1 seconds | $15.00 / 1M tokens | | Option B | 91% | 0.9 seconds | $0.50 / 1M tokens | ### Engineering Decision Although Option A ranks higher overall, Option B delivers nearly the same classification accuracy while significantly reducing token costs and latency. **Lesson:** Custom evaluation sets can prevent over-engineering by showing the real accuracy-to-cost-to-latency tradeoff for a specific business task. ### Online Evaluation of an Exam Grading Application An AI application is deployed to evaluate student essay responses and assign scores between 0 and 100. ### The Challenge In production, the system receives new, ungraded student essays. Because no human answer key exists for these live submissions, the system cannot directly measure scoring correctness in real time. ### The Online Solution The team monitors the distribution of assigned scores across weekly rolling windows. A historical baseline might show that student scores form a predictable distribution centered around a stable average. If a future week suddenly shows a dramatic shift—for example, an unusually large percentage of submissions receiving very high scores—the online pipeline can detect the anomaly without requiring a ground-truth answer key. Investigation may reveal that a prompt modification caused the model to become overly lenient. The team can then roll back the prompt change and restore normal evaluation behavior. ## Standardized Benchmarks vs. Custom Evaluation Sets | Feature | Standardized Benchmarks | Custom Evaluation Sets | | --- | --- | --- | | Primary Goal | Measure general model intelligence and reasoning. | Measure performance on a specific business task. | | Dataset Source | Public academic or industry datasets. | Proprietary, curated production data. | | Target Audience | Frontier AI labs and model researchers. | AI product engineers and application developers. | | Cost & Latency Focus | Often secondary or omitted. | Primary evaluation criteria alongside accuracy. | | Product Relevance | General directional guidance. | Direct evidence of product readiness. | ## Offline Evals vs. Online Evals | Dimension | Offline Evals | Online Evals | | --- | --- | --- | | Execution Phase | Development and CI/CD testing pipelines. | Continuous live production monitoring. | | Primary Objective | Verify functional correctness and prevent regressions. | Monitor operational health, drift, and unexpected failures. | | Dataset Type | Static, pre-curated Golden Dataset. | Dynamic, live production user traffic. | | Answer Key Availability | Uses a ground-truth reference answer key. | Generally operates without a ground-truth answer key. | | Primary Metrics | Factual accuracy, instruction following, context recall. | Latency, token cost, toxicity, drift, score distributions. | | Key Mechanism | Automated tests and release gates. | Telemetry logging, sampling, dashboards, and alerts. | ## Advantages and Limitations ### Model Evals and Custom Benchmarking ### Advantages - Eliminates guesswork during model selection. - Reveals true cost-performance tradeoffs. - Provides evidence for architectural decisions. ### Limitations - Public benchmarks can suffer from data contamination. - Custom datasets require manual curation effort. ### Offline Application Evals ### Advantages - Catches bugs before users see them. - Enables automated CI/CD deployment gates. - Prevents prompt or model changes from causing regressions. ### Limitations - Limited to anticipated scenarios included in the test set. - Cannot fully reproduce real-world scale. - Cannot reproduce every evolving user behavior. ### Online Application Evals ### Advantages - Detects real-world operational issues. - Detects latency spikes and concept drift. - Captures authentic user feedback. - Feeds production failures back into future test sets. ### Limitations - Secondary evaluation models can increase API costs. - Sampling is often needed to control evaluation cost. - Live traffic usually lacks exact ground-truth answer keys. ## Real-World Applications ### Customer Support Automation Online monitoring tracks escalation rates and negative sentiment, helping ensure that support bots answer accurately and maintain a helpful tone without hallucinating company policies. ### Enterprise RAG Search Engines Offline evaluations verify that document retrievers fetch relevant sources, while online evaluations continuously monitor faithfulness and potential false claims. ### Regulated Domain Assistants High-stakes applications can use strict offline CI/CD quality gates alongside real-time online toxicity and PII guardrails. ### Autonomous Task Agents Multi-step agents can rely on tool-use benchmarks to verify function-calling accuracy before release, backed by online latency and error-rate monitoring. ## Important Points for Revision ### Vibe Testing Is Risky Testing casually with arbitrary prompts is non-repeatable and can lead to unexpected production failures. ### Model Evals vs. Application Evals Model evals measure raw model capabilities, while application evals test the full product system including prompts, retrieval, tools, and UI. ### The 8 Core Capabilities Knowledge & Reasoning, Coding, Mathematics, Long Context, Multimodal, Agentic/Tool Use, Safety, and Instruction Following. ### Custom Evals Over Rule of Thumb A smaller model evaluated on a custom dataset can provide better cost-to-performance efficiency than a top-ranked general model. ### Offline Use Cases Release gating, version comparison, and regression testing. ### Production Risks Unanticipated user prompts, emergent or scale failures, and data or concept drift. ## LLM Evaluation Interview Questions Why should an AI engineer run a custom evaluation dataset rather than relying solely on public model benchmark leaderboards? Public benchmarks measure generic model capabilities across academic subjects. They do not reflect an application's specific prompt structures, domain context, or operational requirements. A custom evaluation set allows engineers to evaluate actual task accuracy alongside real-world cost and latency. This can reveal that a smaller, cheaper model may meet business requirements at a fraction of the cost of a top-ranked foundation model. How do Offline Evaluations and Online Evaluations complement each other? Offline and online evaluations address different phases of the software lifecycle. Offline evaluations run before deployment using a static Golden Dataset and ground-truth answer keys to ensure correctness and prevent regressions. Online evaluations run after deployment on live production traffic without relying on predefined answer keys. They monitor operational health, detect latency spikes or data drift, and capture unanticipated edge cases. How can an online evaluation system detect that an application is failing if live user queries do not have ground-truth answer keys? Online evaluations measure normality rather than exact correctness. By logging telemetry over time, the system establishes baseline distributions for metrics such as user sentiment, response length, score distributions, and latency windows. If live production metrics deviate significantly from established baselines, the online system can trigger an anomaly alert for engineering review. What is Stratified Sampling in an online evaluation pipeline, and why is it necessary? Stratified Sampling is the practice of categorizing production conversations and selecting specific sub-groups for detailed evaluation rather than evaluating every interaction. Running secondary evaluation models on 100% of production traffic can be cost-prohibitive. Stratified sampling focuses evaluation budgets on high-risk interactions such as negative user feedback, abrupt drop-offs, billing issues, refunds, or other sensitive topics. ## Quick Revision Summary ## LLM Evaluation Cheat Sheet **Select the Model Thoughtfully:** Use custom evaluation sets alongside core capability benchmarks to find the optimal balance between accuracy, cost, and speed. **Test Rigorously Offline:** Build a curated Golden Dataset to run pre-deployment checks, automate CI/CD release gates, and catch regressions early. **Monitor Continuously Online:** Log structured telemetry, use stratified sampling to compute quality signals, track baseline normality on dashboards, and configure automated alerts. **Close the Loop:** Route production failure cases back into offline test datasets to continuously refine prompts, retrieval pipelines, and model configurations. **Evaluation Lifecycle:** Model Selection → Offline Evaluation → Production Deployment → Online Monitoring → Failure Analysis → Improved Offline Dataset → Next Release. ## Conclusion Building production-ready LLM applications requires much more than testing a few prompts manually. AI systems are probabilistic, dynamic, and exposed to continuously changing real-world inputs. **Model Evals** help engineers understand the underlying capabilities of different models and make informed decisions about accuracy, cost, latency, safety, and deployment architecture. **Offline Application Evals** provide controlled pre-deployment testing, regression detection, version comparison, and automated release gates. **Online Application Evals** provide continuous visibility into real production behavior, helping teams detect unexpected inputs, operational failures, drift, anomalies, and changes in user behavior. The strongest evaluation strategy combines all of these approaches into a continuous feedback loop where production failures become new offline test cases and future releases are evaluated against real-world problems.