RAG Evaluation with RAGAs: Catch Regressions Early

RAG applications fail quietly. The system returns an answer; it looks reasonable; nobody notices it is wrong until a user complains or revenue drops. In 2026, the teams that ship reliable RAG have moved past eyeballing output and build evaluation pipelines that run on every change. This article walks through what to measure, the two main evaluation frameworks (RAGAs and custom metrics), and how to wire evaluation into your CI so a bad prompt or model change never ships.

What to measure

A RAG pipeline has two stages: retrieval (did we find the right documents?) and generation (did we produce the right answer given those documents?). Both need separate metrics.

Retrieval metrics:

• Context Precision: of the chunks we retrieved, how many are actually relevant? Low precision means we are stuffing the LLM with noise.
• Context Recall: of the information needed to answer the question, how much is present in the retrieved chunks? Low recall means we are missing key context.
• Precision@k, Recall@k, MRR: standard information retrieval metrics for ranking quality.

Generation metrics:

• Faithfulness: does the answer only use information from the retrieved context, or does it hallucinate?
• Answer Relevance: does the answer actually address what the user asked?
• Answer Correctness: compared to a known-good answer, how close is this one?

Teams that only measure generation miss retrieval problems. Teams that only measure retrieval miss generation problems. Measure both.

RAGAs: the Python library most teams use

RAGAs (Retrieval Augmented Generation Assessment) is the standard framework. Install:

pip install ragas datasets

Build an evaluation dataset. Each row has: a question, the chunks your retriever returned, the answer your pipeline generated, and the ground-truth answer:

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

samples = [
    {
        "question": "What is our refund policy?",
        "contexts": ["Refunds are issued within 30 days...", "..."],
        "answer": "You can get a refund within 30 days of purchase.",
        "ground_truth": "Full refunds are available within 30 days of purchase.",
    },
    # ... 50 to 500 samples
]

dataset = Dataset.from_list(samples)
result = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)
# {'faithfulness': 0.87, 'answer_relevancy': 0.91, 'context_precision': 0.76, 'context_recall': 0.82}

RAGAs scores each metric from 0 to 1. The LLM-as-judge calls (RAGAs uses GPT-4o by default to compute these metrics) run through your OpenAI API key. For a 50-example dataset, expect around $0.20 to $0.50 in LLM evaluation cost per run.

Custom metrics for your domain

RAGAs covers general RAG. Many production applications have domain-specific concerns that need custom metrics. Examples:

• Citation accuracy: when the answer cites a specific chunk ID, is the citation correct? Write a function that parses citations and checks against the retrieved chunks.
• Numerical precision: for financial or scientific RAG, parse numbers out of the answer and check against ground-truth to a tolerance.
• Format compliance: does the answer follow a required structure (JSON schema, Markdown with specific headings)? Parse and validate.
• Latency per step: not just end-to-end latency, but retrieval time vs generation time separately.
• Cost per answer: track token usage per response and alert if a change causes it to spike.

Custom metrics do not need an LLM. Pure Python functions that compare outputs against expectations are faster, cheaper, and more deterministic.

LLM-as-judge: use carefully

Many RAG metrics (faithfulness, relevance, correctness) are hard to measure without an LLM scoring them. The pattern is called “LLM-as-judge”: run GPT-4o or Claude Opus 5.5 to grade your pipeline’s output against a rubric.

Risks and gotchas:

• Judge drift: the judge model changes behavior as OpenAI updates it. Pin the judge model version and rerun your baseline when you change it.
• Judge bias: GPT-4o tends to rate GPT-generated outputs higher than Claude-generated outputs on the same task. Use multiple judges and average.
• Position bias: in pairwise comparisons, judges often prefer the first option shown. Randomize order or test both orderings.
• Cost: judge calls use frontier models (expensive). For a 500-example dataset with 4 metrics each, expect $5 to $15 per full evaluation.

Use LLM-as-judge for subjective quality (relevance, helpfulness, tone) but prefer deterministic checks for anything measurable in code.

Wire evaluation into CI

The real payoff of evaluation is catching regressions before they ship. Add a GitHub Actions workflow that runs your evaluation on every PR touching the RAG pipeline, embedding model, prompts, or chunking logic:

# .github/workflows/rag-eval.yml
name: RAG Evaluation
on:
  pull_request:
    paths:
      - 'rag/**'
      - 'prompts/**'
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
      - run: pip install -r requirements.txt
      - run: python eval.py --dataset tests/golden.json
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Script the evaluation to fail if any metric regresses by more than a threshold (say 2 percentage points below the baseline). Post results as a PR comment so the reviewer sees exactly which metrics moved.

Build the dataset

Evaluation is only as good as the dataset. Three ways to build one:

• Real user queries: pull the top 50 to 100 real questions from your production logs. Have a human (or an LLM) write the correct answer. Expensive to build, high-signal.
• Synthetic generation: feed your knowledge base to GPT-4o and ask it to generate question-answer pairs. Faster to build (RAGAs has a built-in TestsetGenerator), but lower-signal than real queries.
• Adversarial: write edge-case questions designed to break the pipeline (ambiguous queries, questions outside the knowledge base, trick questions). Smaller set but catches failures nothing else does.

Most production teams combine all three: 70% real user queries, 20% synthetic for coverage, 10% adversarial for safety.

Frequently Asked Questions

Angel Jude Suarez

Full-Stack Developer at PIES IT Solution

Focuses on Python development, machine learning, and AI integration. Has built production AI systems including OpenAI Whisper integration for medical transcription and GPT-4o-powered diagnosis assistance. Strong background in pandas, scikit-learn, and TensorFlow.

Expertise: Python · PHP · Java · VB.NET · ASP.NET · Machine Learning · AI Integration · OpenCV · Django · CodeIgniter  · View all posts by Angel Jude Suarez →