RAG Evaluation Framework 2026: RAGAS, TruLens, Phoenix

RAG systems degrade silently. A new chunking strategy drops top-5 recall by 8% and nobody notices because the generated answers still sound good. Three months later, confidence in the system is quietly dead. Evaluation is the discipline that prevents this. In 2026, three frameworks dominate: RAGAS for metrics-first evaluation, TruLens for observability-focused evaluation, and Phoenix for the LangChain and LlamaIndex-adjacent crowd.

This guide covers what to measure, how each framework measures it, and the small test set that catches most regressions before they ship.

What RAG evaluation actually measures

Four metrics worth tracking per query, from most important to least:

1. Faithfulness. Does the generated answer stay grounded in the retrieved chunks? Fabrication is the worst RAG failure mode.
2. Answer relevance. Does the answer address the actual question, or does it ramble into adjacent topics?
3. Context precision. Of the retrieved chunks, how many are actually relevant? Precision at rank matters because the top chunks are what the generator sees.
4. Context recall. Of the information needed to answer, how much is in the retrieved chunks? Low recall means the retriever missed something.

Everything else (fluency, format compliance, cost, latency) matters but is downstream of these four.

RAGAS: metrics-first evaluation

RAGAS (Retrieval-Augmented Generation Assessment) ships the four metrics above as drop-in functions. Each metric runs via LLM-as-judge calls against a reference answer and the retrieved context.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

eval_data = Dataset.from_dict({
    "question": [...],
    "answer": [...],              # what your RAG system produced
    "contexts": [...],            # retrieved chunks per query
    "ground_truth": [...],        # labeled correct answer
})

result = evaluate(
    eval_data,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)

The LLM-as-judge model is configurable. Claude Opus 5.5 or GPT-5 produce the most reliable scores. Smaller models (Haiku, GPT-5-mini) are 5-10x cheaper but introduce noise on edge cases. For CI/CD gates, use the big model on a smaller test set; for exploratory analysis, use the small model on a bigger set.

TruLens: observability-first evaluation

TruLens wraps your RAG app and records every retrieval and generation. Metrics run against the recorded trace, so evaluation does not require a separate pipeline.

from trulens.core import TruSession
from trulens.apps.llamaindex import TruLlama
from trulens.providers.openai import OpenAI

session = TruSession()
provider = OpenAI(model_engine="gpt-5")

f_groundedness = Feedback(provider.groundedness_measure_with_cot_reasons).on_input().on_output()
f_relevance = Feedback(provider.relevance).on_input().on_output()
f_context_relevance = Feedback(provider.context_relevance_with_cot_reasons).on_input().on_output()

tru_app = TruLlama(
    my_rag_app,
    app_name="MyRAG",
    feedbacks=[f_groundedness, f_relevance, f_context_relevance],
)

with tru_app as recording:
    response = my_rag_app.query("example question")

session.get_leaderboard()

TruLens shines when evaluation needs to happen in production, not just in a test harness. It records every real user query, scores it, and surfaces the lowest-scoring interactions in a dashboard. Teams use that queue as the input to improvement work.

Phoenix: tracing + evaluation

Arize Phoenix is the open-source observability + evaluation tool for LLM applications. OpenTelemetry-based tracing captures every call in the pipeline; evaluators score the trace.

import phoenix as px
from phoenix.evals import HallucinationEvaluator, QAEvaluator, run_evals

px.launch_app()
# Instrument your RAG pipeline (OpenInference auto-instrumentors for LangChain,
# LlamaIndex, Anthropic SDK, OpenAI SDK all available)

# Later: evaluate the recorded traces
spans_df = px.Client().get_spans_dataframe()
evaluators = [
    HallucinationEvaluator(model),
    QAEvaluator(model),
]
eval_results = run_evals(
    dataframe=spans_df,
    evaluators=evaluators,
    provide_explanation=True,
)

Phoenix is particularly strong at drilling from a quality regression down to the specific prompt that caused it. Dashboards show per-feature-flag, per-model-version, per-chunking-strategy slices of quality metrics. For teams that treat RAG quality as a product-ops concern, Phoenix is the one to pick.

Building a small, useful test set

You do not need thousands of labeled examples. A good test set is 50-150 queries that cover:

• Common. The questions users actually ask most. Pull these from a week of production logs if available.
• Edge cases. Specific identifiers, odd phrasing, acronyms, questions that span multiple documents.
• Failures. Historical cases where the system got it wrong. Each one earns a permanent slot in the test set.
• Known-answerless. Questions whose answer is NOT in the corpus. The right behavior is “I don’t know”; the test verifies the model respects the fallback.

Each query carries the ground-truth answer and the ground-truth chunk IDs that should be retrieved. Labeling 100 queries takes about 2-4 hours of focused work from someone who knows the domain. It is one of the highest-ROI investments in a RAG system.

Automated evaluation gates in CI

Once the test set exists, wire it into CI. Every pull request that touches the RAG pipeline (chunking, retrieval, prompts, models) runs the test set and reports deltas.

# eval_ci.py
import json
from ragas import evaluate
from datasets import Dataset

TEST_SET = json.load(open("eval/test_set.json"))

def run_pipeline(query):
    retrieved = my_rag.retrieve(query)
    answer = my_rag.generate(query, retrieved)
    return retrieved, answer

eval_data = Dataset.from_dict({
    "question": [t["question"] for t in TEST_SET],
    "answer": [run_pipeline(t["question"])[1] for t in TEST_SET],
    "contexts": [run_pipeline(t["question"])[0] for t in TEST_SET],
    "ground_truth": [t["answer"] for t in TEST_SET],
})

result = evaluate(eval_data, metrics=[faithfulness, answer_relevancy, context_precision, context_recall])

baseline = json.load(open("eval/baseline.json"))
for metric, value in result.items():
    delta = value - baseline[metric]
    print(f"{metric}: {value:.3f} (delta: {delta:+.3f})")
    if delta < -0.03:
        exit(1)

A 3-point drop on any metric fails the build. Numbers are tunable; the discipline matters more than the exact threshold. Teams that run this gate catch regressions in the PR, not three months later when a user escalates.

Diagnosing where quality drops

When a metric moves in the wrong direction, the four metrics localize the problem:

• Context recall drops: the retriever misses relevant chunks. Fix is in chunking, embedding, or reranking.
• Context precision drops: the retriever returns irrelevant chunks. Fix is reranking or query rewriting.
• Faithfulness drops: the generator makes claims not in the retrieved chunks. Fix is in the system prompt or the model.
• Answer relevance drops: the generator does not address the question. Fix is prompt wording or model choice.

This localization alone is worth the setup cost. Without it, every quality problem looks like “the RAG is bad” and the team tunes random knobs until something changes.

Human-in-the-loop evaluation for high-stakes systems

LLM-as-judge is 85-95% accurate on faithfulness and relevance. For low-stakes systems (support bots, internal search), that is fine. For high-stakes (medical, legal, finance), add human review on 5-10% of production queries.

The pattern: TruLens or Phoenix records every query, LLM-judge scores run automatically, lowest-scoring 5% get flagged for human review. Human corrections flow back into the test set and into fine-tuning data for the LLM judge.

Over 6-12 months this loop produces a judge model that approaches human agreement rate on your specific domain, which is the ceiling of what evaluation can be.

Which framework to pick in 2026

• RAGAS: pure metrics. Best when you want numbers in a notebook or a CI script. Minimal footprint.
• TruLens: observability + evaluation. Best when evaluation happens on production traffic and you want a dashboard with per-query scores.
• Phoenix: tracing + evaluation + explainability. Best when the team cares about debugging specific trace spans, not just aggregate numbers.

Many teams run RAGAS in CI and TruLens or Phoenix in production. The two serve different stages of the lifecycle and are not mutually exclusive.

Frequently asked questions

Measure before you optimize. The four-metric framework catches most regressions, localizes most problems, and is cheap enough to run on every PR. Teams that evaluate ship better RAG systems; teams that eyeball retrievals rediscover the same bugs every quarter.

Joken E. Villanueva

Founder & Lead Developer at PIES IT Solution

Founder of PIES Information Technology Solutions, a software company building production-grade applications for institutions across the Philippines. Over 8 years of hands-on full-stack development experience, currently leading the development of ClinicAI, an AI-powered clinic management platform.

Expertise: PHP · MySQL · JavaScript · AI Integration · SaaS Architecture · VB.NET · Database Design · Capstone Documentation · Java  · View all posts by Joken E. Villanueva →