Setting Up Automated LLM Evaluation Pipelines for Hallucination Detection Using Ragas

What Ragas actually measures

Ragas scores a RAG system's outputs against four core reference-free-ish metrics: faithfulness (does the generated answer's claims actually follow from the retrieved context, i.e. is it hallucinating), answer relevancy (does the answer actually address the question asked), context precision (is the retrieved context ranked with the useful chunks first), and context recall (did retrieval pull in the information actually needed, measured against a ground-truth answer). This tutorial builds a golden test set, a scoring script, and a CI gate that fails a pull request if faithfulness or context recall regresses beyond a threshold.

1. Technical architecture

┌─────────────────────┐   ┌───────────────────────┐   ┌──────────────────────┐
│ Golden test set          │──▶│ Eval runner: replay        │──▶│ RAG system under test    │
│ (question, ground_truth,   │   │ each question through the   │   │ (your existing pipeline)  │
│  reference contexts)         │   │ live RAG pipeline             │   └──────────┬───────────┘
└─────────────────────┘   └───────────────────────┘                          ▼
                                                                    ┌──────────────────────┐
                                                                    │ (question, answer,        │
                                                                    │  contexts, ground_truth)   │
                                                                    │ dataset for Ragas            │
                                                                    └──────────┬───────────┘
                                                                                 ▼
                                                                    ┌──────────────────────┐
                                                                    │ Ragas metrics: faithfulness,│
                                                                    │ answer_relevancy,           │
                                                                    │ context_precision/recall     │
                                                                    └──────────┬───────────┘
                                                                                 ▼
                                                        ┌──────────────────────────────────────┐
                                                        │ Threshold gate → JSON report + exit code │
                                                        │ (wired into GitHub Actions CI)             │
                                                        └──────────────────────────────────────┘

2. Complete code implementation

2.1 Golden test set format

# eval/golden_dataset.py
import json
from dataclasses import dataclass, asdict


@dataclass
class GoldenExample:
    question: str
    ground_truth: str
    reference_contexts: list[str]  # what SHOULD be retrieved, for context_recall


GOLDEN_SET_PATH = "eval/golden_set.jsonl"


def load_golden_set(path: str = GOLDEN_SET_PATH) -> list[GoldenExample]:
    examples = []
    with open(path, "r", encoding="utf-8") as fh:
        for line_num, line in enumerate(fh, start=1):
            if not line.strip():
                continue
            try:
                row = json.loads(line)
                examples.append(GoldenExample(**row))
            except (json.JSONDecodeError, TypeError) as exc:
                raise ValueError(f"Malformed golden set entry at line {line_num}: {exc}") from exc
    if not examples:
        raise ValueError(f"Golden set at {path} is empty")
    return examples


def append_example(example: GoldenExample, path: str = GOLDEN_SET_PATH):
    with open(path, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(asdict(example)) + "\n")

2.2 Eval runner

# eval/run_eval.py
import sys
import json
import logging
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

from golden_dataset import load_golden_set
from rag_pipeline_adapter import query_rag_system  # your project's RAG entrypoint

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("llm-eval")

THRESHOLDS = {
    "faithfulness": 0.85,
    "answer_relevancy": 0.80,
    "context_precision": 0.75,
    "context_recall": 0.75,
}


def build_eval_dataset(golden_examples) -> Dataset:
    rows = {"question": [], "answer": [], "contexts": [], "ground_truth": []}
    failures = []

    for ex in golden_examples:
        try:
            result = query_rag_system(ex.question)
        except Exception as exc:
            failures.append({"question": ex.question, "error": str(exc)})
            continue

        rows["question"].append(ex.question)
        rows["answer"].append(result["answer"])
        rows["contexts"].append(result["contexts"])  # list[str] of retrieved chunks
        rows["ground_truth"].append(ex.ground_truth)

    if failures:
        logger.warning("%d/%d golden questions failed to get a response from the RAG system",
                        len(failures), len(golden_examples))
        for f in failures:
            logger.warning("  - %s: %s", f["question"][:80], f["error"])

    if not rows["question"]:
        raise RuntimeError("Every golden question failed against the RAG system — aborting eval")

    return Dataset.from_dict(rows), failures


def run():
    golden_examples = load_golden_set()
    dataset, failures = build_eval_dataset(golden_examples)

    result = evaluate(
        dataset,
        metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    )
    scores = result.to_pandas().mean(numeric_only=True).to_dict()

    report = {
        "scores": scores,
        "thresholds": THRESHOLDS,
        "n_evaluated": len(dataset),
        "n_failed_to_run": len(failures),
        "passed": True,
    }

    regressions = []
    for metric, threshold in THRESHOLDS.items():
        score = scores.get(metric)
        if score is None:
            regressions.append(f"{metric}: metric did not compute (missing/NaN)")
            continue
        if score < threshold:
            regressions.append(f"{metric}: {score:.3f} < required {threshold}")

    if regressions:
        report["passed"] = False
        report["regressions"] = regressions

    with open("eval/eval_report.json", "w") as fh:
        json.dump(report, fh, indent=2)

    print(json.dumps(report, indent=2))
    return 0 if report["passed"] else 1


if __name__ == "__main__":
    sys.exit(run())

2.3 GitHub Actions CI integration

# .github/workflows/llm-eval.yml
name: LLM Evaluation Gate

on:
  pull_request:
    paths:
      - "rag/**"
      - "eval/golden_set.jsonl"

jobs:
  ragas-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install ragas datasets openai --break-system-packages

      - name: Run Ragas evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python eval/run_eval.py

      - name: Upload eval report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: ragas-eval-report
          path: eval/eval_report.json

3. Step-by-step configuration guide

1. Install dependencies:

   pip install ragas datasets openai --break-system-packages
   

2. Build the golden set — hand-curate 30-100 representative (question, ground_truth, reference_contexts) triples that reflect real user queries against your RAG system; quality here matters more than quantity, since Ragas scores are only as meaningful as the golden answers they're compared against.
3. Implement rag_pipeline_adapter.query_rag_system(question) to wrap your actual RAG pipeline (e.g., the one from the companion LangChain/Pinecone tutorial) and return {"answer": str, "contexts": list[str]}.
4. Set OPENAI_API_KEY — Ragas's default metrics use an LLM judge (configurable to any LangChain-compatible chat model) to score faithfulness and relevancy.
5. Run locally first: python eval/run_eval.py and inspect eval/eval_report.json.
6. Calibrate thresholds — run the eval against your current system to get a baseline, then set THRESHOLDS slightly below that baseline so the gate catches real regressions without being so strict it blocks unrelated PRs on noise.
7. Add the GitHub Actions workflow from §2.3 and store OPENAI_API_KEY as a repo secret; the workflow triggers on PRs touching rag/** or the golden set itself.

4. Error handling and edge cases

• Individual golden questions failing against the RAG system (network error, empty retrieval, exception in your pipeline) are caught per-question in build_eval_dataset and logged, rather than crashing the entire evaluation run — the report still includes n_failed_to_run so silent degradation is visible.
• All questions failing raises a hard RuntimeError rather than silently reporting misleading scores on zero real examples — a fully broken pipeline should fail the CI job loudly, not pass with a hollow "0 evaluated, all thresholds met" report.
• Missing or NaN metric scores — Ragas can return NaN for a metric if the judge LLM's output couldn't be parsed for a given row; run() treats a missing/NaN metric as a regression by default rather than silently ignoring it, since a broken metric hiding a real regression is worse than a noisy false failure.
• Malformed golden set entries raise a descriptive ValueError naming the exact line number in load_golden_set, so a bad JSON line in a hand-edited file is easy to locate and fix.
• LLM judge cost and flakiness — Ragas metrics call an LLM for every row/metric combination, which is both a cost and a source of run-to-run variance; keep the golden set to a curated, stable size (not thousands of rows) and consider averaging over 2 runs before treating a borderline threshold miss as a real regression, since small run-to-run judge variance is expected.
• Golden set drift — when your product's expected answers legitimately change (a policy update, a new feature), the golden set itself needs updating; treat golden set changes as a reviewed PR just like code, not an automatic sync, so the eval doesn't silently start grading against stale expectations.
• CI secret misconfiguration — if OPENAI_API_KEY is missing in the CI environment, the workflow should fail fast with a clear authentication error on the first Ragas call rather than hanging; this is a good candidate for a preflight check step that pings the API before running the full golden set.