Building Reliable LLM Evaluation Pipelines

You can't ship what you can't measure. LLM-powered features are notoriously hard to evaluate because outputs are non-deterministic, correctness is often subjective, and traditional software testing (assert output == expected) doesn't apply. But "it's hard" isn't an excuse for shipping without evaluation. Here's how to build an evaluation pipeline that catches regressions before your users do.

Why Traditional Metrics Fail

BLEU and ROUGE measure n-gram overlap between generated text and reference text. They were designed for machine translation and summarization, not for open-ended generation. Two responses can be semantically identical with zero n-gram overlap:

• Reference: "The server crashed due to a memory leak"
• Generated: "A memory leak caused the application to go down"
• ROUGE-L score: 0.22 (low, despite being correct)

These metrics are useful only when you have highly constrained outputs with known reference answers (e.g., extracting specific fields from a document).

The Three-Layer Evaluation Stack

Production LLM evaluation needs multiple layers, each catching different types of failures:

Layer 1: Deterministic Checks

These are binary pass/fail checks that run in milliseconds. They catch structural failures without any LLM calls.

These checks run in your CI pipeline on every prompt change. If a prompt edit causes the model to start refusing queries or producing malformed JSON, you know immediately.

Layer 2: LLM-as-Judge

Use a strong model to evaluate the output of your application model. This sounds circular, but it works because evaluation is easier than generation — a model that can't write a perfect legal brief can still identify when a brief is missing key arguments.

Response to Evaluate: {response}

Output Format

Return a JSON object: {{ "relevance": {{"score": int, "reason": "..."}}, "accuracy": {{"score": int, "reason": "..."}}, "completeness": {{"score": int, "reason": "..."}}, "clarity": {{"score": int, "reason": "..."}}, "overall_score": float, "critical_issues": ["list of any serious problems"] }}"""

async def llm_judge(query: str, context: str, response: str) -> dict: result = await judge_model.ainvoke([{ "role": "user", "content": JUDGE_PROMPT.format( query=query, context=context, response=response, ), }]) return json.loads(result.content)

2. Anchor with reference answers. Provide the judge with a known-good reference answer and ask it to compare.

3. Validate judge agreement with humans. On 50-100 examples, have both the LLM judge and a human rate the same outputs. If agreement (Cohen's kappa) is below 0.6, your judge prompt needs tuning.

Layer 3: Human Evaluation

Automated evaluation catches most issues, but human review is the ground truth for subjective quality. Make it sustainable with sampling:

Review 2% of production traffic weekly. Focus review time on edge cases: low judge scores, new prompt versions, and query types that historically cause problems.

Prompt Regression Testing in CI

Every prompt change should be tested against an evaluation dataset before deployment:

Run deterministic checks on every PR. Run LLM-as-judge evaluations nightly or on prompt-related PRs (they're slower and cost money).

Building the Evaluation Dataset

Your eval dataset is a living artifact. Start small and grow it:

1. Seed with 20-30 representative queries covering your main use cases
2. Add failure cases as you find them in production (user complaints, bad judge scores)
3. Include adversarial cases — queries designed to trigger hallucination, refusal, or format breakage
4. Version the dataset in git alongside your prompts

Aim for 100+ examples within the first month. Each production bug that's caught by users but not by your eval set should result in a new test case.

Monitoring in Production

Evaluation doesn't stop at deployment:

• Track judge scores over time. A gradual decline indicates model drift or changing query patterns.
• Monitor deterministic check failure rates. A spike in JSON parse failures after a deploy means your prompt broke.
• Log all inputs and outputs. You need this data for debugging, eval dataset expansion, and fine-tuning.
• Set up alerts on judge score drops (>0.5 point drop from baseline) and deterministic check failure rate (>5%).

The goal isn't perfect evaluation — it's catching the regressions that matter before users report them. Start with deterministic checks (free, fast), add LLM-as-judge (moderate cost, good coverage), and sample for human review (expensive, ground truth). Each layer catches issues the others miss.