How to Evaluate AI Output (When You Can Not Check Every Answer)

Your AI assistant aced the demo. The CEO loved it. The pilot users gave it a thumbs up. You ship it to production, and now it handles 10,000 requests per day. Here is the question nobody asked: how do you actually know it is still working?

In traditional software, this is a solved problem. You write tests, you assert expected outputs, and you get a green or red light. With LLMs, there is no assertEquals. The output is probabilistic. Two identical inputs can produce different outputs. And "correct" often depends on context, tone, and nuance that no simple rule can capture. This is the evaluation problem, and it is the single biggest gap we see in production AI systems.

Why LLM evaluation is fundamentally different

Traditional software testing assumes deterministic behavior. You know the expected output for a given input, and any deviation is a bug. LLMs break every one of those assumptions. The same question asked twice may yield two different phrasings of the same correct answer, or occasionally, two subtly different answers where one is better than the other.

This means you cannot rely on exact matching. You cannot rely on unit tests in the traditional sense. And you absolutely cannot rely on "it seemed fine when I tried it a few times." That approach works for demos. It does not work when your system is answering customer questions at scale, generating reports for compliance teams, or making recommendations that influence business decisions.

Three evaluation strategies we use in production

After building evaluation systems for multiple production AI deployments, we have settled on a layered approach. No single method is sufficient on its own, but together they provide reliable coverage.

1. Automated metrics

These run on every single response and catch structural and factual problems immediately. We typically measure:

• Format validation: Does the output follow the expected structure? JSON that parses, required fields present, length within bounds.
• Relevance scoring: Using embeddings to measure semantic similarity between the question and the answer. Answers that drift off topic score low.
• Factual consistency checks: Cross referencing generated claims against the source documents. If the system says "the policy allows 30 days" but the source document says 14, that is a flaggable inconsistency.

Automated metrics are fast, cheap, and catch roughly 60 to 70% of quality issues. They are your first line of defense.

2. LLM as judge

This is the most powerful and most dangerous evaluation method. You use a stronger model (or the same model with a structured rubric) to grade the output of your production system. It scales well and can assess nuanced qualities like helpfulness and tone.

The catch: reliability. According to recent industry research, 93% of teams using LLM as judge report significant reliability problems. The judge model can be biased toward verbose answers, may disagree with itself on repeated evaluations, and can miss subtle factual errors.

We mitigate this by using explicit scoring rubrics, running multiple judge passes, and always calibrating the judge against human ratings on a known test set.

def evaluate_response(question, answer, context):
    """Use an LLM judge to score a response on key dimensions."""
    rubric = {
        "relevance": "Does the answer address the specific question asked? (1-5)",
        "faithfulness": "Is every claim supported by the provided context? (1-5)",
        "completeness": "Does the answer cover all key aspects? (1-5)",
        "harmfulness": "Does the answer contain misleading or harmful content? (1-5, 5=safe)",
    }

    scores = judge_model.evaluate(
        question=question,
        answer=answer,
        source_context=context,
        rubric=rubric,
    )

    # Flag for human review if any dimension scores below 3
    if any(score < 3 for score in scores.values()):
        flag_for_human_review(question, answer, scores)

    return scores

3. Human sampling

Automated metrics and LLM judges handle volume. Human review handles ground truth. We recommend reviewing a random 2 to 5% sample of production responses weekly. This serves two purposes: it catches issues that automated systems miss, and it provides the calibration data you need to keep your automated evaluations honest.

Track your human review scores over time. A slow downward trend often signals model drift, context degradation, or changes in user query patterns that your system was not designed for.

The eval metrics that actually matter

Not all metrics are created equal. After running evaluations across many client systems, these four consistently prove most valuable:

• Answer relevance: Does the response actually address what the user asked? This sounds obvious, but retrieval systems frequently surface related but incorrect context, leading to technically accurate answers to the wrong question.
• Faithfulness to context: Is the answer grounded in the provided source material? This is the hallucination detector. Any claim not traceable to a source document is a red flag.
• Completeness: Did the response cover all the important points? Partial answers are one of the most common failure modes, especially when context windows are tight.
• Harmfulness: Does the response contain anything misleading, biased, or potentially damaging? This is non negotiable for customer facing systems.

A real example: catching degradation before customers do

One of our clients runs a RAG system that answers employee questions about company policies. After a routine document update, the retrieval pipeline started pulling in outdated document chunks alongside the new ones. The answers were not obviously wrong. They were a mix of old and new policy information, blended together in plausible sounding paragraphs.

Because we had automated faithfulness scoring running on every response, we detected a 15% drop in consistency scores within 48 hours. We traced the issue to the document ingestion pipeline, fixed the chunking logic, and reindexed. Without automated evals, this client would have discovered the problem weeks later through confused employees filing HR tickets.

Where to start

If you are running an AI system in production without structured evaluation, start small. Pick one of the three strategies above, implement it this week, and iterate. The companies that treat evaluation as a core engineering discipline, rather than an afterthought, are the ones whose AI systems actually improve over time.

We help teams design and implement evaluation pipelines that catch problems before users do. If you are scaling an AI system and want confidence in its output quality, let's talk.