Evaluating models

You trained an adapterAdapter A small set of extra weights trained and attached to a model that otherwise stays unchanged, so its behavior changes without retraining the whole model., the trainingTraining Adjusting model weights from data so its predictions improve. numbers improved, and a couple of answers look good. Did the model get better? You can't tell. Training numbers only show how well it fits examples it has already seen, and a few hand-picked answers are not evidence. Unlike a software test, a model's output varies, and "correct" is often a matter of degree.

Evaluation answers three questions: does it work on cases it never saw, is it better than what we had, and is it good enough to ship? This lesson covers what an evaluation needs, how test data leaks, how to pick a baselineBaseline The reference result a new model or change must beat to count as an improvement. and a metric, how to turn the result into a rule that blocks a bad release, and how to evaluate a RAGRAG Retrieval-augmented generation: finding relevant passages in your documents and adding them to the prompt before the model answers. system.

What an evaluation needs

Without evaluation, every change is a bet. A useful evaluation has four parts:

1. Data the model did not learn from.
2. A reference point, or baseline, to compare against.
3. A metric that captures what matters for the task.
4. A threshold that turns the number into a decision.

Test data and how it leaks

A student who saw the exam in advance proves nothing with a perfect score. The same holds for models. Before training, part of the data is set aside as the held-out set, and the model never learns from it; scoring on training examples measures memory, not ability (Machine learning in one lesson).

The exam can also leak in quieter ways, called data leakage:

• Near-duplicates: the same ticket, with a different date, in both the training and the test data.
• Shared templates: if every answer follows one fixed sentence, the test only checks that sentence.
• Contamination: a public benchmark may have been in the model's original training data.
• File order: if rows are sorted by label and you set aside the last 20%, the test may hold a single category.

Baselines and metrics

A number on its own says nothing: 0.33 can be excellent or terrible depending on what you get without your change. Always compare against the base modelBase model A model after its first, general training, which only continues text and has not yet been tuned to follow instructions. with the same instructions on the same test set and, if there is one, against the version in production. The metric then depends on the task:

LLMLLM A large language model: a neural network trained to predict the next token, then tuned to follow instructions.-as-judge means asking a model to grade answers. It scales well, but judges favor the first option shown, longer answers and their own model family's style. Counter this with a concrete rubric, swapped order, a check against a sample graded by people, and never letting one model both answer and judge.

The eval gate

An eval gateEval gate An automatic check that blocks a model or change from going live unless it passes an evaluation. turns evaluation into a barrier: if a candidate fails the rule, it does not ship. It is not a report someone can ignore, but a pipeline step that fails.

Evaluating a RAG system

A RAG system searches your documents, then answers from what it found (Embeddings and RAG). Measure the two steps separately; they fail for different reasons.

• Search: with a set of questions and the document each should find, check whether the right one appears among the top few results (recall@k) and how high it ranks (MRR).
• Answer: check faithfulness (is every claim supported by what was found?) and relevance (does it answer the question?). Ragas popularized these, usually graded by an LLM judge.

All of the above is offline evaluation: fixed data before release, cheap and repeatable, but limited to the cases you thought of. Online evaluation watches real use: A/B tests, user feedback, how often people retry or ask for a human. Failures found in production should become new offline test cases. For agentsAgent A system in which a model, in a loop, chooses the next action until it decides it is done or hits a stop condition., see Testing and evals for agents.

Putting it into practice

• Set the rule before training: which metric, against which baseline, with which threshold. Put it in code, not in a wiki.
• Run a negative control. If a dataset with no pattern passes your gate, the gate protects nothing.
• Give the test cases an owner close to the business, or the gate ends up measuring an old problem.
• Treat the threshold as a risk decision. Raising it blocks more good candidates; lowering it lets more bad ones through.

Go deeper

Adapta's own documentation explains its evalEval A measurement of how well the whole system behaves on realistic cases, including the steps it took along the way, scored with imperfect checks and read with statistics. gate in detail and records real results:

• Learning the system has a section on the evaluation gate: the held-out rows, how the score is computed and why there are two ways to pass.
• Fine-tuning use cases lists gate results for several tasks next to the base model's score, including a negative control that is blocked, and warns that a near-perfect score is not proof of quality.
• Fine-tuning, proven reads a real gate result line by line, from the score to the comparison with the base model.

Key takeaways

• Evaluate on data the model never saw, and watch for quiet leaks such as file order and shared templates.
• Compare against a baseline on the same test set, with a metric chosen for the task.
• An eval gate is a rule in code that blocks release, and a negative control proves it blocks.
• In RAG, measure search and answer quality separately.

Check yourself