What it means to evaluate an AI system

Evaluation begins with a decision

Before selecting cases or metrics, identify the decision. Comparing alternatives, authorizing limited use, and expanding an agent's autonomy are different decisions. Their tests may be related without being interchangeable.

A comparison looks for meaningful differences between options. An acceptance check verifies minimum conditions. Reviewing a change also asks which capabilities have been preserved and where regressions have appeared. Confusing these purposes can produce a precise report that leaves the actual question unanswered.

The intended users of the evidence should be clear too. Engineers may need to locate a failure's cause, while operations needs to understand the remaining review workload. One test can inform both, provided its findings are presented in a form each can use.

Criteria should be formulated before inspecting the outputs being considered for approval. Defining them afterward risks fitting the judgment to the behavior available. A criterion may legitimately change, but the change needs to remain visible and have a reason beyond improving the score.

A good evaluation does not require everyone to agree from the beginning. It may reveal disagreement about what counts as useful. Resolving that disagreement is part of defining the system; hiding it behind an average merely postpones the decision.

The outcome does not explain the whole path

An acceptable output may have been produced in a way the organization does not permit. The system might reach its goal using unauthorized information, skipping review, or consuming resources beyond the agreed limit.

Evaluation must therefore cover relevant constraints as well as the outcome. This does not mean judging every step in equal detail. It means identifying the conditions that must hold even when the final output is correct.

For agents using tools, distinguishing claims from events is also important. Anthropic's explanation of agent evaluations separates the execution record from the final state left in the environment. That distinction prevents the agent's account from becoming sufficient proof of the outcome.

The evidence needed to check this must be included in the test design. Retaining only the final message may make it impossible to determine whether a permission was respected or an operation remained pending. Evaluation cannot recover evidence that was never recorded.

Recording more does not automatically improve understanding. Traces need a relationship to the task and the criteria being checked. A large collection of events without operation identity or interpretation can produce volume without a basis for judgment.

Outcome, path, and effects provide complementary perspectives. Evaluation should determine which perspective each commitment requires and prevent a favorable result in one dimension from concealing a violation in another.

Test cases represent part of the world

A test collection bounds what has been observed. Its value depends on its relationship to the intended use, not simply the number of examples it contains.

Common cases reveal ordinary behavior. Edge cases examine boundaries. Adversarial situations test constraints that well-prepared inputs may never exercise. No group automatically replaces the others.

Selection must preserve differences that matter to the work. If documents vary in quality, requests are ambiguous, or sources conflict, testing only clean inputs leaves part of the service unexamined. Testing only extreme situations and presenting the result as everyday performance would be equally misleading.

A sample intended to estimate typical behavior should be distinguished from a collection designed to uncover failures. Both are useful but answer different questions. A deliberately difficult set cannot, on its own, establish how often those problems occur in production.

Cases need sufficiently clear expected results or acceptance criteria. Some tasks allow several valid solutions. Comparing them with a single reference sentence can penalize correct answers expressed differently.

Coverage always has limits. Reports should explain which inputs, tools, and conditions were included and which remain untested. That boundary makes findings usable and prevents “passed the tests” from being interpreted as a general guarantee. It also gives the next evaluation a concrete starting point.

Evaluators need evaluation too

Some conditions can be checked directly: a required field exists, a calculation follows a rule, or an expected state appears. Others require judgment about clarity, relevance, or the quality of a proposal.

The evaluation mechanism should match the criterion. A precise automatic check may be better than a broad opinion for verifying a specific restriction. Literal matching may be inadequate for judging an explanation with several correct formulations.

Another model can help assess open-ended criteria, but it introduces a component whose behavior also needs examination. Detailed instructions and numerical scores do not make its judgment objective.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena examined limitations including position and verbosity biases in the evaluators studied. It does not establish that every current evaluator fails in the same way. It does provide a documented reason to examine how the evaluator behaves in the actual context.

Human review is not infallible either. Two people may interpret a rubric differently or have unequal information. Making disagreement visible, clarifying criteria, and retaining examples of difficult decisions can improve the instrument.

The evaluator is part of the measurement system. Changing it can change the score even when the evaluated application has not improved. Its version and criteria therefore belong with the findings, particularly when successive releases are being compared. Without them, a change in judgment can be mistaken for a change in capability.

Comparing changes requires a reference

An improvement should be traceable to a meaningful difference. If the model, instructions, sources, and test cases all change together, explaining the observed result becomes difficult.

Isolating every variable is not always possible, but documenting changes is. That record distinguishes a controlled comparison from an evaluation of a new solution as a whole. Both can be valid if their scope is interpreted correctly.

Repetition matters when results vary. One success establishes that the task was resolved in that execution. It does not establish how consistently it will be resolved or how failures are distributed.

The number of repetitions should reflect the decision and the cost of a mistaken judgment. No universal count makes every test sufficient. Evidence adequate for continuing exploration may remain inadequate for expanding an operation with substantial consequences.

Cases the system already handled should be retained and checked after meaningful changes. If a new version improves one group while worsening another, the decision needs visibility of both. An aggregate average can hide the regression most important to the service.

The reference set should not become the only objective either. Repeated adjustment against the same examples can improve that collection without demonstrating equivalent improvement on new situations. Holding cases back for later checks and renewing coverage helps preserve an honest relationship between test performance and use.

Findings need acceptance criteria

An evaluation report can describe behavior without deciding whether it is acceptable. That decision requires relating the findings to the intended use and the consequences of errors.

Some criteria permit tradeoffs. Better quality may justify additional delay. Others represent limits that should not dissolve into an average score. Good writing does not compensate for an action that exceeds agreed permissions.

Pass conditions should therefore be separated from dimensions assessed on a scale. The team can compare cost, time, and usefulness without turning a mandatory restriction into a small weight in a sum.

Evidence may support a narrower scope than initially planned. A system might be useful for preparing work without being ready to execute it without review. Accepting that limited role is a different decision from declaring failure or approving full autonomy.

Exceptions must remain in the report. If some cases require manual review, that effort is part of the service conditions. Evaluation should reveal it so the organization can decide whether it has the capacity to absorb the work.

The acceptance threshold belongs to the design of the use. It should be explainable through its relationship to the task and remain visible when conditions change. Copying another application's percentage does not provide that justification. The same score can support different decisions when the consequences and available supervision differ.

Operation supplies evidence tests do not contain

Tests before deployment observe selected conditions. Operation introduces other combinations, uses, and changes. Evaluation consequently does not end with an initial approval.

Service monitoring may reveal new input types, unexpected delays, or tasks users understand differently. That evidence can inform changes to the system and the test collection. Not every newly observed problem calls for a different model.

Monitoring must also be distinguished from controlled evaluation. An alert reports an observed condition without necessarily explaining its cause. A complaint identifies a relevant experience without establishing its frequency. Each signal needs to be placed within the available evidence.

Incidents and corrections can become regression cases, provided relevant conditions are retained and the information involved is appropriately protected. The aim is to preserve learning rather than indiscriminately turn operational data into test material.

The organization needs a record of decisions as well as results: what was authorized, within which limits, and against which version of the tests? That traceability makes it possible to revisit approval when the system or its environment changes.

Good evaluation does not eliminate uncertainty. It makes uncertainty specific enough to work with: what has been demonstrated, where doubts remain, and which decision is reasonable on the current evidence. That is a more useful basis for operation than a score detached from the conditions that produced it.

The decision record completes that evidence by preserving what the organization concluded from it. Later teams can then revisit the same reasoning rather than infer an approval from a favorable number alone.