An evaluation framework for AI agents needs three tiers: deterministic checks that verify structure, semantic checks that score meaning with an LLM judge, and behavioral checks that verify the agent's tool-call path. Run them in that order — cheapest first — on every prompt change, and block deploys when scores drop. This is the harness I built for the agent engine behind withhammad.com after the standard failure everyone hits: it worked in the demo, then broke silently in production.
Why 'it sounds right' isn't a test
LLM output is non-deterministic: the same prompt can produce different phrasings, different structures, and occasionally different meanings across runs. Traditional assertions — expect exact string X — are useless against that. But the opposite instinct, eyeballing a few outputs and shipping because they 'sound right', is worse, because it tests the happy path once and never again. The failure mode that motivates all of this: you tweak a prompt to fix one complaint, the fix works, and three unrelated behaviors quietly regress. Without a repeatable eval suite, you discover those regressions from users. The goal is the same as any test suite in software — turn 'seems fine' into a number you can compare across versions.
Tier 1 — deterministic checks
Deterministic checks are ordinary code assertions on structure: does the output parse as valid JSON, does it contain every required field, does the date match the expected format, is the response under the length limit, does it avoid strings it must never contain. They cost nothing to run — no model call — and they catch an outsized share of real production failures, because agents feed their output into downstream systems that break on malformed structure long before anyone judges quality. Run this tier first and fail fast: an output that does not parse should never reach the judge model, both because scoring it wastes money and because the failure is already conclusive.
Tier 2 — semantic checks with an LLM judge
Semantic checks ask whether the output means the right thing, and the only scalable way to score that is LLM-as-judge: a second model compares the agent's output against a reference answer or a rubric and returns a score. The judge prompt is where this succeeds or turns to mush. What works: score one narrow criterion per call rather than 'rate overall quality', force a structured verdict (a numeric score plus a one-line reason, in JSON), anchor each score level with a concrete description of what it looks like, and include the reference answer in the prompt rather than trusting the judge's own knowledge. A vague judge prompt produces scores that drift run to run, which makes the whole tier noise. A tight one turns fuzzy quality into a threshold you can gate on.
Tier 3 — behavioral checks on the decision path
ONE TACTIC A WEEK
The first two tiers only see the final text. Behavioral checks inspect the trace: which tools the agent called, with what arguments, in what order, and whether it stopped when it should have. This tier catches bugs the others structurally cannot — an agent that produced a perfectly formatted, semantically correct answer by skipping the lookup and inventing the data, or one that called the write tool before the validation tool, or one that looped through five redundant searches to reach an answer one call should have produced. Assertions here look like: expected tool sequence contains a calendar check before a calendar write; no tool called more than N times; the run terminated without a forced cutoff. For agents whose entire job is taking actions, this tier is the test suite; the text tiers are just formatting checks.
Choosing a judge model without burning your budget
Here is the cost problem eval vendors rarely lead with: LLM-as-judge means a model call per test case per criterion, and if you run evals on every commit — which you should — an expensive judge makes the suite something you avoid running, which defeats it. My practical answer is Groq's free-tier models as the default judge: fast, costs nothing, and for well-anchored single-criterion rubrics a mid-size open model agrees with a frontier model often enough that the threshold still means something. The honest limits: subtle reasoning quality, tone judgments, and domain-expert correctness checks are where a cheap judge gets unreliable, and for those few cases I escalate to a frontier model. Tiering the judge the same way you tier the checks keeps the suite cheap enough to run constantly.
Wiring evals into the actual workflow
An eval suite that runs 'sometimes' is decoration. The wiring that makes it real: every prompt is versioned, and every prompt version change triggers the eval suite against a fixed set of test cases before the new version can go live. The suite also runs on a schedule against samples of live traffic, because prompts are not the only thing that drifts — model updates and changing user inputs shift behavior under a frozen prompt. Each run logs the prompt version alongside the scores, which is what lets you point at a regression and name the exact diff that caused it.
What a failing eval run should block
Treat an eval score drop exactly like a failing CI test: the deploy does not happen. This sounds obvious and is culturally the hardest part, because the new prompt usually 'looks better' to whoever wrote it, and overriding a red suite by vibes is how silent regressions ship. Set explicit thresholds per check — deterministic checks at 100%, semantic and behavioral scores at whatever floor your history supports — and make lowering a threshold a reviewed change, not a bypass. The framework only pays for itself the day it stops a confident bad change; until then it is scaffolding, and after that day nobody on the team argues about whether it is worth running.