For engineers who have never evaluated an agent before: a fundamental mental model, not a tool tutorial.
If you can't define success, you can't measure it. If you can't measure it, you can't improve it.
01 · Why bother
Development without evaluation is guessing
Without evals
"It seems better" after every prompt tweak
Fix one case, silently break three others
No way to compare models, prompts, or architectures
Demos work; production surprises you
With evals
Every change scored against the same yardstick
Regressions caught before users see them
An objective basis for choosing model, prompt, design
Evaluation drives iteration, not the other way around
02 · Where we came from
Classic ML: clean ground truth, clean metrics
Regression · predict a number
House price from square footage, bedrooms, location. Fit a curve; measure the distance from reality.
MAE · MSE · RMSE · R²
Classification · pick a category
Spam or not-spam, from keywords, sender, formatting. Every prediction is simply right or wrong.
Accuracy · Precision · Recall · F1 · AUC-ROC
The luxury of this era was a mathematically defined ground truth. Hold that thought: it is exactly what we lose next.
03 · What changed
A model we didn't build, wrapped in an agent
We no longer train the model; our levers are context, prompts, and parameters. And "good" became subjective, graded by similarity scores, judges, and humans. Then agents raised the stakes further:
They act
Any tool call can fail even when the words sound right.
They converse
Context must survive the session; each turn should advance the task.
They retrieve
Wrong context in, wrong answer out, whatever the model.
So evaluation must mirror the full lifecycle: not was this sentence good? but did the agent do the right thing, the right way, end to end?
04 · The core practice
Eval-driven development
1Goals & metricsDefine what "good" looks like.
2Core datasetReal examples, expected outcomes.
3EvaluateScore against the dataset.
4AnalyzeWhat failed, and why?
5OptimizeAdjust, then repeat the cycle.
Steps 1 and 2 are the whole game. Without defined success and concrete examples, every iteration after them is just noise.
05 · Ground truth, rebuilt
The core dataset
What it is
Input, context, and expected outcome, per example
Drawn from real situations: past conversations, tickets, frequent questions
Covers the frustrating and slow paths, not just the happy ones
Paired with a rubric: a scoring guide that encodes expert judgment
Small and honest beats large and synthetic; start with dozens
One example, in full
Input
Where's my refund for order #8123?
Context
Order record (return received Jul 2) · refund policy (5 to 10 business days)
Expected
Confirms the return was received; states the 5-to-10-day window; makes no promise outside policy
Rubric
Grounded in policy (0 to 2) · order facts correct (0 to 2) · tone (0 to 1) · offers next step (0 to 1)
A support-agent case. Note the expected outcome describes behavior, not exact wording.
05b · Ground truth, rebuilt
The regression dataset
Every fixed failure becomes a permanent test case, run on every change like a test suite. Here is one case being born:
1IncidentJul 3: the agent promises an instant refund. Policy says 5 to 10 days. A user screenshot circulates.
2TracePull the full interaction: the input, what was retrieved, which tools ran, the exact output.
3DistillFreeze it as a test case. Same input and context; expected: quote the policy window, never promise "instant".
4GuardThe case joins the regression set. Any future prompt or model change that reintroduces the promise fails loudly.
Remember "fix one case, break three others"? This is the set that catches the other three.
LLM-as-a-judge with weighted criteria: helpfulness, reasoning, tone
Evaluation is not one-size-fits-all. Frameworks such as DeepEval and LangSmith ship these as ready-made scorers; choose yours before you build.
06b · Choosing metrics
The same refund case, under each lens
Metric
The question it asks
What failing looks like
Answer relevancyrag
Did the reply address what was asked?
Asked "where's my refund?", got a tutorial on starting a return.
Faithfulnessrag
Does the answer stick to the retrieved context?
Policy says 5 to 10 days; the agent invents "usually within 48 hours".
Contextual recallrag
Did retrieval fetch everything the answer needs?
Policy retrieved, order record missed: can't confirm the return arrived.
Contextual precisionrag
Is the relevant material ranked on top?
Eight shipping-label docs first; the refund policy buried at rank 7.
Tool & argument correctnesstools
Right tool, right inputs?
Calls lookup_order with #8132 instead of #8123.
Knowledge retentionmulti-turn
Does it remember what it was told?
Order number given in turn 1, asked for again in turn 4.
Turn relevancymulti-turn
Does each turn move the task forward?
Mid-case, the agent pivots to promoting the loyalty program.
Role adherencemulti-turn
Does it stay the agent you defined?
The refund assistant starts offering tax advice.
A metric is just a question you ask of the same trace. If the name sounds fuzzy, say the failure out loud.
07 · The judge
LLM-as-a-judge, demystified
What it is
A model handed your rubric and asked to grade another model's output
Exists because subjective criteria don't scale to humans reading every case
Returns scores with reasons, so failures stay debuggable
The judge is a model too: it has biases, and it needs spot-checks against human labels before you trust it
The rubric, applied
Judge prompt
You are grading a refund-support reply. Policy: refunds take 5 to 10 business days; return received Jul 2. Score the reply: grounded in policy (0 to 2), order facts (0 to 2), tone (0 to 1), offers next step (0 to 1). Give a one-line reason per score.
Reply under test
Your return arrived July 2, so your refund is on the way! It usually lands within 48 hours.
Judge output
grounded 0/2 ("48 hours" contradicts policy) · facts 2/2 · tone 1/1 · next step 0/1 (none offered) → 3 of 6, fail
The rubric from the core-dataset slide, executed by a model instead of a human.
One question per criterion, scores with reasons, and a human audit of the judge itself. That's the whole trick.
08 · Reading the numbers
What a run actually looks like
Suite
Prompt v3
Prompt v4
Core set · 50 cases
39 / 50 (78%)
42 / 50 (84%)
Regression set · 12 cases
12 / 12
11 / 12
Case #12 · "instant refund"
pass
fail
Verdict
shipped
held
Pass rates are over three runs per case. Agents are nondeterministic: the same eval run twice gives different numbers, so score repeated runs, never a single pass.
v4 wins the average and still can't ship: it re-broke case #12. Aggregates hide regressions; the regression set catches them.
09 · After you ship
Online evaluation: production signals
Did it work?
Resolution rate, thumbs up or down
Escalation rate: did they ask for a human?
Turns to resolution
Was it worth it?
Token usage, and so cost per resolution
Latency and time-to-first-token; no one likes a slow agent