Evaluating AI Systems: Building Evals for Models You Did Not Train
UIT University 365 Institute of Technology
Series AI Evaluation
Duration 25 minutes
IT Engineering, AI and Applied AI, Data Science, Software Development, Digital Transformation
UNOP Sound (University 365 Neuroscience Oriented Pedagogy)
Take five minutes to prepare your brain. Play the isochronous tone track (40Hz gamma frequency) with your eyes closed. Gamma-frequency tones before a learning session raise attention and make the material easier to absorb.
[Audio player: UNOP Pre-Lecture Isochrone (40Hz, 5 minutes)]
In this Lecture
Evaluating AI Systems: Building Evals for Models You Did Not Train
The Model Passed the Benchmark
A model arrives with a score. It sits near the top of a public leaderboard, it answers thousands of exam questions correctly, and it costs less than the model you used last year. Your team runs a pilot for three weeks, the pilot looks good, and the model goes into production. Six weeks later a customer sends you a transcript where the assistant invented a refund policy, promised a delivery date the warehouse cannot meet, and apologized in a tone your brand manager describes as "wrong for us."
Nothing in that story is unusual. The benchmark measured one thing: how well the model answers questions that someone else wrote, about subjects someone else chose, graded by a method someone else designed. Your workflow is a different task, with different inputs, different failure costs, and a definition of "good" that lives in your business, not in a leaderboard.
This lecture teaches you to build the missing layer: an evaluation that belongs to your system. You will learn why public benchmarks cannot answer your question, how to construct a small set of cases that reflects your real work, how to grade those cases without deceiving yourself, which metrics survive a management review, how to turn the result into a release gate, and how to keep the evaluation honest after the model changes underneath you.
The promise is modest and useful. You will not achieve certainty about a model you did not train. You will produce evidence you can defend, a decision you can reverse, and a habit that catches the failures your vendor's benchmark was never designed to see.
Evaluating AI Systems: Building Evals for Models You Did Not Train
Why a Benchmark Is Not an Eval
A benchmark and an evaluation look similar. Both produce a number. Both compare systems. The difference is ownership, and ownership changes everything.
What a benchmark measures
A public benchmark is a fixed test built by a third party. It usually measures a capability in isolation: mathematical reasoning, reading comprehension, code generation from a specification, or factual recall. The test items are chosen to discriminate between models, the grading is standardized so results can be compared across vendors, and the whole thing is published so the market can rank the field.
That design serves the market. It does not serve your workflow. Three properties make a benchmark a poor proxy for your job.
The task is someone else's. Benchmark questions are selected because they separate strong models from weak ones, not because they resemble your customer emails, your internal policies, or the messy PDFs your operations team actually receives.
The grading is coarse. Most benchmarks reduce a rich answer to correct or incorrect. Your work rarely has that shape. A support reply can be accurate, polite, and still unacceptable because it promises something the company cannot deliver. A summary can contain every fact and miss the decision the reader needed.
The items leak into training. Public benchmarks are, by definition, public. Model builders train on data that overlaps with them, sometimes deliberately and sometimes not. A high score may measure familiarity with the test rather than competence on the task.
None of this makes benchmarks useless. They tell you whether a model can do a class of reasoning at all, and they are a reasonable first filter when you compare vendors. They cannot tell you whether this model, in your system, with your prompts and your tools, will do your work acceptably.
The three layers you actually need
A production evaluation has three layers, and a benchmark covers none of them completely.
Capability. Can the model perform the kind of reasoning the task requires? Public benchmarks and small internal probes answer this.
Task performance. Does the model, inside your workflow, produce outputs your reviewers accept? This requires your cases, your rubric, and your data.
System behaviour. Does the whole system, including retrieval, tools, prompts and guardrails, behave safely and predictably over time? This requires traces, regression runs, and a release process.
Most teams skip layer two and discover it in production. The cost of that discovery is not the model bill. It is the trust you spend when a customer sees a failure you could have caught with thirty cases and an afternoon.
The evaluation is a product, not a report
The mental shift that makes this work: treat the evaluation as a product with users, versions and maintenance. Its users are the people who approve releases. Its version number matters because a score is meaningless without the test that produced it. Its maintenance is the work of keeping cases current as the business changes.
A team that treats evaluation as a one-time report will produce a slide. A team that treats it as a product will produce a gate. The gate is what changes behaviour, because it makes the release decision repeatable instead of personal.
Evaluating AI Systems: Building Evals for Models You Did Not Train
The Four Decisions Every Eval Makes
Before you write a single test case, settle four decisions. Each one is a judgment about your business, and each one is reversible only at a cost. Teams that skip this step build evaluations that measure the wrong thing with great precision.
Decision one: the task boundary
Define the unit of work you are evaluating. "Customer support" is not a unit. "Draft a first reply to a tier-one billing ticket, using the refund policy and the order record, in our brand voice" is a unit. It names the input, the resources, the output and the constraints.
Write the boundary as a sentence a new reviewer could apply without asking you a question. If the sentence needs a meeting to interpret, the evaluation will disagree with itself.
A useful test: take three real cases from last week and ask whether they fall inside the boundary. If one of them is ambiguous, the boundary is still too vague.
Decision two: the rubric
A rubric is the definition of acceptable. It turns "good reply" into observable properties. A workable rubric has three to six criteria, each with a short description and a scale.
For a billing reply, the criteria might be:
Correctness. The policy applied matches the current policy document, including the exception for orders older than thirty days.
Completeness. The reply answers the customer's actual question and states the next step.
Commitment safety. The reply promises only actions the system can perform today.
Tone. The reply follows the brand guide: direct, warm, no exclamation marks.
Escalation. The reply routes to a human when the case meets a defined escalation trigger.
Notice that four of the five criteria are not about intelligence. They are about the business. That is the point. A rubric written by the model vendor can never contain your refund exception or your tone rule.
Decision three: the data
Your evaluation data is a sample of real work. It should include typical cases, hard cases, and cases that previously failed. It should be stored with its expected output or its expected judgment, and it should be versioned.
The most common mistake is to build cases from imagination. Imagined cases are clean, and clean cases flatter the model. Real cases carry the typos, the missing context, the conflicting instructions and the emotionally difficult customer, which is exactly where systems break.
The second mistake is to use only recent cases. Old failures are valuable because they encode a lesson the organisation already paid for. Keep a regression set of every failure that reached a customer, and run it forever.
Decision four: the threshold
A threshold converts a score into a decision. "The model passes if it scores above 0.8" is not a threshold until you say what happens at 0.8 and who accepts the risk below it.
A threshold has three parts: the metric, the value, and the consequence. For example: "Release is blocked if severe-error rate on the golden set exceeds 2 percent. Release is allowed between 2 and 5 percent only with a written exception from the workflow owner. Above 5 percent, the release is rejected."
Thresholds should be set before you see the model's score. A threshold set after the result is a rationalisation, and everyone in the room knows it.
The four decisions in one page
Write them down in one page and keep it next to the evaluation. The page becomes the contract between the people who build the system and the people who accept its risk. When someone asks "is the model good", the answer is not a number. It is a reference to this page.
Evaluating AI Systems: Building Evals for Models You Did Not Train
Building the Golden Set
The golden set is the small, curated collection of cases you run every time. It is not a random sample of everything. It is a deliberately chosen sample of what matters, with known correct judgments attached.
Thirty cases is a good starting size. It is small enough to build in a day and large enough to expose patterns. It is not statistically representative, and you should not pretend it is. It is a decision instrument, not a survey.
How to choose the thirty
A balanced set has five kinds of cases.
Typical cases (ten). The work the system will do most days. If your workflow handles billing tickets, these are ordinary billing tickets.
Hard cases (eight). Ambiguous requests, missing data, conflicting policies, unusual formats. These are where models fail in interesting ways.
Adversarial cases (five). Inputs designed to break the system: a prompt injection inside a customer email, a request to ignore the policy, a document with hidden instructions. Safety is part of quality.
Historical failures (five). Every failure that reached a customer in the last two quarters. These cases have known bad outcomes, which makes them easy to grade and impossible to argue with.
Edge cases (two). The boundary of the task: an empty message, a language your team does not support, a request outside the workflow. The correct answer may be a refusal or an escalation, and that is worth testing.
Label each case with the slice it belongs to. Slices let you see whether a model that improved overall has become worse on the cases you care most about.
The structure of a case
A case is more than an input string. It has five fields.
Input. Exactly what the system receives, including attachments or retrieved documents.
Context. The state of the world the system is allowed to use: policy version, customer tier, order status.
Expected output or expected properties. Either a reference answer or a rubric-based judgment.
Slice. Typical, hard, adversarial, historical, edge.
Source and date. Where the case came from, to support later review and to detect drift.
Store the cases in a spreadsheet or a small database. A spreadsheet is fine. The important property is that a non-engineer can read it, because your reviewers will need to.
The trap of the perfect answer
For open-ended work, there is rarely one correct output. Do not force a single reference answer. Instead, encode the properties that must hold. "The reply must cite the current refund window, must not promise a same-day refund, and must offer the escalation path" is a better expected output than a paragraph the model is unlikely to reproduce.
For tasks with a checkable answer, keep the exact answer. Classification labels, dates, amounts and policy identifiers should be exact, because exactness is cheap to grade and hard to fake.
Keep the set small enough to run
A golden set that takes two hours to run will be run once a quarter. A set that takes ten minutes will be run on every change. Speed is a feature of the evaluation, not a compromise.
Aim for a run that completes while you make coffee. If the workflow is slow, cache the retrieval step, freeze the documents, and run the generation step only. The evaluation should measure the decision you are changing, not the latency of your infrastructure.
Version the set
Give the golden set a version, and record which version produced which score. When you add a case, the score changes for reasons that have nothing to do with the model. Without a version, you cannot tell an improvement from a change in the test.
A simple convention works: golden-v1, golden-v1.1 for additions, golden-v2 when you change the rubric. State the version in every result you report.
Evaluating AI Systems: Building Evals for Models You Did Not Train
Grading Without Fooling Yourself
Grading is where evaluations quietly become fiction. A model that grades another model can be useful, but only if you understand its biases and calibrate it against humans. A rubric applied by a tired reviewer can drift over a long afternoon. A checklist can reward the appearance of compliance.
The rule is simple: every grading method has a failure mode, and your job is to know yours.
Method one: deterministic checks
Use code when the answer is checkable. A date must fall in a range. A refund amount must equal the policy calculation. A required field must be present. A link must resolve. An output must parse as JSON.
Deterministic checks are fast, cheap and objective. They are also narrow. They cannot judge whether a reply is empathetic or whether a summary captures the decision. Use them for everything they can cover, then move to judgment for the rest.
Method two: rubric grading by a human
A human reviewer reads the output and scores each rubric criterion. This is the reference method. It is also the most expensive, which is why it is usually reserved for a subset of cases and for calibration.
Make the reviewer's job concrete. Give them the rubric, the case, the expected behaviour and a place to record a short reason. A score without a reason is not reviewable later.
Watch for two human failure modes. Leniency drift is the tendency to score higher as the afternoon goes on. Anchoring is the tendency to score the second output relative to the first. Randomise the order, hide the model identity, and have two reviewers score a sample independently. If their agreement is low, the rubric is ambiguous, not the reviewers.
Method three: LLM as judge
Using a model to grade outputs is now common, and it works when the task is narrow and the rubric is explicit. It fails when the judge is asked to apply taste, when the judge sees the model's name, or when the judge is the same model that produced the output.
If you use an LLM judge, do five things.
Write the judge prompt as a rubric, not a question. Ask for a score per criterion and a one-sentence reason, in a fixed format.
Hide the source. Remove model names, timestamps and any metadata that could bias the score.
Calibrate on humans. Have humans score fifty cases, then measure how often the judge agrees within one point. If agreement is below about 80 percent, the judge is not ready.
Test the judge for position bias. Swap the order of two outputs and check whether the verdict changes. If it does, the judge is measuring order, not quality.
Freeze the judge version. A judge upgrade is a change to your measurement instrument. Record it and re-run the baseline.
Method four: pairwise comparison
When you cannot define a good answer but you can recognise one, compare two outputs. Pairwise comparison is useful for tone, clarity and usefulness. It produces a preference rate rather than an absolute score.
The trap is that preference is relative. A model can win every comparison and still be unacceptable in absolute terms. Pairwise results answer "which is better", not "is this good enough". Use them to choose between candidates, and keep an absolute rubric for the release decision.
The calibration habit
Once a quarter, take twenty cases and grade them three ways: deterministic where possible, a human reviewer, and your judge. Compare the three. Where they disagree, read the case and decide which method is right. Update the rubric.
This habit is the difference between an evaluation that ages well and one that becomes a ritual. The ritual produces numbers. The habit produces judgment.
Evaluating AI Systems: Building Evals for Models You Did Not Train
Metrics That Survive a Meeting
A score is easy to report and hard to act on. The metrics that change decisions share three properties: they are defined in business terms, they separate failure types, and they come with a cost.
Pass rate is necessary, not sufficient
Pass rate answers one question: what share of cases met the rubric. It is the first number most teams report, and it is useful for tracking a trend. It hides the thing that matters most: how the failures fail.
A system with a 90 percent pass rate that fails by inventing a legal commitment is worse than a system with an 80 percent pass rate that fails by asking a clarifying question. The averages are similar; the risk is not.
Severe-error rate is the number executives understand
Define a severe error as a failure the business cannot tolerate, regardless of frequency. In a billing workflow, a severe error might be a wrong refund amount, a promise of a service the company does not offer, or disclosure of another customer's data.
Report the severe-error rate separately. It is usually small, and it is the number that decides whether a deployment proceeds. A useful target is zero severe errors on the golden set before launch, and a monitored rate after launch with an alert threshold.
Cost per completed task
Model quality is only half of the economics. The other half is the cost of producing an acceptable output. A model that passes 95 percent of cases but requires two retries and a human review costs more than a model that passes 90 percent and needs neither.
Measure cost per accepted task, not cost per call. Include retries, judge calls, human review time and the share of cases that escalate. This number belongs next to the pass rate in every release decision.
Latency and abstention
Two behavioural metrics deserve a place in the report.
Latency is the time from request to acceptable output, measured at the 50th and 95th percentile. The average hides the tail, and the tail is what users remember.
Abstention is the share of cases where the system declines, escalates or asks for missing information. A high abstention rate is not automatically bad. On an adversarial set, abstention is the correct behaviour. On typical cases, a rising abstention rate often signals that the model has become cautious after a prompt change.
Coverage and slice results
Report the score for each slice as well as the total. A model can improve on typical cases and degrade on historical failures, which is exactly the trade you must refuse. The slice table is the antidote to a single aggregate number.
A report format that works
One page, four blocks:
Decision asked. Release, hold, or run a limited pilot.
Results table. Pass rate, severe-error rate, cost per accepted task, p95 latency and abstention, broken down by slice.
What changed. Model version, prompt version, retrieval version and golden-set version.
Open risks. The failures the team accepts, with an owner and a review date.
If the report cannot fit on one page, the evaluation is probably measuring too many things.
Evaluating AI Systems: Building Evals for Models You Did Not Train
The Release Gate
An evaluation becomes valuable when it is attached to a decision that happens repeatedly. That attachment is the release gate: a rule that says what evidence is required before a change reaches users, and what happens when the evidence is missing.
Four gate levels
Not every change deserves the same ceremony. A four-level gate keeps the process proportionate.
Level 0, prompt typo. A wording change that cannot alter behaviour. No eval required, but the change is logged.
Level 1, prompt or retrieval change. Run the golden set. Release if the score is within the noise band of the last accepted run and no new severe error appears.
Level 2, model version change. Run the golden set plus the regression set of historical failures. Release only with a written comparison against the previous model and an explicit decision about changed behaviour.
Level 3, new workflow or new tool access. Run the full evaluation, a human review of twenty outputs, and a limited pilot with a rollback plan.
The level is assigned before the change is made. Assigning it afterwards invites the argument that this particular change is special.
The noise band
Scores move between runs for reasons that have nothing to do with quality: sampling temperature, judge variance, retrieval order. Establish the noise band by running the same version three times and measuring the spread. A change that moves the score less than the band is not evidence of anything. Report it as "within noise" and resist the temptation to claim an improvement.
Canary and rollback
The gate decides whether a change may reach users. It does not decide how fast. Send a small share of traffic to the new version, watch the metrics that production can see, and keep the previous version ready.
A rollback is a first-class outcome, not a failure of the team. Write the rollback trigger before launch: "Roll back if severe-error rate exceeds 2 percent over 200 tasks, or if escalation rate rises by more than 20 percent week over week."
The exception process
Sometimes a team needs to ship despite a failed gate. That is a legitimate business decision, and it should be recorded, not hidden. An exception names the failed criterion, the reason, the compensating control, the owner and the expiry date. An exception without an expiry date is a permanent lowering of the standard.
Who signs
The gate needs one accountable signer. In a small team it is the workflow owner. In a larger organisation it is the product owner with the risk function informed. The signer should not be the person who built the prompt, because the builder has an interest in the result.
Evaluating AI Systems: Building Evals for Models You Did Not Train
Evals for Agents
An agent is a system that takes a sequence of actions. That changes the evaluation. You are no longer grading a single output. You are grading a trajectory: what the agent decided, which tools it called, in what order, with what arguments, and what side effects those calls produced.
Grade the trace as well as the answer
A final answer can look correct while the path to it was unacceptable. An agent that reads a customer's full order history to answer a question about a delivery date may produce the right sentence and still violate a data-minimisation rule.
Store the trace: every tool call, its arguments, its result, and the model's stated reason. Then grade four things.
Outcome. Did the task complete correctly?
Trajectory. Did the agent use permitted tools, in a sensible order, without unnecessary steps?
Side effects. Did any action change state outside the sandbox: an email sent, a record updated, a payment initiated?
Recovery. When a tool failed, did the agent notice and recover, or did it continue confidently on bad data?
Tool call accuracy
For each expected tool call, define the acceptable arguments. A refund tool that is called with the right order identifier and the wrong amount is a failure, even if the final message is apologetic. Score tool calls separately from prose.
The side-effect budget
Give every evaluation run a side-effect budget: the maximum number of state-changing actions it may take. If the agent exceeds the budget, the run fails regardless of the answer. In production, enforce the same budget as a hard limit.
Adversarial trajectories
Agents are exposed to inputs that try to redirect them. Include cases where a retrieved document contains an instruction, where a tool result contains a misleading error, and where the user asks the agent to ignore its policy. The correct behaviour is usually refusal or escalation, and it must be tested, not assumed.
A minimal agent eval set
Ten tasks, each with a defined starting state, permitted tools, expected outcome and a side-effect budget. Run each task three times to expose variance. Record the trace and grade it. This is enough to catch the failures that single-turn evaluations miss.
Evaluating AI Systems: Building Evals for Models You Did Not Train
Keeping the Eval Honest
An evaluation is a measurement instrument, and instruments drift. The cases age, the judge changes, the business rules move and the model learns from data that resembles the test. Four maintenance habits keep the instrument honest.
Contamination checks
Ask, for every case, whether the input or the expected answer could have appeared in training data. Public data, published examples and widely shared documents are high risk. Private, recent, organisation-specific cases are lower risk.
For high-risk cases, change the surface details while keeping the structure: rename the customer, shift the dates, alter the amounts. The task stays the same and memorisation stops helping.
Judge drift
If you grade with a model, the judge is a dependency. Pin its version, record it with every result, and re-run the human calibration sample whenever the judge changes. A judge upgrade can move your score by several points without any change to the system under test.
Case freshness
Review the golden set every quarter. Retire cases that no longer reflect the business. Add cases from the last quarter's incidents. Keep the historical failures, because their value does not expire.
Ownership
Name one person who owns the evaluation. That person maintains the cases, runs the calibration, signs the version and answers the question "why did the score change". Without an owner, the evaluation becomes a dashboard nobody trusts.
Evaluating AI Systems: Building Evals for Models You Did Not Train
A 30-Case Eval in One Day
This is the practical sequence. It assumes one workflow, one reviewer and access to real cases.
Morning: define and collect
Hour one. Write the task boundary in one sentence. Write the rubric with three to five criteria. Write the threshold with its three parts. Get the workflow owner to agree in writing.
Hour two. Pull twenty real cases from the last month: ten typical, five hard, five edge or adversarial. Add five historical failures from incident records. If you do not have five failures, use five cases the team remembers as difficult and label them as such.
Hour three. Write the expected behaviour for each case. Use exact answers where the task is checkable and rubric properties where it is open. Record the slice and the source.
Afternoon: run and grade
Hour four. Run the system on the thirty cases with a fixed prompt, fixed model version and fixed retrieval snapshot. Save the inputs, outputs and traces.
Hour five. Grade the first ten yourself. Then grade the next twenty with your judge and compare the judge against your ten. If agreement is poor, fix the judge prompt before continuing.
Hour six. Build the one-page report. Compute the overall pass rate, the severe-error rate, cost per accepted task and the slice table. Write the open risks.
The same day, a decision
Take the report to the workflow owner. Decide one of three things: release, hold, or pilot with a named risk. Record the decision and the date. Put the next run in the calendar.
This is not a perfect evaluation. It is a real one, built in a day, owned by someone, and attached to a decision. It will catch the failures that a public benchmark cannot see, and it will improve every time you run it.
Evaluating AI Systems: Building Evals for Models You Did Not Train
Six Ways First Evaluations Fail
Most first evaluations fail in predictable ways. The failures are worth naming, because each one feels like progress while it is happening.
1. Testing the model instead of the system
A team runs the prompt in a playground, grades the answers, and declares the system ready. The playground has no retrieval, no tools, no guardrails and no real latency. The evaluation measured a component and the team shipped a system.
Fix: every case must run through the same path the user will take, including retrieval, tool calls and output filtering. If the path is too slow to test, cache the retrieval and still run the real prompt and the real tools in a sandbox.
2. A rubric written by committee with no owner
Six people contribute criteria. The result has fourteen dimensions, several of them overlapping, and no one can say which criterion takes precedence when two conflict. Reviewers score differently, the judge is confused, and the numbers stop moving in a way anyone can explain.
Fix: three to five criteria, one owner, one precedence rule. If two criteria conflict, the owner decides which one wins and writes it down.
3. A judge from the same family as the model under test
A model grading its own family tends to prefer its own style. The evaluation looks strong and the production experience disagrees. This is not a hypothetical bias; it is a structural one, and it survives prompt engineering.
Fix: use a judge from a different provider where possible, hide the source of the output, and calibrate against humans on at least fifty cases. If you must use the same family, treat the judge as a screening tool and keep a human sample in the loop.
4. Cherry-picked cases
The golden set is built from cases the team handled well. The evaluation passes, the launch happens, and the first real customer brings a case nobody imagined.
Fix: sample from the inbox, not from memory. Include the awkward, the incomplete and the angry. Ask the support team which cases they dread, and put those in the set.
5. A score with no cost
The report says the new model is four points better. Nobody asks what it costs per accepted task, how often it retries, or how much human review it needs. The team ships a quality gain that destroys the unit economics.
Fix: report cost per accepted task next to every quality metric. Make the release decision on both.
6. The evaluation that never runs again
The first run is careful. The second run happens after a prompt change. The third never happens, because the person who built it changed team. Six months later nobody knows whether the system still works.
Fix: put the run in the release process, not in someone's calendar. Automate the execution, version the cases, and name an owner. An evaluation that is not in the release path is a historical document.
The pattern behind the six
Every failure above replaces a decision with a artefact. A playground transcript, a long rubric, a judge score, a curated set, an average, a report. The artefact looks like evidence, and the decision quietly disappears.
The correction is the same in every case: return to the four decisions, write them down, and attach them to a release that happens on a schedule. The method is not sophisticated. It is owned, which is what makes it work.
Evaluating AI Systems: Building Evals for Models You Did Not Train
Feynman Summary: Explain It Like You Are 12
Imagine you buy a football and the box says "best ball in the world". The box was tested by kicking the ball against a wall, one thousand times, in a laboratory. That tells you the ball is round and strong. It does not tell you whether it works on a wet field, with your friends, in your garden.
A benchmark is the laboratory test. An evaluation is playing a real match with your own rules. You choose thirty situations that look like your real games: easy passes, hard passes, muddy ground, a friend who tackles badly, and the one time last year the ball went over the fence. You decide what a good pass looks like. You score every attempt. Then you decide whether the ball is allowed into the match.
The important part is not the number. It is that you wrote the rules, you chose the situations, and you can explain the decision to someone else.
Evaluating AI Systems: Building Evals for Models You Did Not Train
Mindmap: The Complete Picture
The mindmap shows the whole method in one view: the four decisions at the centre, the golden set and grading branches, the metrics and gate branches, and the maintenance loop that keeps the instrument honest.
UNOP Sound (University 365 Neuroscience Oriented Pedagogy)
Take five minutes to consolidate your memory. Play the isochronous tone track (10Hz alpha frequency) with your eyes closed. Alpha-frequency tones after a learning session support consolidation, helping move what you just learned from short-term to long-term memory.
[Audio player: UNOP Post-Lecture Isochrone (10Hz, 5 minutes)]
Evaluating AI Systems: Building Evals for Models You Did Not Train
Practical Exercise: Build and Gate One Workflow
Choose one workflow you use this week. It can be small: drafting replies to supplier emails, summarising meeting notes, or classifying support tickets. Complete the six steps and keep the artefacts.
Write the boundary. One sentence naming input, resources, output and constraints. Test it against three real cases.
Write the rubric. Three to five criteria, each observable, each with a scale. Ask a colleague to apply it to one case without your help.
Collect ten cases. Four typical, three hard, two adversarial, one historical failure. Record the expected behaviour for each.
Run the system once. Fixed prompt, fixed model, fixed data. Save the outputs and the traces.
Grade and report. Use deterministic checks where possible, a human rubric for the rest. Compute pass rate, severe-error rate and cost per accepted task.
Set the gate. Write the threshold and the consequence. Name the signer. Put the next run in the calendar.
If you complete step six, you have something most teams do not have: a release decision that does not depend on how confident someone feels on the day.
Evaluating AI Systems: Building Evals for Models You Did Not Train
Glossary
Term | Definition |
Benchmark | A fixed, usually public test built by a third party to compare models on a capability |
Eval | A task-specific evaluation owned by the team that runs the system, with its own cases, rubric and threshold |
Golden set | The curated, versioned collection of cases run on every change |
Rubric | The written definition of acceptable output, expressed as observable criteria and a scale |
Slice | A labelled subset of cases, such as typical, hard, adversarial or historical |
Severe error | A failure the business cannot tolerate regardless of how rarely it occurs |
LLM judge | A model used to grade outputs against a rubric, calibrated against human reviewers |
Noise band | The score variation observed when the same version is run repeatedly |
Release gate | The rule that attaches evidence requirements to a change before it reaches users |
Trajectory | The sequence of steps, tool calls and results an agent produces while completing a task |
Contamination | The presence of evaluation cases or answers in the model's training data |
Abstention | The share of cases where the system declines, escalates or asks for missing information |
Evaluating AI Systems: Building Evals for Models You Did Not Train
Quiz: TEST YOUR UNDERSTANDING
1. Why can a high public benchmark score fail to predict performance on your workflow?
A) Public benchmarks are always outdated
B) The benchmark measures a different task, with different inputs, grading and failure costs
C) Benchmarks are designed only for coding
D) The model was not trained on enough data
2. What are the four decisions every evaluation makes?
A) Model, prompt, temperature, tokens
B) Task boundary, rubric, data, threshold
C) Vendor, price, latency, hosting
D) Users, budget, timeline, scope
3. What is the main risk of using only recent, typical cases in a golden set?
A) The set becomes too large to run
B) The evaluation misses hard, adversarial and historical failures where systems break
C) The judge becomes unnecessary
D) The model learns the cases too quickly
4. Why must an LLM judge be calibrated against humans?
A) To reduce the cost of grading
B) To measure how often the judge agrees with human judgment before trusting its scores
C) To make the judge faster
D) To remove the need for a rubric
5. What does a release gate add to an evaluation?
A) A larger test set
B) A rule that defines the evidence required before a change reaches users, and the consequence
C) A new model version
D) A marketing claim
Answers: 1-B, 2-B, 3-B, 4-B, 5-B
Related Resources
U365 INSIDE Publications
External Resources
Related U365 Lectures (Coming Soon)
Reasoning Models in Practice: Test-Time Compute and When It Pays (AI Evaluation and AI Foundations)
AI Observability: Tracing, Logging and Debugging Agents (AI Engineering)
AI Governance in Practice: Auditing Models, Agents and Vendors (Cross-Institute Series)
Evaluating AI Systems: Building Evals for Models You Did Not Train
U.Copilot for This Lecture
You are my evaluation coach. I am building a task-specific evaluation for one workflow at my organisation. Workflow: [describe the unit of work in one sentence] Inputs: [what the system receives] Resources: [documents, databases or tools it may use] Output: [what it must produce] Constraints: [policy, tone, commitments, escalation rules] Do five things: 1. Rewrite my task boundary as one testable sentence, and list the ambiguities you had to resolve. 2. Propose a rubric with three to five observable criteria and a three-level scale for each. 3. Design a 30-case golden set: ten typical, eight hard, five adversarial, five historical failures, two edge cases. For each case, state the input, the context and the expected behaviour. 4. Recommend a grading method per criterion: deterministic check, human rubric or LLM judge, and explain the risk of the method you chose. 5. Propose a release gate with a metric, a threshold and a consequence, and name the person who should sign it. Ask me one clarifying question before you start if the workflow description is ambiguous.
Evaluating AI Systems: Building Evals for Models You Did Not Train
Next Steps
Choose the one workflow you will evaluate first, and write the task boundary today.
Build the ten-case pilot this week, before you touch the full thirty.
Run the calibration between your judgment and an LLM judge on ten cases.
Put the first gate in front of a real release, even a small one.
Send me your one-page report through the U365 Fellow portal so we can review the rubric together.
Evaluating AI Systems: Building Evals for Models You Did Not Train
IMPORTANT NOTICE
Copyright University 365, Inc. All rights reserved. This lecture is provided for educational purposes and does not constitute legal, financial or professional advice. For enrollment, visit university-365.com/tuition. For permissions or inquiries, contact uda@university-365.com.
Published by the Department of Academics, University 365.
Lecture delivered by the University 365 Institute of Technology (UIT).
Sam Utteker, Dean of Technology, UIT
Signed for the academic year 2026.
Comments