Skip to content

L04.md

Latest commit

 

History

History
104 lines (81 loc) · 4.17 KB

L04.md

File metadata and controls

104 lines (81 loc) · 4.17 KB
Raw
id course-llm-features-l04
type course
title Lesson 4: Read the eval honestly
summary A green suite proves the contract; the eval measures the feature. Run all three skins, accept the one that fails its threshold, and write down what a handful of recorded replies can and cannot tell you.
lang en-US
content_version 1
status reviewed
reviewed_on 2026-10-11
hints
title body
What this checkpoint tests
That starter AND solution both pass — the maintainer pair `verify.py starter` / `verify.py solution` must both be green, and `python verify.py` prints this code once your starter suite passes.
title body
A failing eval can be the correct result
`shop-support` reports 0.750 against a 0.800 threshold and exits 1. Your code is right; the feature on that data is not good enough yet. Lowering the threshold is not a fix.
title body
When you are done
You have a three-line evidence record — verified / not verified / known limits — that a teammate could act on.

Lesson 4: Read the eval honestly

Objective

Both suites pass, you have run the eval on every skin, and you hold a written record of what the numbers prove and what they do not.

Why this lesson exists

In Lesson 1 the starter's eval said 100% on a skin where one case in six had crashed. That is the most common way model features get shipped too early: not a bad model, an eval that hides its failures. Now that the eval divides by every case, the numbers can go down — and that is the point.

The lesson

Run the maintainer pair in a terminal yourself:

python verify.py starter
python verify.py solution

Then run your eval on every skin:

python starter/ticket_triage.py scenario/shop-support
python starter/ticket_triage.py scenario/it-helpdesk
python starter/ticket_triage.py scenario/saas-feedback

You should see:

Skin Result Exit
shop-support accuracy=0.750 correct=6/8 fallbacks=1 1
it-helpdesk accuracy=0.833 correct=5/6 fallbacks=1 0
saas-feedback accuracy=0.800 correct=4/5 fallbacks=0 0

Compare it-helpdesk with Lesson 1: the starter said 1.000, your eval says 0.833. Same recorded replies, honest denominator.

Now read each failure line as a reviewer:

  1. shop-support fails its threshold — correctly. Case s08 got a valid but wrong category; s07 fell back to a human after two prose replies. Nothing in your code is wrong. The feature, on this data, is below the bar you set. Try --min-accuracy 0.7 and watch it "pass" — then write down why that is not a fix.
  2. A fallback is safe but not free. h06 fell back because both replies broke the 120-character summary limit. Count fallbacks separately: a feature that falls back on half the tickets is safe and useless.
  3. Look at what the eval does not score. Correctness here is category + needs_human. priority and summary are validated for shape but never compared to a label. Decide whether that is acceptable for your feature.
  4. Check the edges the suite skips. What happens if cases.json is empty? If recordings.json has fewer replies than the loop asks for? (ReplayClient raises — your loop should let it.) Run one and write down what happened.

Exercise

Write the three-line evidence record for this feature:

  • Verified: the commands you ran and what they proved.
  • Not verified: at least — live model behavior, traffic you have not recorded, unscored fields, and anything the recordings do not imitate.
  • Known limits: the threshold each skin missed or barely met, and the fallback rate.

Checkpoint

Run python verify.py. This checkpoint's code prints once the starter suite passes; the maintainer pair above must both be green for the gate to mean what it says. You pass the lesson when you can answer:

  1. Why is shop-support exiting 1 the correct behavior of your code?
  2. What does accuracy=0.833 on six hand-written cases tell you about a live model on live tickets?
  3. Which field would you add to the eval's scoring first, and why?

Expected evidence

Both green runs, the three eval lines, and your three-line evidence record.