Skip to content

L03.md

Latest commit

 

History

History
99 lines (77 loc) · 4.46 KB

L03.md

File metadata and controls

99 lines (77 loc) · 4.46 KB
Raw
id course-llm-features-l03
type course
title Lesson 3: Validate, allowlist tools, retry — one group at a time
summary Drive the starter to green in four bounded turns — the parser, the tool gate, the retry-and-fallback loop, the eval — and watch one fix expose a bug the old code was hiding.
lang en-US
content_version 1
status reviewed
reviewed_on 2026-10-11
hints
title body
What this checkpoint tests
That the starter passes all fifteen tests through changes you supervised — `python verify.py` shows this code only once the starter suite is green.
title body
Expect one test to turn red
After the parser turn, `test_tool_result_is_fed_back_to_the_model` fails. The old loop spotted tool calls by parsing them loosely; a strict parser rejects them. The loop has to decode first, then decide.
title body
Never loosen to pass
If a recorded reply fails validation, the fix is never to accept it. Tests and recordings are read-only; the code moves.

Lesson 3: Validate, allowlist tools, retry — one group at a time

Objective

The starter passes all fifteen tests through four bounded turns, and you can explain every diff without reading the solution — including the one test that went red in the middle.

Why this lesson exists

The tempting move is "make it robust" in one prompt. You get a large diff that wraps everything in try/except, accepts anything that looks close, and passes some tests for the wrong reasons. Four bounded turns, each pinned to one contract decision from Lesson 2, keep every line explainable.

The lesson

Start a fresh agent session and say:

"Work through TASK.md against starter/ticket_triage.py, one function per turn: parse_triage, then run_tool_call, then triage, then evaluate. After each turn run python verify.py starter, show me the diff, and wait for me before the next turn. Do not edit tests/, solution/, or scenario/."

Expect the failing count to move like this:

After turn Failing What changed
start 12 —
parse_triage 9 four parser tests pass; one passing test turns red
run_tool_call 7 the allowlist and argument checks
triage 3 retry with feedback, fallback, the tool-call budget, and the tool path repaired
evaluate 0 the denominator, fallbacks, and the two end-to-end tests

The red test after turn one is the most useful moment in the course. test_tool_result_is_fed_back_to_the_model passed only because the starter's triage ran every reply through a parser that accepted any JSON, then looked for a tool key. A strict parser rejects a tool call — it is not a triage answer. The fix belongs in triage (decode the object, then branch on tool), not in loosening the parser. If the agent proposes to make parse_triage accept tool calls again, reject it and point at the contract.

Hold every turn to these boundaries:

  • Only starter/ticket_triage.py changes. Any diff in tests/, solution/, or scenario/ is rejected.
  • No bare except: and no except Exception in triage — the loop catches TriageError (which ToolCallError extends) and nothing else. A client that raises is a different problem and must stay visible.
  • Error messages name the field or argument. "Invalid" is not a message.
  • The fallback never guesses a category.
  • Standard library only.

Exercise

After the triage turn, read the messages a retry sends. Run this in the course folder and read the last message of the second call:

python -c "import sys; sys.path.insert(0, 'starter'); import ticket_triage as t; c = t.ReplayClient(['{\"category\": \"login\", \"priority\": 2, \"summary\": \"x\", \"needs_human\": false}', '{\"category\": \"account\", \"priority\": 2, \"summary\": \"x\", \"needs_human\": false}']); print(t.triage('cannot log in', c, categories=['account', 'other'])); print(c.calls[1][-1])"

Explain in one sentence why that message is what makes a retry worth doing.

Checkpoint

Run python verify.py. This checkpoint's code appears only when the starter suite is green. You pass the lesson when you can answer:

  1. Which test turned red after the parser turn, and why was the fix in triage rather than parse_triage?
  2. What exactly does your run_tool_call check before it calls the function, and in which order?
  3. Why does the loop catch only TriageError?

Expected evidence

A green python verify.py starter, the four diffs, and your explanation of the red-then-green test.