Skip to content

L02.md

Latest commit

 

History

History
96 lines (76 loc) · 4.35 KB

L02.md

File metadata and controls

96 lines (76 loc) · 4.35 KB
Raw
id course-llm-features-l02
type course
title Lesson 2: Write the output contract
summary The prompt asks; the contract checks. Read TASK.md as the boundary between your code and the model — what a reply must look like, which tools it may run, and what happens when it does not comply.
lang en-US
content_version 1
status reviewed
reviewed_on 2026-10-11
hints
title body
What this checkpoint tests
That you can read `TASK.md` as testable statements and trace each to a test — and say why the format request in `build_messages` is not enough on its own.
title body
A prompt is a request, not a guarantee
`build_messages` tells the model the exact format. The model usually follows it. 'Usually' is the reason `parse_triage` exists.
title body
A tool call is the model choosing what your code runs
The tool registry is an allowlist and the arguments are untrusted input — the same rule you would apply to a web form.

Lesson 2: Write the output contract

Objective

You can read TASK.md as a set of testable statements about model replies, trace each statement to a test in tests/test_ticket_triage.py, and explain the three decisions the contract makes for you: what a valid reply is, which tool calls may run, and what happens when the model does not comply.

Why this lesson exists

Open build_messages in the starter. It already tells the model the exact JSON fields, the allowed categories, the priority range, and how to call a tool. Many teams stop there — the prompt says what we want, so the reply will be what we want. The prompt is a request. The contract is what your code enforces when the request is ignored, and the tests are how you know it does.

Provider APIs often offer a JSON or schema-constrained output mode. Where you use one, it reduces malformed replies; it does not tell you that the category is right, that a tool argument is safe to act on, or that your eval is honest. The contract in this course still applies.

The lesson

Read TASK.md with the tests open beside it. The contract makes three decisions:

1. What a valid reply is. Exactly four fields, each with a type and a range. Anything else is rejected with an error that names the field.

Contract line Test
"strip … one surrounding Markdown code fence" test_parse_strips_a_markdown_fence
"category is one of categories" test_parse_rejects_an_unknown_category_naming_the_field
"priority is an int from 1 to 4 (bool and float are rejected)" test_parse_rejects_bad_priorities
"a missing field or an extra field raises TriageError naming that field" test_parse_rejects_missing_extra_and_mistyped_fields

Why name the field? Because the error message is going back to the model in Lesson 3. "Invalid reply" gives it nothing to correct; "category must be one of billing, shipping, …; got 'login'" tells it exactly what to change.

2. Which tool calls may run. The tools dict is the allowlist. An unregistered name is refused before anything runs, and arguments are checked against declared names and types before the function is called. The recorded reply {"tool": "issue_refund", …} in shop-support is the case to remember: a model that can name any function can make your code do anything that function does.

3. What happens on failure. One retry with the error fed back, then a fallback that sends the ticket to a human with no category. The fallback never guesses — a confident wrong category is worse than an honest "needs a person".

Exercise

Take a model call from a project you know — or invent one — and write its output contract in the same shape: the exact fields with types and ranges, the tools it may call with their argument types, and the failure policy (how many retries, what the fallback returns). Then mark each sentence: could a test assert it without reading your mind?

Checkpoint

When you can answer these, claim the checkpoint with python verify.py --attest l02:

  1. build_messages already states the format. Why does parse_triage still have to check it?
  2. Why must a rejected tool call never reach the tool function — what could issue_refund have done?
  3. Why does the fallback return category: None instead of the most likely category?

Expected evidence

Your drafted output contract with each sentence marked testable or not.