Skip to content

L01.md

Latest commit

 

History

History
124 lines (94 loc) · 5.67 KB

L01.md

File metadata and controls

124 lines (94 loc) · 5.67 KB
Raw
id course-llm-features-l01
type course
title Lesson 1: A demo reply is not a feature
summary Fetch the course through the FlyPython Skill, run the starter's eval, and catch the number that cannot be true — then map every failing test to the real-world reply that breaks it.
lang en-US
content_version 1
status reviewed
reviewed_on 2026-10-11
hints
title body
What this checkpoint tests
That you can reproduce the failure before fixing anything: `python verify.py` must show l03 and l04 as `[open]`, and you can explain what the starter's eval output gets wrong.
title body
Read the eval line twice
On `it-helpdesk` the starter prints `accuracy=1.000 correct=5/6`. Both numbers cannot be right. Find which one is lying and why.
title body
When you are done
You can name, for one skin, which recorded reply each failing test stands for. Then claim the checkpoint with `python verify.py --attest l01`.

Lesson 1: A demo reply is not a feature

Objective

The course files are in your working folder, fetched by your agent. You have run the starter's eval, found the number in its output that cannot be true, and can say which real-world model reply each failing test stands for. No code is changed in this lesson.

Step 1 — Let the agent fetch this course (you download nothing)

Open your coding agent in an empty working folder and paste this one sentence:

Read https://flypython.com/skills/flypython/SKILL.md and start the FlyPython course llm-features-in-python.

The agent gives you an authorization link with a short code. Open it, log in, make sure the code matches what the agent showed you, and click Allow — then tell the agent "done". It writes this course's files to courses/llm-features-in-python/.

Prerequisite: these courses need a coding agent that can run commands and reach the network. A chat-only web AI cannot do them. The exercise itself needs no API key and makes no network calls.

Why this lesson exists

The first version of almost every model-powered feature looks like the starter: send a prompt, json.loads the reply, use the fields. It works in the demo because the demo reply was well-formed. In a product, the model eventually wraps its JSON in a Markdown fence, invents a category, says "priority": "high", asks for a tool you never built, or answers in prose. Every one of those is a normal reply — the code that handles it is the feature.

The lesson

Ask the agent to survey without changing anything:

"Read COURSE.md — it is the teaching contract for this folder. Then read TASK.md and starter/ticket_triage.py. Do not change any file. Tell me which behaviors the tests expect that the starter does not have."

Then run the learner's command — your single check for the rest of the course:

python verify.py

Now run the starter's eval on two skins:

python starter/ticket_triage.py scenario/it-helpdesk
python starter/ticket_triage.py scenario/shop-support

Read the first line of the it-helpdesk output: accuracy=1.000 correct=5/6. Five out of six is not 100%. The starter's evaluate skips every case that crashed and divides by the survivors — so the harder the model fails, the better the eval looks.

The suite has twelve failing tests. Grouped by the reply that breaks them:

Failing tests The reply that breaks the starter
test_parse_strips_a_markdown_fence ```json { … } ``` instead of bare JSON
test_parse_rejects_an_unknown_category_naming_the_field, test_parse_rejects_bad_priorities, test_parse_rejects_missing_extra_and_mistyped_fields "category": "login", "priority": "high", an extra "confidence", prose, a JSON list — all accepted or crashing
test_unregistered_tool_is_refused, test_tool_arguments_are_validated_before_the_call {"tool": "issue_refund", …} or {"asset_tag": 301} — the model choosing what your code runs
test_invalid_reply_is_retried_with_the_error_fed_back, test_repeated_invalid_replies_fall_back_to_a_human, test_tool_call_budget_ends_in_the_fallback one bad reply ends the request instead of getting a second chance — or a safe fallback — and a model that keeps calling tools is never stopped
test_accuracy_divides_by_every_case_and_counts_fallbacks the eval quietly drops crashed cases
test_shop_support_eval_is_reported_below_threshold, test_it_helpdesk_eval_passes_its_threshold end-to-end: the eval line you just read

Three tests already pass — a valid reply parses, a tool result is sent back to the model, and a missing argument prints usage. The starter is not all wrong; it is wrong exactly where replies stop being polite.

The maintainer command prints the failing names exactly:

python verify.py starter --expect-failure

Exercise

Pick the skin you most identify with, open its recordings.json next to its cases.json, and for each case write one line: what the starter does with that reply (accepts, crashes, or gets it wrong) and what it should do. Check yourself by asking the agent to confirm — not to fix.

Checkpoint

Run python verify.py. When you can answer these without looking, claim the checkpoint with python verify.py --attest l01:

  1. Why does the starter's it-helpdesk eval print accuracy=1.000 with correct=5/6, and which of the two numbers is honest?
  2. Name two recorded replies that crash the starter and one it accepts even though it is wrong.
  3. Why is a reply in a Markdown fence a normal reply rather than a model bug you can ignore?

Expected evidence

The default command's output, the two eval lines, and your per-case notes for one skin. Lesson 4 compares them against the fixed eval.