Change only starter/ticket_triage.py. Standard library only. Never modify
tests/, solution/, or scenario/. ReplayClient, build_messages,
load_scenario, and main are provided and already correct; the work is in
the four functions below. A client is any callable
client(messages) -> str; every string it returns is untrusted input.
parse_triage(raw, categories) -> dict:- Strip surrounding whitespace and one surrounding Markdown code fence
(
```or```json) before parsing. - The reply must be a JSON object with exactly the fields
category,priority,summary,needs_human— a missing field or an extra field raisesTriageErrornaming that field. categoryis one of categories;priorityis anintfrom 1 to 4 (boolandfloatare rejected);summaryis a non-empty string of at most 120 characters after stripping;needs_humanis abool.- Every rejection raises
TriageErrorwhose message names the offending field. Non-JSON text and non-object JSON also raiseTriageError. - Return exactly the four fields, with
summarystripped.
- Strip surrounding whitespace and one surrounding Markdown code fence
(
run_tool_call(call, tools) -> (name, result):- tools maps a tool name to
{"function": callable, "parameters": {name: type}}— it is the allowlist. - call must have exactly the keys
toolandarguments. An unregistered tool name raisesToolCallErrornaming the tool. - arguments must have exactly the declared parameter names, each an
instance of its declared type (
boolis not accepted forint). Missing, unexpected, or mistyped arguments raiseToolCallErrornaming the argument. The tool function is never called on a refused call. - Return
(name, function(**arguments)).
- tools maps a tool name to
triage(ticket_text, client, *, categories, tools=None, max_attempts=2, max_tool_calls=3) -> dict:- Start from
build_messages(...). After each reply, append{"role": "assistant", "content": raw}. - A reply that decodes to an object with a
toolkey is a tool call. On success, append{"role": "tool", "name": name, "content": json.dumps(result)}and ask the client again; successful tool calls do not count as attempts.tool_callscounts only calls that ran. A tool call that would exceed max_tool_calls is not run and ends in the fallback. - Any rejected reply — invalid answer or refused tool call — counts as one
attempt and appends
{"role": "user", "content": ...}that contains theTriageErrormessage, then asks again. - A valid answer returns the four fields plus
attempts(rejected replies- 1),
tool_calls, andfallback: False.
- 1),
- After max_attempts rejected replies, return the fallback:
{"category": None, "priority": None, "summary": ticket_text.strip()[:120], "needs_human": True}plusattempts(the number of rejected replies),tool_calls, andfallback: True. Never guess a category.
- Start from
evaluate(cases, client_for, *, categories, tools=None) -> dict:- Run
triagefor every case withclient_for(case). A case is correct whencategoryandneeds_humanboth matchcase["expected"]. - Return
total(every case),correct,fallbacks,schema_valid(total - fallbacks),accuracy(round(correct / total, 3),0.0for no cases), andfailures: one{"id", "reason"}per incorrect or fallback case, in case order. - Never skip a case: the denominator is every case you were given.
- Run
python ticket_triage.py <scenario-dir> [--min-accuracy X] prints
accuracy=… correct=N/T fallbacks=F min_accuracy=…, one line per failure,
and returns 0 when accuracy >= min_accuracy, else 1; usage errors return 2.
The threshold defaults to the scenario's min_accuracy.
Done means python verify.py starter exits 0 with all fifteen tests
passing.