Skip to content

W2c: AgentCli diagnostics (#41), detection tests (#42), Qwen real-GGUF validation (#43), native-vs-generic docs (#44) - #299

Merged
michalharakal merged 1 commit into
feat/tool-calling-w2bfrom
feat/tool-calling-w2c
Aug 11, 2026
Merged

michalharakal merged 1 commit into
feat/tool-calling-w2bfrom
feat/tool-calling-w2c

Conversation

@michalharakal

Copy link
Copy Markdown
Contributor

Summary

W2c — final PR of the tool-calling stack (#35 epic). Stacked on #297 (W2b) — review only the last commit until #296/#297 merge.

#41 — AgentCli resolution diagnostics

ToolCallingDemo already printed Provider: <family> (mode=…, reason: …); AgentCli (chat + agent modes) now prints the same line, so native-vs-generic selection is visible in every CLI mode. Explicit --template selection and auto-detection both flow through the identical resolver path (unchanged).

#42 — detection/diagnostics tests

ResolutionDiagnosticsTest pins resolveWithDiagnostics: explicit-override reason, auto-detected reason, GENERIC fallback mode + reason, and GGUF-extraction→diagnostics carry-through. Together with the stack this gives: resolver 21 (pre-existing) + GGUF extraction 11 (#296) + HF-side detection & parser registration 17 (#297) + diagnostics 4 (here).

#43 — Qwen validation against a real instruct GGUF

QwenToolCallSmokeTest (kllama jvmTest), env-gated on QWEN_MODEL_PATH, two stages:

  1. Metadata → resolution (header-only read, seconds): verified PASSING against a local Qwen2.5-0.5B-Instruct-F16.gguf
    family=qwen, arch=qwen2, hints=[<tool_call>, <tool_response>, <|im_start|>] → provider qwen, mode=NATIVE, reason auto-detected from metadata (no explicit family anywhere). This is the architecture-validation core of Validate the generalized design against existing Qwen support #43.
  2. Full agent-loop round-trip over the real checkpoint (same DecoderGgufWeightLoaderQwenNetworkLoaderOptimizedLLMRuntime DIRECT path the CLI uses): resolution, ChatSession wiring, Qwen template rendering, and the agent loop all execute — but the run surfaced that the DSL Qwen runtime currently emits degenerate text on real checkpoints (garbage multilingual tokens at temp 0; 57-minute CPU run captured in the test report). That is an inference-quality problem squarely in Phase 4 readiness: validate DSL Qwen output against HF reference on real GGUFs #118's scope (validate DSL Qwen output vs HF reference), not a tool-calling-architecture problem — the test documents this and becomes the round-trip pin once Phase 4 readiness: validate DSL Qwen output against HF reference on real GGUFs #118 lands.

#44 — docs

  • README: new "Tool calling: native vs. generic support" section — resolution order (explicit → auto-detect → generic fallback), NATIVE vs GENERIC semantics, runtime registration pointers, and a compatibility matrix (family / provider / mode / wire format / detection source) covering llama3, qwen, gemma, gemma4, functiongemma, apertus, smollm, chatml, generic.
  • CHANGELOG [Unreleased] entry for the whole W2 stack.
  • Registry + diagnostics themselves already existed (34ec586) — with this, all four Tool calling follow-ups #44 checkboxes are covered; nothing needs to stay open as a follow-up.

Verification

  • :llm-agent:jvmTest 210 green; :llm-runtime:kllama:jvmTest green (gated tests skip cleanly without env); :llm-runtime:gemma-iree:jvmTest green (functiongemma provider registration against this stack)
  • :llm-core:apiCheck :llm-agent:apiCheck :llm-runtime:kllama:apiCheck green — no public API changes in this PR
  • Provider resolution verified for llama3 / chatml / qwen / gemma / gemma4 / apertus / smollm (llm-agent tests), functiongemma (gemma-iree tests), and live against a real Qwen GGUF (above)

Found while validating (pre-existing, unrelated): :llm-apps:kllama-cli and :llm-apps:skainet-cli fail dependency resolution on clean develop (Could not find sk.ainet.core:skainet-lang-core: — empty version on the app-module classpath), so CLI-level demo validation isn't currently possible; worth a small separate fix.

Refs #41 #42 #43 #44 #35 #118

🤖 Generated with Claude Code

…tests, Qwen real-GGUF validation, native-vs-generic docs

Final PR of the W2 stack (#35 epic):

- #41: AgentCli (chat/agent modes) now prints the same provider/mode/
  reason resolution diagnostics ToolCallingDemo already printed, so
  native-vs-generic selection is visible in every CLI mode.
- #42: ResolutionDiagnosticsTest pins resolveWithDiagnostics — explicit
  override reason, auto-detect reason, GENERIC fallback mode/reason, and
  extraction→diagnostics carry-through. (GGUF/HF extraction tests landed
  in the earlier PRs of this stack.)
- #43: QwenToolCallSmokeTest, env-gated on QWEN_MODEL_PATH. Two stages:
  (1) metadata extraction + auto-resolution against the real file —
  header-only, runs in seconds; verified PASSING against
  Qwen2.5-0.5B-Instruct-F16.gguf (family=qwen, arch=qwen2, hints
  [<tool_call>, <tool_response>, <|im_start|>] → provider qwen, NATIVE,
  no explicit family). (2) full agent-loop round-trip over the real
  checkpoint via the same DSL path the CLI uses — currently fails for a
  reason outside this architecture: the DSL Qwen runtime emits
  degenerate text on real checkpoints (tracked as #118); documented in
  the test so it becomes the round-trip pin once #118 lands.
- #44: README gains a "Tool calling: native vs. generic support"
  section with resolution-order docs and a family/provider/mode/format/
  detection compatibility matrix; CHANGELOG entry for the W2 stack.

Verification: llm-agent jvmTest 210 green; kllama jvmTest green
(gated tests skip cleanly without env vars); apiCheck green on
llm-core/llm-agent/kllama (no public API changes in this PR);
gemma-iree jvmTest green (functiongemma provider registration).

Note: :llm-apps:kllama-cli / :llm-apps:skainet-cli fail dependency
resolution on develop (empty engine version on app classpaths),
pre-existing and unrelated — demo validation therefore ran through the
env-gated smoke tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant