Skip to content

fix(langgraph): recover agent outputs from the final message when no structured response exists - #260

Open
fede-kamel wants to merge 1 commit into
oracle:mainfrom
fede-kamel:fix/langgraph-agentnode-outputs-224
Open

fix(langgraph): recover agent outputs from the final message when no structured response exists#260
fede-kamel wants to merge 1 commit into
oracle:mainfrom
fede-kamel:fix/langgraph-agentnode-outputs-224

Conversation

@fede-kamel

Copy link
Copy Markdown
Member

Fixes #224.

Root cause

The ["name", "CEO", "country"] returned by the CTS test is not the tool's field names leaking through: it is the declared default of the search_results output in the test's Agent Spec. LangChain's create_agent(response_format=ToolStrategy(...)) only sets structured_response when the model calls the structured-output tool; when the model answers in plain text instead, the run ends with no structured response and extract_outputs_from_invoke_result fell through to the declared defaults. AgentNodeExecutor also seeded structured_response: {}, which the extractor treated as a present-but-empty response.

The CTS deterministic model answers the second turn with the tool result as plain text (it only understands provider-native response_format), which is why Wayflow passes and LangGraph returns the default. Reproduced live against OCI Generative AI models as well (a model answering in prose after the tool call produces the same default), and deterministically with a fake chat model.

Changes

  • langgraph/_node_execution.py: extract_outputs_from_invoke_result treats a missing or empty structured_response as absent and recovers the outputs from the final AI message when it is compatible with the declared schema: a JSON object keyed by output names, a JSON value for a single non-string output, or the text itself for a single string output (text content blocks are joined). Precedence is defaults, then recovered values, then structured_response, then values already present in the state. A logging warning names the outputs that fell back to their defaults. Final messages carrying tool calls are never used.
  • tests/adapters/langgraph/flows/test_agentnode_output_recovery.py: the CTS scenario for the three names with a fake chat model, the unchanged structured-response path, the default fallback with its warning, and unit cases of the extractor. Eight tests fail on main, thirteen pass with the fix. No LLM calls.
  • Changelog entry under Bug fixes.

Verification

  • SKIP_LLM_TESTS=1 pytest tests/adapters/langgraph: 163 passed, 89 skipped.
  • Public CI steps (black, isort, flake8 + copyright, bandit, mypy, tests/run_tests.sh) reproduced locally on Python 3.10 through 3.14.

Notes for reviewers

  • PR feat(adapters/langgraph): make agent structured output not hang or silently vanish #210 adds a StructuredOutputGuard that raises when structured_response is absent; if it lands first, the guard should run after (or skip when) this recovery succeeds.
  • The schema-fidelity PR also edits extract_outputs_from_invoke_result (wraps the result in to_json_value); whichever lands second needs a trivial rebase.
  • Two related observations were left out of scope: some models emit the structured-output call in the same turn as the tool call under tool_choice="any", and Cohere models on OCI reject tool_choice entirely, so ToolStrategy fails for them at load time.

…structured response exists

With ToolStrategy, LangChain only sets `structured_response` when the
model calls the structured output tool. A model that calls a tool and
then repeats the result as a plain message (the behaviour of the
conformance test suite's deterministic LLM server, and of real models
on some turns) ends the run without a structured response, and
extract_outputs_from_invoke_result silently fell back to the declared
defaults. For the CTS AgentNode scenario the default of `search_results`
is ["name", "CEO", "country"], which is exactly the "field names instead
of values" reported in oracle#224.

The extractor now reads the final agent message when the structured
response is missing or empty (AgentNodeExecutor seeds it with {}) and
maps it to the declared outputs when it is compatible with them: a JSON
object keyed by output name, the JSON value of a single non-string
output, or the free text of a single string output. Precedence is
defaults < recovered values < structured_response < values already in
the result state. A warning is logged when outputs still cannot be
filled and defaults are used.

Fixes oracle#224.

Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com>
@fede-kamel
fede-kamel requested a review from a team September 11, 2026 16:22
@oracle-contributor-agreement oracle-contributor-agreement Bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

OCA Verified All contributors have signed the Oracle Contributor Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LangGraph AgentNode returns tool-output field names instead of values

1 participant