fix(langgraph): recover agent outputs from the final message when no structured response exists - #260
Open
fede-kamel wants to merge 1 commit into
Open
Conversation
…structured response exists With ToolStrategy, LangChain only sets `structured_response` when the model calls the structured output tool. A model that calls a tool and then repeats the result as a plain message (the behaviour of the conformance test suite's deterministic LLM server, and of real models on some turns) ends the run without a structured response, and extract_outputs_from_invoke_result silently fell back to the declared defaults. For the CTS AgentNode scenario the default of `search_results` is ["name", "CEO", "country"], which is exactly the "field names instead of values" reported in oracle#224. The extractor now reads the final agent message when the structured response is missing or empty (AgentNodeExecutor seeds it with {}) and maps it to the declared outputs when it is compatible with them: a JSON object keyed by output name, the JSON value of a single non-string output, or the free text of a single string output. Precedence is defaults < recovered values < structured_response < values already in the result state. A warning is logged when outputs still cannot be filled and defaults are used. Fixes oracle#224. Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #224.
Root cause
The
["name", "CEO", "country"]returned by the CTS test is not the tool's field names leaking through: it is the declared default of thesearch_resultsoutput in the test's Agent Spec. LangChain'screate_agent(response_format=ToolStrategy(...))only setsstructured_responsewhen the model calls the structured-output tool; when the model answers in plain text instead, the run ends with no structured response andextract_outputs_from_invoke_resultfell through to the declared defaults.AgentNodeExecutoralso seededstructured_response: {}, which the extractor treated as a present-but-empty response.The CTS deterministic model answers the second turn with the tool result as plain text (it only understands provider-native
response_format), which is why Wayflow passes and LangGraph returns the default. Reproduced live against OCI Generative AI models as well (a model answering in prose after the tool call produces the same default), and deterministically with a fake chat model.Changes
langgraph/_node_execution.py:extract_outputs_from_invoke_resulttreats a missing or emptystructured_responseas absent and recovers the outputs from the final AI message when it is compatible with the declared schema: a JSON object keyed by output names, a JSON value for a single non-string output, or the text itself for a single string output (text content blocks are joined). Precedence is defaults, then recovered values, thenstructured_response, then values already present in the state. Aloggingwarning names the outputs that fell back to their defaults. Final messages carrying tool calls are never used.tests/adapters/langgraph/flows/test_agentnode_output_recovery.py: the CTS scenario for the three names with a fake chat model, the unchanged structured-response path, the default fallback with its warning, and unit cases of the extractor. Eight tests fail onmain, thirteen pass with the fix. No LLM calls.Verification
SKIP_LLM_TESTS=1 pytest tests/adapters/langgraph: 163 passed, 89 skipped.tests/run_tests.sh) reproduced locally on Python 3.10 through 3.14.Notes for reviewers
StructuredOutputGuardthat raises whenstructured_responseis absent; if it lands first, the guard should run after (or skip when) this recovery succeeds.extract_outputs_from_invoke_result(wraps the result into_json_value); whichever lands second needs a trivial rebase.tool_choice="any", and Cohere models on OCI rejecttool_choiceentirely, soToolStrategyfails for them at load time.