[TRTLLM-6842][feat] Support Response API for general purpose #9392
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JunyiXu-nv
force-pushed
the
dev-junyi-general-responses-api
branch
from
6088257 to
ceb7ded
Compare
JunyiXu-nv
force-pushed
the
dev-junyi-general-responses-api
branch
from
ceb7ded to
1378f31
Compare
Collaborator
Author
|
/bot run |
Contributor
📝 WalkthroughWalkthroughThis PR refactors the Responses API to support configurable reasoning and tool parsers, replacing direct Harmony adapter dependency with a flexible use_harmony flag. Changes span chat template tokenization control, reasoning parser whitespace handling, protocol/response format updates, server orchestration of streaming paths, and comprehensive responses utility refactoring with Harmony/non-Harmony processing pipelines. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant OpenAIServer as openai_server.py
participant ResponseUtils as responses_utils.py
participant HarmonyAdapter
participant TokenRenderer
rect rgb(200, 240, 255)
Note over OpenAIServer,ResponseUtils: use_harmony = true (Harmony path)
Client->>OpenAIServer: POST /responses
OpenAIServer->>ResponseUtils: request_preprocess<br/>(use_harmony=true)
ResponseUtils->>HarmonyAdapter: Initialize & prepare
OpenAIServer->>ResponseUtils: process_streaming_events<br/>(use_harmony=true, reasoning_parser, tool_parser)
ResponseUtils->>ResponseUtils: _construct_harmony_messages()
ResponseUtils->>HarmonyAdapter: Process & emit harmony events
ResponseUtils->>ResponseUtils: _apply_reasoning_parser()
ResponseUtils->>ResponseUtils: _apply_tool_parser()
ResponseUtils->>Client: Stream response.created, response.in_progress, ..., response.completed
end
rect rgb(240, 200, 255)
Note over OpenAIServer,ResponseUtils: use_harmony = false (Non-Harmony path)
Client->>OpenAIServer: POST /responses
OpenAIServer->>ResponseUtils: request_preprocess<br/>(use_harmony=false)
ResponseUtils->>ResponseUtils: _create_input_tokens()
OpenAIServer->>ResponseUtils: process_streaming_events<br/>(use_harmony=false, reasoning_parser, tool_parser)
ResponseUtils->>TokenRenderer: Render messages
ResponseUtils->>ResponseUtils: _apply_reasoning_parser()
ResponseUtils->>ResponseUtils: _apply_tool_parser()
ResponseUtils->>ResponseUtils: _create_output_content()
ResponseUtils->>Client: Stream response.created, response.in_progress, delta events, response.completed
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Key areas requiring attention:
Possibly related PRs
Suggested reviewers
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Tip 📝 Customizable high-level summaries are now available in beta!You can now customize how CodeRabbit generates the high-level summary in your pull requests — including its content, structure, tone, and formatting.
Example instruction:
Note: This feature is currently in beta for Pro-tier users, and pricing will be announced later. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Collaborator
|
PR_Github #25550 [ run ] triggered by Bot. Commit: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (4)
tensorrt_llm/serve/openai_server.py (1)
104-111: Guard againstself.model_configbeingNonebefore accessing.model_type
load_pretrained_configfailures are caught andself.model_configis set toNone(Lines 104–110), but a few lines laterself.use_harmonyis computed asself.model_config.model_type == "gpt_oss"(Line 146). If config loading fails andDISABLE_HARMONY_ADAPTERis not set, this will raise anAttributeErrorat server startup.Consider something like:
- except Exception: - logger.debug("Failed to load AutoConfig for %s", hf_tokenizer_path) - self.model_config = None + except Exception: + logger.debug("Failed to load AutoConfig for %s", hf_tokenizer_path) + self.model_config = None ... - if disable_harmony: - self.use_harmony = False - else: - self.use_harmony = (self.model_config.model_type == "gpt_oss") + if disable_harmony: + self.use_harmony = False + else: + self.use_harmony = ( + getattr(self.model_config, "model_type", None) == "gpt_oss" + )This preserves existing behavior when the config is available, while falling back to
use_harmony=Falseif it cannot be loaded.Also applies to: 140-146
tensorrt_llm/serve/responses_utils.py (3)
203-252: ConversationHistoryStore: mutable defaults, missing prev_resp_id handling, and possible infinite loop in_pop_conversationThere are three related correctness issues here:
Mutable default argument for
resp_msgs(Line 204–207).
resp_msgs= []creates a shared list across calls and conversations. Combined withself.conversations[conversation_id] = resp_msgs, different conversations can accidentally share the same underlying list.
prev_resp_idnot found still indexed (Lines 239–245).
You log a warning whenprev_resp_idisn’t inresponse_to_conversation, but then still do
conversation_id = self.response_to_conversation[prev_resp_id], which will raiseKeyError. That contradicts the “warn and continue” intent.
_pop_conversationcan be a no-op in a capacity loop (Lines 341–394).
If the conversation doesn’t contain a complete(user → assistant)(non-harmony) or(user → final)(harmony) span,start_index/end_indexend up such thatdel conversation[start_index:end_index + 1]deletes nothing. Becausestore_responsecalls_pop_conversationin awhile len(conversation) > conversation_capacityloop, this can become an infinite loop.I recommend fixing these together so capacity trimming is always safe and side‑effect free between conversations. For example:
- async def store_response(self, - resp: ResponsesResponse, - resp_msgs: Optional[ - Union[List[Message], - List[ChatCompletionMessageParam]]] = [], - prev_resp_id: Optional[str] = None) -> None: + async def store_response( + self, + resp: ResponsesResponse, + resp_msgs: Optional[ + Union[List[Message], List[ChatCompletionMessageParam]] + ] = None, + prev_resp_id: Optional[str] = None, + ) -> None: @@ - async with self.conversations_lock: + # Make a private copy so we never alias the caller's list + resp_msgs_list: List[Union[Message, ChatCompletionMessageParam]] = ( + list(resp_msgs) if resp_msgs is not None else [] + ) + + async with self.conversations_lock: conversation_id: str if resp_id in self.response_to_conversation: conversation_id = self.response_to_conversation[resp_id] - self.conversations[conversation_id].extend(resp_msgs) - elif prev_resp_id is not None: - if prev_resp_id not in self.response_to_conversation: - logger.warning( - f"Previous response id {prev_resp_id} not found in conversation store" - ) - - conversation_id = self.response_to_conversation[prev_resp_id] - self.conversations[conversation_id].extend(resp_msgs) + self.conversations[conversation_id].extend(resp_msgs_list) + elif prev_resp_id is not None and prev_resp_id in self.response_to_conversation: + conversation_id = self.response_to_conversation[prev_resp_id] + self.conversations[conversation_id].extend(resp_msgs_list) while len(self.conversations[conversation_id] ) > self.conversation_capacity: self._pop_conversation(resp_id) else: - conversation_id = _random_uuid() - self.conversations[conversation_id] = resp_msgs + if prev_resp_id is not None: + logger.warning( + "Previous response id %s not found in conversation store, " + "starting a new conversation", prev_resp_id + ) + conversation_id = _random_uuid() + self.conversations[conversation_id] = resp_msgs_list @@ - def _pop_conversation(self, resp_id) -> None: + def _pop_conversation(self, resp_id) -> None: @@ - conversation = self.conversations[conversation_id] - if len(conversation) == 0: - return - - is_harmony_conversation = isinstance(conversation[0], Message) - - def get_first_conversation_range_harmony(): - start_index = 0 - end_index = 0 - for i, msg in enumerate(conversation): - if msg.author.role == Role.USER: - start_index = i - elif msg.channel == "final": - end_index = i - break - - return start_index, end_index - - def get_first_conversation_range(): - start_index = 0 - end_index = 0 - for i, msg in enumerate(conversation): - if msg.get("role", "") == "user": - start_index = i - elif msg.get("role", "") == "assistant": - end_index = i - break - - return start_index, end_index - - start_index, end_index = 0, 0 - if is_harmony_conversation: - start_index, end_index = get_first_conversation_range_harmony() - else: - start_index, end_index = get_first_conversation_range() - - del conversation[start_index:end_index + 1] + conversation = self.conversations[conversation_id] + if not conversation: + return + + is_harmony_conversation = isinstance(conversation[0], Message) + + def get_first_conversation_range_harmony() -> tuple[int, int]: + start_index: Optional[int] = None + end_index: Optional[int] = None + for i, msg in enumerate(conversation): + if msg.author.role == Role.USER and start_index is None: + start_index = i + elif start_index is not None and msg.channel == "final": + end_index = i + break + if start_index is None: + # Fallback: drop from the beginning + start_index = 0 + if end_index is None: + # Fallback: drop up to the last message + end_index = len(conversation) - 1 + return start_index, end_index + + def get_first_conversation_range() -> tuple[int, int]: + start_index: Optional[int] = None + end_index: Optional[int] = None + for i, msg in enumerate(conversation): + role = msg.get("role", "") + if role == "user" and start_index is None: + start_index = i + elif start_index is not None and role == "assistant": + end_index = i + break + if start_index is None: + start_index = 0 + if end_index is None: + end_index = len(conversation) - 1 + return start_index, end_index + + start_index, end_index = ( + get_first_conversation_range_harmony() + if is_harmony_conversation else + get_first_conversation_range() + ) + + # Always remove at least one element to guarantee progress + del conversation[start_index:end_index + 1] Also applies to: 341-394 --- `402-495`: **Harmony message construction: mutable default for `prev_msgs`** `_construct_harmony_messages` uses a mutable default: ```python def _construct_harmony_messages( request: ResponsesRequest, prev_response: Optional[ResponsesResponse], prev_msgs: List[Message] = [], ) -> List[Message]:If any caller mutates
prev_msgs(e.g., appending), that mutation will leak across calls. Even if current code only reads it, this is brittle.Refactor to use
Noneand normalize inside:-def _construct_harmony_messages( - request: ResponsesRequest, - prev_response: Optional[ResponsesResponse], - prev_msgs: List[Message] = [], -) -> List[Message]: +def _construct_harmony_messages( + request: ResponsesRequest, + prev_response: Optional[ResponsesResponse], + prev_msgs: Optional[List[Message]] = None, +) -> List[Message]: @@ - messages: List[Message] = [] + messages: List[Message] = [] + if prev_msgs is None: + prev_msgs = []The rest of the logic can remain unchanged.
1054-1112:create_response: implicit Optional annotation and overall flowThe non-streaming response assembly looks correct: you reuse
generation_resultwhen provided, otherwiseawait generator, then choose Harmony/non-Harmony output builders and store toConversationHistoryStorewhen enabled.One minor typing issue:
create_time: int = None,This is an implicit
Optional[int]. Prefer making that explicit:- create_time: int = None, + create_time: Optional[int] = None,Everything else in this block LGTM.
🧹 Nitpick comments (14)
tensorrt_llm/llmapi/reasoning_parser.py (1)
50-62: Whitespace stripping only inparse– consider parity with streaming path
parse()now strips bothreasoning_contentandcontent, butparse_delta()leaves whitespace as-is when it finds</think>. This means non‑streaming and streaming parsing can diverge slightly at boundaries (leading/trailing newlines or spaces). If the goal is normalized reasoning/content across both paths, consider trimming in the finalparse_deltabranch as well, or explicitly documenting the intended difference.tensorrt_llm/inputs/utils.py (1)
564-577: Newenable_tokenizeflag is useful; just confirm tokenizer API compatibilityWiring
enable_tokenizethrough totokenizer.apply_chat_template(..., tokenize=enable_tokenize)is sound and preserves previous behavior when left at the defaultFalse. Two minor follow‑ups:
- Ensure all
TokenizerBase/TransformersTokenizerimplementations used here accept atokenizekeyword; otherwise this will raiseTypeErrorat runtime for custom tokenizers.- When
enable_tokenize=True, the underlying tokenizer may return token IDs (e.g.,List[int]), while the annotation still says(str | List[str]). If you plan to rely on the tokenized form elsewhere, consider broadening the return type hint for clarity.Also applies to: 596-604
tests/unittest/llmapi/apps/_test_openai_responses.py (2)
15-21: Parametrizedmodelfixture and per-model server args look goodThe module-scoped, parametrized
modelfixture and the conditionalargsconstruction (reasoning/tool parser flags based on model prefix) are a nice way to reuse the same tests across Harmony and non‑Harmony models. Just note that any future change in model naming conventions (e.g., different prefixes for Qwen/DeepSeek variants) will require updating these string checks; if that becomes painful, you might eventually want a small helper that maps model IDs to parser choices in one place.Also applies to: 24-38
58-84: Enhancedcheck_tool_callingdiagnostics – consider aggregating all content partsCapturing
reasoning_contentandmessage_contentfor use in assertion messages is helpful. Right now you only look atoutput.content[0].text; if multiple text parts are ever present, the error messages will show only the first. For more complete debugging, you could join all text parts (e.g.,"".join(part.text for part in output.content)), while keeping the rest of the logic unchanged.tensorrt_llm/serve/openai_server.py (1)
905-917: Responses API wiring looks consistent; clarify assumptions around parsers and non-Harmony pathThe
openai_responsespath now:
- Sends
use_harmony,reasoning_parser=self.llm.args.reasoning_parser, andtool_parser=self.tool_parserintoresponses_api_process_streaming_eventsandresponses_api_create_response.- Routes tokenizer/model_config/processor into
responses_api_request_preprocessonly whennot self.use_harmony.This is a good split between Harmony and non‑Harmony flows. A couple of points to verify:
- It assumes
self.llm.argsalways has areasoning_parserattribute; if that’s not guaranteed for all existing deployment configs, this will crash withAttributeError.- For the non‑Harmony path,
tokenizer,model_config, andprocessorbeingNonewill now be surfaced insideresponses_api_request_preprocess. If any models intentionally run without a HF config/processor, make sure that function either handlesNonegracefully or you gateopenai_responsesaccordingly.If those assumptions hold across your supported models, this wiring looks solid.
Also applies to: 937-946, 962-972
tensorrt_llm/serve/openai_protocol.py (2)
15-28: Helper bridgingResponseFormatTextConfigtoGuidedDecodingParams– validate field mapping against OpenAI typesThe new
_response_format_text_config_to_guided_decoding_paramscorrectly reuses_response_format_to_guided_decoding_paramsby constructing a localResponseFormat. Two things to double‑check againstopenai.types.responses:
- Using
getattr(text_format, "schema_", None)assumes the OpenAI SDK exposes the JSON schema fortext.formatunder that attribute (likely via an alias). If the SDK ever changes this (e.g., usesjson_schemainstead), guided decoding will silently stop working.ResponseFormat.typeis constrained to"text" | "json" | "json_schema" | "json_object" | "regex" | "ebnf" | "structural_tag". Ensure thattext_format.typeis always one of these values; if the OpenAI SDK introduces new format types,_response_format_to_guided_decoding_paramswill raise a ValueError.Given how tightly this depends on the OpenAI Python model definitions, it’s worth re‑verifying against the exact SDK version you target.
Also applies to: 226-236
809-839:ResponsesRequest.to_sampling_paramssemantics: explicit max_tokens and text-format guided decodingThis implementation:
- Sets
max_tokensonly fromself.max_output_tokens(or leaves itNone), ignoring any defaultmax_tokensthat might be present indefault_sampling_params.- Derives
temperature/top_pfrom request fields when set, otherwise fromdefault_sampling_params(falling back to_DEFAULT_SAMPLING_PARAMS).- Pulls
stop_token_idsexclusively fromdefault_sampling_params.- Builds
guided_decodingfromself.text.formatvia_response_format_text_config_to_guided_decoding_params.That’s coherent, but it slightly changes how global defaults can influence
max_tokensfor Responses requests. If you intend to allow a server‑side default formax_output_tokenswhen the client omits it, consider optionally wiring a"max_tokens"entry fromdefault_sampling_paramsintomax_tokenswhenself.max_output_tokensisNone. Otherwise, this is a clean separation of request‑level vs. default behavior.tensorrt_llm/serve/responses_utils.py (7)
260-303:store_messages: logging nit and behavior checkFunctionally this looks fine and matches the new “whole conversation messages” contract. Two minor points:
- The debug line
_responses_debug_log(f"ConversationHistoryStore storing msg:")is an f-string with no placeholders (Ruff F541). You can drop thefprefix.- When
prev_resp_idis not found inresponse_to_conversation, you silently start a new conversation. That’s probably OK, but it’s worth double-checking this matches the intended behavior givenstore_responsenow logs whenprev_resp_idis missing.If you want, I can propose a small diff to clean up the logging and optionally add a warning in the “start new conversation” branch.
767-813:_create_input_tokens: unusedprev_responseargument and dependency assumptions
prev_responseis in the signature but not used, which Ruff flags (ARG001). Either drop it or add a comment if you’re keeping it for symmetry/future use.- For the non-harmony path, this function assumes
tokenizer,model_config, andprocessorare non-None(passed fromrequest_preprocess). Ifuse_harmonyisFalsebut any of those is accidentallyNone,apply_chat_templatewill fail.If call sites in
openai_server.pyalways provide these for non-harmony models, this is fine. Otherwise, consider asserting or validating them here.
836-893:request_preprocess: split Harmony vs non-Harmony paths are sane; minor nit on debug f-stringsThe Harmony/non-Harmony branching and reuse of
ConversationHistoryStorefor both_create_input_tokens_harmonyand_create_input_tokenslook consistent. The final_responses_debug_log(_decode_tokens(...))is useful for debugging.The only small nit is
_responses_debug_log("======= Complete Inputs to model =======")etc. don’t need f-strings; if you’re cleaning up Ruff F541 warnings, you can dropfwhere present (e.g., the earlier “Prev msgs” logs).
1115-1236: Streaming state & events helper: overall design good; consider item_id managementThe
ResponsesStreamingStateTracker/ResponsesStreamingEventsHelperabstractions and the variety ofget_*_eventhelpers are nicely structured and keep per‑stream state centralized.One concern: in this file,
item_idis never set onResponsesStreamingEventsHelperbefore being used to construct items/events (e.g., inget_message_output_added_events/get_reasoning_output_added_events). Unless some external code mutatesstreaming_events_helper.item_idbetween calls, events will carry an empty string asitem_id.If the Responses API consumers rely on stable, non-empty item ids, you may want to set a default per output index, e.g.:
- def output_index_increment(self): - self.state_tracker.current_output_index += 1 + def output_index_increment(self): + self.state_tracker.current_output_index += 1 + # Initialize a fresh item_id for each new output index if not set + if not self.state_tracker.current_item_id: + self.state_tracker.current_item_id = _random_uuid()Or explicitly set
item_idinprocess_streaming_eventswhenever you begin a new output.Also,
_get_output_added_eventsis implemented as a generator but annotated as returningList[StreamingResponsesResponse]; you’re using it viayield from, so the generator signature is correct—only the return annotation/docstring are a bit misleading.
1323-1408:_should_send_done_events: behavior is reasonable; note cost of re-parsing full outputThe done-event decision logic (reasoning→text, text→tool_call, and end-of-generation fallbacks) is sound and aligns with the comments.
The only caveat is performance: on each streaming chunk, you call
_apply_reasoning_parser(..., streaming=False)and_apply_tool_parser(..., streaming=False)overoutput.text(the full accumulated text). Depending on sequence lengths and parser complexity, this could be non-trivial. If this becomes a bottleneck, consider incremental tracking of last-sent content rather than re-parsing from scratch each time.No immediate correctness issues here.
1642-1737:process_streaming_events: orchestration looks correct; minor nits
- The overall flow—emit
response.createdandresponse.in_progress, consume the async generator, delegate to Harmony vs non-Harmony streaming generators, then callcreate_responsewithgeneration_result—is solid and avoids double-awaiting the generator.- You always construct a
harmony_adapter = get_harmony_adapter()even whenuse_harmonyisFalse. It’s a minor cost but could be avoided if construction is non-trivial.If you care to micro-optimize:
- stream_request_id = f"responses-api-{request.request_id}" - harmony_adapter = get_harmony_adapter() + stream_request_id = f"responses-api-{request.request_id}" + harmony_adapter = get_harmony_adapter() if use_harmony else None @@ - if use_harmony: + if use_harmony: event_generator = _generate_streaming_event_harmony( - harmony_adapter=harmony_adapter, + harmony_adapter=harmony_adapter,(Or simply move
get_harmony_adapter()inside theif use_harmony:branch.)Otherwise, this block looks good.
117-120: Minor logging/style nits (f-strings without placeholders)Several debug logs use f-strings without any
{}placeholders, e.g.:_responses_debug_log(f"------- Parsing input -----------") ... _responses_debug_log(f"Prev msgs:")These are harmless but trigger Ruff F541. If you’re cleaning up lint:
- _responses_debug_log(f"------- Parsing input -----------") + _responses_debug_log("------- Parsing input -----------")Same idea for similar lines around 224, 277, 303, 864, etc.
Also applies to: 224-227, 277-281, 303-305, 864-867
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
tensorrt_llm/inputs/utils.py(2 hunks)tensorrt_llm/llmapi/reasoning_parser.py(1 hunks)tensorrt_llm/serve/openai_protocol.py(6 hunks)tensorrt_llm/serve/openai_server.py(3 hunks)tensorrt_llm/serve/responses_utils.py(21 hunks)tests/unittest/llmapi/apps/_test_openai_responses.py(5 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
tensorrt_llm/serve/openai_server.py (4)
tensorrt_llm/llmapi/llm.py (2)
tokenizer(741-745)tokenizer(748-749)tensorrt_llm/_torch/models/modeling_llava_next.py (2)
tokenizer(73-74)processor(81-82)tensorrt_llm/_torch/models/modeling_qwen2vl.py (2)
tokenizer(124-125)processor(132-133)tensorrt_llm/_torch/models/modeling_vila.py (2)
tokenizer(905-906)processor(909-910)
tensorrt_llm/serve/openai_protocol.py (1)
tensorrt_llm/sampling_params.py (1)
GuidedDecodingParams(15-37)
🪛 Ruff (0.14.5)
tensorrt_llm/serve/responses_utils.py
117-117: f-string without any placeholders
Remove extraneous f prefix
(F541)
206-206: Do not use mutable data structures for argument defaults
Replace with None; initialize within function
(B006)
224-224: f-string without any placeholders
Remove extraneous f prefix
(F541)
277-277: f-string without any placeholders
Remove extraneous f prefix
(F541)
303-303: f-string without any placeholders
Remove extraneous f prefix
(F541)
462-462: Do not use mutable data structures for argument defaults
Replace with None; initialize within function
(B006)
636-636: Avoid specifying long messages outside the exception class
(TRY003)
659-660: Avoid specifying long messages outside the exception class
(TRY003)
769-769: Unused function argument: prev_response
(ARG001)
864-864: f-string without any placeholders
Remove extraneous f prefix
(F541)
1063-1063: PEP 484 prohibits implicit Optional
Convert to T | None
(RUF013)
1297-1298: Avoid specifying long messages outside the exception class
(TRY003)
1451-1452: Avoid specifying long messages outside the exception class
(TRY003)
1454-1455: Avoid specifying long messages outside the exception class
(TRY003)
1720-1720: Avoid specifying long messages outside the exception class
(TRY003)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (5)
tests/unittest/llmapi/apps/_test_openai_responses.py (1)
145-149: Tool-call skips for DeepSeek-R1 and streaming guard are appropriate
- Skipping tool-call tests when
model.startswith("DeepSeek-R1")aligns with the current model capabilities and prevents misleading failures.- In the streaming tool-call test, explicitly asserting
function_call is not Nonebefore accessing.argumentsavoids silentAttributeErrors and produces a clearer failure if the server ever stops emitting tool calls.These changes improve test robustness without altering behavior for supported models.
Also applies to: 217-221, 250-261
tensorrt_llm/serve/openai_protocol.py (2)
874-892:StreamingResponsesResponseunion type is a good consolidation of streaming eventsThe
StreamingResponsesResponsealias consolidates all supported streaming event types (created/in‑progress/completed, output/content part added/done, reasoning deltas/done, code interpreter and web search events). This should simplify typing and downstream handling in streaming utilities likeprocess_streaming_eventswithout impacting runtime behavior.
911-912:ResponsesResponse.max_output_tokensnow optional and sourced from sampling paramsMaking
ResponsesResponse.max_output_tokensoptional and populating it fromsampling_params.max_tokensinfrom_requestaligns the serialized response with the actual generation configuration:
- If the client does not set
max_output_tokens, bothsampling_params.max_tokensand the response field remainNone.- If it is set, the same value flows through to sampling and back to the response.
This is a sensible representation of what was actually used during generation; just be aware that any implicit server‑side defaults for
max_tokenswill no longer be reflected here unless wired intoSamplingParamsupstream.Also applies to: 935-960
tensorrt_llm/serve/responses_utils.py (2)
623-690:_response_output_item_to_chat_completion_messageand_create_input_messageslook correctThe mapping from Responses items to
ChatCompletionMessageParam(including function/tool roles and function_call_output →tool) and the filtering of"reasoning"messages from history is consistent with the intended chat format. No functional issues stand out here.
895-957: Reasoning & tool parser helpers: logic OK, but streaming dict requirement later must match usage
_apply_reasoning_parserand_apply_tool_parsercorrectly:
- Lazily create parsers per
output_indexwhen a dict is supplied.- Support both full (
parse/detect_and_parse) and streaming (parse_delta/parse_streaming_increment) modes.- Fall back to returning the original text when no parser or tools are configured.
No issues here, but note that later
_generate_streaming_eventassumes thatreasoning_parser_dictcontains an entry foroutput_idx; see my separate comment there about guarding that usage whenreasoning_parser_idisNone.
Collaborator
|
PR_Github #25550 [ run ] completed with state |
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #25667 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #25667 [ run ] completed with state |
JunyiXu-nv
force-pushed
the
dev-junyi-general-responses-api
branch
from
6db1ac4 to
fd21c8a
Compare
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #25716 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #25716 [ run ] completed with state |
LinPoly
reviewed
Nov 26, 2025
LinPoly
approved these changes
Nov 26, 2025
Collaborator
LinPoly
left a comment
LinPoly
left a comment
There was a problem hiding this comment.
LGTM. Seems like the tool calling test for responses API can be moved to tests/unittest/llmapi/apps/_test_openai_tool_call.py, but anyway we can do it afterwards, like in the PR of supporting streaming tool call.
JunyiXu-nv
force-pushed
the
dev-junyi-general-responses-api
branch
from
ff9fd66 to
c6dd6f8
Compare
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #26092 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #26092 [ run ] completed with state |
JunyiXu-nv
force-pushed
the
dev-junyi-general-responses-api
branch
from
c6dd6f8 to
73cb11a
Compare
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #26351 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #26351 [ run ] completed with state |
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #26375 [ run ] triggered by Bot. Commit: |
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #26525 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #26525 [ run ] completed with state |
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #26534 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #26534 [ run ] completed with state |
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #26554 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #26554 [ run ] completed with state |
JunyiXu-nv
force-pushed
the
dev-junyi-general-responses-api
branch
from
f74ded0 to
4d2462d
Compare
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #26573 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #26573 [ run ] completed with state |
Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>
Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>
Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>
JunyiXu-nv
force-pushed
the
dev-junyi-general-responses-api
branch
from
4d2462d to
a4c6445
Compare
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #26613 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #26613 [ run ] completed with state |
Collaborator
Author
|
/bot run |
Collaborator
|
PR_Github #26681 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #26681 [ run ] completed with state |
MinaHuai
pushed a commit
to davidmlw/TensorRT-LLM
that referenced
this pull request
Dec 10, 2025
…VIDIA#8779) The performance results of some kernels could be easily affected by the warm/cold L2 cache status. To achieve more precise profiling results, the L2 cache is cleared for every execution by the circular buffer method for better benchmarking during autotuning. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> [None][infra] Waive failed cases for main branch on 11/25 (NVIDIA#9429) Signed-off-by: qqiao <qqiao@nvidia.com> [NVIDIA#8391][chore] test_perf.py to lock clocks read from gpu_configs.yml instead of max freq (NVIDIA#9409) Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com> [None][ci] Move more test stages to use OCI machines (NVIDIA#9395) Signed-off-by: Yanchao Lu <yanchaol@nvidia.com> Co-authored-by: Matt Lefebvre <matthewelefebvre@gmail.com> [None][feat] Improve TRTLLM MoE in small hidden size throughput cases (NVIDIA#9377) Signed-off-by: Anthony Chang <27950904+rosenrodt@users.noreply.github.com> [https://nvbugs/5537996][fix] Let KV cache manager block initialization be aware whether it is doing a dry run or not (NVIDIA#9093) Before this commit, the kv cache manager does the same regardless, which causes a mis-calculation in free memory available to allocate for the KV cache manager, hence causing a crash. This commit fixes this by letting KV cache manager initialization be aware whether it is doing the dry run or not. If it is a dry run, use the max_tokens setting that is already pre-calculated and filled into kv_cache_config.max_tokens. Signed-off-by: eopXD <yuehtingc@nvidia.com> [https://nvbugs/5667922][fix] Update long context evaluation config (NVIDIA#9426) Signed-off-by: mni <125171826+baize97@users.noreply.github.com> [None][fix] Mitigate test timeout issues (NVIDIA#9445) Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> [None][chore] Fix trtllm-eval for PyTorchLLM (NVIDIA#9427) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> [None][feat] Add a parser to layer-wise benchmarks (NVIDIA#9440) Signed-off-by: Tailing Yuan <yuantailing@gmail.com> [None][feat] Support custom chat template for tool calling (NVIDIA#9297) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com> [TRTLLM-8160][feat] Add draft token tree runtime on CDL (NVIDIA#8586) Signed-off-by: Yue Weng <25103990+yweng0828@users.noreply.github.com> [None][ci] waive a test (NVIDIA#9458) Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com> [https://nvbugs/5680905][fix] Relax the MMLU accuracy requirement for DS-v3.2 (NVIDIA#9439) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> [TRTLLM-8376][feat] top-p optimization (removes redundant softmax) (NVIDIA#9411) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com> [TRTLLM-9490][feat] use FlashInfer's top_k_sampling_from_probs (NVIDIA#9457) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com> [https://nvbugs/5647400] [fix] Enlarged the AllReduce workspace size to 64MB. Added AllReduce strategy to AD config. (NVIDIA#9145) Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com> [TRTLLM-909][feat] Overlap context chunks in pipeline parallel mode (NVIDIA#9308) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> [None][chore] AutoDeploy add multi stream moe pass to default.yaml (NVIDIA#9430) Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> [https://nvbugs/5685143][fix] avoid cudaFree overlap with cuda graph (NVIDIA#9438) Signed-off-by: Chuang Zhu <111838961+chuangz0@users.noreply.github.com> [None][chore] Bump version to 1.2.0rc5 (NVIDIA#9455) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com> [TRTLLM-8936][test] Add disagg and wideep multi-node multi-gpu test cases (NVIDIA#9356) Signed-off-by: FredricZ-2007 <226039983+fredricz-20070104@users.noreply.github.com> [None][ci] move some slow test cases of DGX-B200 to post merge (NVIDIA#9467) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> [TRTLLM-9293][feat] Enable partial weight loading to support streaming update weights (NVIDIA#9224) Signed-off-by: shuyix <219646547+shuyixiong@users.noreply.github.com> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <90828364+tensorrt-cicd@users.noreply.github.com> [TRTLLM-9264][fix] Add accuracy/unit tests/doc for phi4mm (NVIDIA#9246) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com> [https://nvbugs/5580099][fix] Cherry pick IMA issue fix from release/1.1 (NVIDIA#9032) Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com> [None][chore] Upgrade CuteDSL to 4.3.0 (NVIDIA#9444) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> [None][feat] Support MLA chunked prefill for DeepSeek V3.2 model (NVIDIA#9376) Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com> [None][feat] Add environment variable to force spec-dec number of accepted tokens (NVIDIA#9371) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com> [None][infra] Update allowed list 2025.11.25 (NVIDIA#9468) Signed-off-by: Yuanjing Xue <197832395+yuanjingx87@users.noreply.github.com> [None][infra] Fail the pipeline when slurm ssh dropped (NVIDIA#9157) Signed-off-by: Yuanjing Xue <197832395+yuanjingx87@users.noreply.github.com> [None][feat] AutoDeploy: Remove redundant copies in mamba layers (NVIDIA#9461) Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> Co-authored-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> [None][feat] AutoDeploy: Add A_log fusion for Mamba layers (NVIDIA#9422) Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> [None][ci] Waive blackwell test on spec gate. (NVIDIA#9502) Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com> [https://nvbugs/5608930][fix] Fix a typo (NVIDIA#9487) Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> [NVIDIA#9463][feat] Add revision option to trtllm commands (NVIDIA#9498) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com> [TRTLLM-9085][doc] fix math formula rendering issues (NVIDIA#9481) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> [None][chore] update comments in llm_args.py (NVIDIA#9472) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <90828364+tensorrt-cicd@users.noreply.github.com> [https://nvbugs/5680310][fix] Fix ctx only timed out test (NVIDIA#9410) Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com> [https://nvbugs/5547414][fix] enable case after using local cache model (NVIDIA#9473) Signed-off-by: Hui Gao <huig@nvidia.com> [None][fix] Replace PYTORCH_CUDA_ALLOC_CONF with PYTORCH_ALLOC_CONF to fix deprecation warning (NVIDIA#9294) Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com> [https://nvbugs/5698581][fix] Init draft tokens for CUDA graph dummy request (NVIDIA#9505) Signed-off-by: ziyixiong-nv <219238287+ziyixiong-nv@users.noreply.github.com> [None][infra] Waive failed case in pre-merge on 11/27 (NVIDIA#9507) Signed-off-by: qqiao <qqiao@nvidia.com> [TRTLLM-9513][docs] Qwen3 deployment guide (NVIDIA#9488) Signed-off-by: Lanyu Liao <laliao@laliao-mlt.client.nvidia.com> Co-authored-by: Lanyu Liao <laliao@laliao-mlt.client.nvidia.com> [None][chore] revert batch_size=1 to prevent timeout and lower accuracy reference by 0.12% as a WAR (NVIDIA#9447) Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com> Co-authored-by: Shi Xiaowei <39303645+Shixiaowei02@users.noreply.github.com> [TRTLLM-9279][infra] Use flexcache for gh200 nodes since they locate in Austin (NVIDIA#9405) Signed-off-by: qqiao <qqiao@nvidia.com> Signed-off-by: Emma Qiao <qqiao@nvidia.com> Co-authored-by: Yanchao Lu <yanchaol@nvidia.com> [cherry-pick][https://nvbugs/5670793][fix] Solve trtllm-serve launch_disaggregated issue (NVIDIA#9346) Signed-off-by: xxi <xxi@nvidia.com> [None][infra] Fix Slurm job script (NVIDIA#9508) Signed-off-by: Yuanjing Xue <197832395+yuanjingx87@users.noreply.github.com> [None][fix] change allreduce workspace dtype to torch.int64 to avoid overflow (NVIDIA#9479) Signed-off-by: Zhenhuan Chen <zhenhuanc@nvidia.com> [None][feat] add qwen3-next CI test of accuracy on BF16 and NVFP4 (NVIDIA#9330) Signed-off-by: jiant <107457950+JadoTu@users.noreply.github.com> [None][fix] fix TP support for DeepSeek-V3.2 on hopper (NVIDIA#9484) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> [TRTLLM-9389][chore] Refactor AlltoallMethodType. (NVIDIA#9388) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com> [https://nvbugs/5674665][chore] Add test coverage for https://nvbugspro.nvidia.com/bug/5674665 (NVIDIA#9518) Signed-off-by: eopXD <yuehtingc@nvidia.com> [TRTLLM-7288][infra] Download merged waive list in slurm script (NVIDIA#8999) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com> Signed-off-by: Yanchao Lu <yanchaol@nvidia.com> Co-authored-by: Yanchao Lu <yanchaol@nvidia.com> [https://nvbugs/5687820][fix] Remove self.abort() in DetokenizedGenerationResult (NVIDIA#9449) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> [NVIDIA#9150][feat] AutoDeploy Nemotron-Flash support (NVIDIA#9504) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> [None] [chore] Update to cutlass 4.3 (NVIDIA#8637) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> [https://nvbugs/5637037][chore] Update waive lists. (NVIDIA#9386) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com> Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> Co-authored-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <90828364+tensorrt-cicd@users.noreply.github.com> [TRTLLM-8970][infra] Fix generate report when has isolation test result (NVIDIA#8861) Signed-off-by: qqiao <qqiao@nvidia.com> Signed-off-by: Emma Qiao <qqiao@nvidia.com> [https://nvbugs/5685015][fix] Update invalid max_token test (NVIDIA#9435) Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com> [None][fix] Fix on-disk cache and revise logger/statistics for AutoTuner. (NVIDIA#9211) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> [https://nvbugs/5689658][test] Fix gpu lock issue running on cluster (NVIDIA#9441) Signed-off-by: yufeiwu <230315618+yufeiwu-nv@users.noreply.github.com> [None][chore] add spec_decoding configs in perf benchmark scripts and fix typos (NVIDIA#9533) Signed-off-by: Lanyu Liao <lancelly@users.noreply.github.com> Co-authored-by: Lanyu Liao <lancelly@users.noreply.github.com> [None][fix] Remove FP8 K/V buffer from TRTLLM sparse MLA attention kernel (NVIDIA#9529) Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com> [None] [chore] Enhancements and clean up to slurm scripts (NVIDIA#9493) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> [None][chore] Revert "[None][fix] change allreduce workspace dtype to torch.int64 t… (NVIDIA#9538) Signed-off-by: Zhenhuan Chen <zhenhuanc@nvidia.com> [None][infra] Waive failed cases for main branch on 11/28 (NVIDIA#9539) Signed-off-by: qqiao <qqiao@nvidia.com> [None][fix] Pass checkpoint_format to create_input_processor (NVIDIA#9521) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> [TRTLLM-9541][infra] Use artifactory mirror for download.pytorch.org (NVIDIA#9477) Signed-off-by: ZhanruiSunCh <184402041+ZhanruiSunCh@users.noreply.github.com> Signed-off-by: Zhanrui Sun <184402041+ZhanruiSunCh@users.noreply.github.com> Co-authored-by: Yanchao Lu <yanchaol@nvidia.com> [TRTLLM-9488][feat] add 'disable_flashinfer_sampling' config option (NVIDIA#9454) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com> [None][infra] Waive failed case in pre-merge on 11/28 (NVIDIA#9537) Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com> [None][perf] Helix: improve all-to-all perf for large CP size (NVIDIA#9494) Signed-off-by: Matthias Jouanneaux <mjoux@nvidia.com> Signed-off-by: Zheyu Fu <zheyuf@NVIDIA.com> Co-authored-by: Zheyu Fu <zheyuf@nvidia.com> [None][feat] support for more accurate AR calculation (NVIDIA#9323) Signed-off-by: binghanc <176802681+binghanc@users.noreply.github.com> [TRTLLM-9488][fix] llmapi references (NVIDIA#9547) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com> [NVIDIA#8948][feat] Support custom sharding config (NVIDIA#9143) Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <90828364+tensorrt-cicd@users.noreply.github.com> [None][chore] Weekly mass integration of release/1.1 -- rebase (NVIDIA#9522) Signed-off-by: yunruis <205571022+yunruis@users.noreply.github.com> Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> Signed-off-by: Wangshanshan <30051912+dominicshanshan@users.noreply.github.com> Signed-off-by: qgai <qgai@nvidia.com> Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com> Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com> Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com> Signed-off-by: Simeng Liu <simengl@nvidia.com> Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com> Signed-off-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com> Signed-off-by: Vincent Zhang <vinczhang@nvidia.com> Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com> Signed-off-by: Michal Guzek <mguzek@nvidia.com> Signed-off-by: Michal Guzek <moraxu@users.noreply.github.com> Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com> Signed-off-by: leslie-fang25 <leslief@nvidia.com> Signed-off-by: Shunkang <182541032+Shunkangz@users.noreply.github.co> Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> Co-authored-by: yunruis <205571022+yunruis@users.noreply.github.com> Co-authored-by: sunnyqgg <159101675+sunnyqgg@users.noreply.github.com> Co-authored-by: brb-nv <169953907+brb-nv@users.noreply.github.com> Co-authored-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com> Co-authored-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com> Co-authored-by: Simeng Liu <109828133+SimengLiu-nv@users.noreply.github.com> Co-authored-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com> Co-authored-by: Jin Li <59594262+liji-nv@users.noreply.github.com> Co-authored-by: Ivy Zhang <25222398+crazydemo@users.noreply.github.com> Co-authored-by: Vincent Zhang <vcheungyi@163.com> Co-authored-by: peaceh-nv <103117813+peaceh-nv@users.noreply.github.com> Co-authored-by: Michal Guzek <moraxu@users.noreply.github.com> Co-authored-by: Chang Liu <9713593+chang-l@users.noreply.github.com> Co-authored-by: Leslie Fang <leslief@nvidia.com> Co-authored-by: Shunkangz <182541032+Shunkangz@users.noreply.github.com> Co-authored-by: Shunkang <182541032+Shunkangz@users.noreply.github.co> Co-authored-by: QI JUN <22017000+QiJune@users.noreply.github.com> [TRTLLM-5971][feat] Integrate helix parallelism (NVIDIA#9342) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <90828364+tensorrt-cicd@users.noreply.github.com> [None][infra] - Request idle time exemption for OCI jobs (NVIDIA#9528) Signed-off-by: Yanchao Lu <yanchaol@nvidia.com> [None][infra] Wiave failed tests for main branch on 11/30 (NVIDIA#9555) Signed-off-by: qqiao <qqiao@nvidia.com> [None][fix] Fix port conflict in disagg tests (NVIDIA#9474) Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com> [None][ci] Split H100_PCIe-PyTorch-Post-Merge test stage (NVIDIA#9558) Signed-off-by: Yanchao Lu <yanchaol@nvidia.com> [None][ci] Split H100_PCIe-PyTorch-Post-Merge test stage (NVIDIA#9559) Signed-off-by: Yanchao Lu <yanchaol@nvidia.com> [TRTLLM-8958][feat] and [TRTLLM-8960]: create ConfigurableMoE and support TRTLLMGenFusedMoE as backend (NVIDIA#9486) [None] [feat] Optimize the algorithm part of RocketKV (NVIDIA#9333) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com> [https://nvbugs/5690172][fix] Fix Qwen3-235B ATP accuracy issue with PDL (NVIDIA#9530) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> [TRTLLM-6222][feat] Extend cute_dsl_nvfp4_gemm to sm103. (NVIDIA#9543) Signed-off-by: Mindy Li <11663212+limin2021@users.noreply.github.com> [None][fix] Correct virtual memory allocation alignment (NVIDIA#9491) Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <90828364+tensorrt-cicd@users.noreply.github.com> [https://nvbugs/5684703][fix] Unwaive disagg guided decoding test (NVIDIA#9466) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> [https://nvbugs/5503479][fix] Temporarily lower reference accuracy to stabilize CI (NVIDIA#9398) Signed-off-by: Pengbo Wang <221450789+pengbowang-nv@users.noreply.github.com> [None][chore] remove qwen3-next accuracy tests (NVIDIA#9534) Signed-off-by: jiant <107457950+JadoTu@users.noreply.github.com> [None][doc] fix mtp.py typo (NVIDIA#9307) Signed-off-by: liugaoji <757394026@qq.com> [None][feat] add chat template kwargs support to longbench-v2 (NVIDIA#9544) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> [NVIDIA#9496][fix] AutoDeploy: remove auto-tuner from nvfp4_gemm forward (NVIDIA#9497) Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> [None][fix] Replace hash method with unique_id for cutedsl MoE runners. (NVIDIA#9569) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> [None][chore] refactor disaggregated scripts to use named arguments (NVIDIA#9581) Signed-off-by: Zhenhuan Chen <zhenhuanc@nvidia.com> [TRTLLM-6222][feat] Several perf opt for cuteDSL nvf4 gemm (NVIDIA#9428) Signed-off-by: Yuhan Li <51736452+liyuhannnnn@users.noreply.github.com> [None][chore] reduce the layers of the `devel` docker image (NVIDIA#9077) Signed-off-by: Martin Marciniszyn Mehringer <11665257+MartinMarciniszyn@users.noreply.github.com> [https://nvbugs/5651854][infra] Enable perf metrics during accuracy testing (NVIDIA#9140) [None][fix] Skip Allreduce init for Attention DP (NVIDIA#9542) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> [None][test] [None][test] Waive main branch test failures 12/1 (NVIDIA#9566) Signed-off-by: Yanchao Lu <yanchaol@nvidia.com> [None][ci] Minor change for Slurm scripts (NVIDIA#9561) Signed-off-by: Yanchao Lu <yanchaol@nvidia.com> [TRTLLM-6768][infra] Fix params for not updating github status (NVIDIA#6747) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com> [None][infra] Update the pytest options after MI (NVIDIA#9579) Signed-off-by: qqiao <qqiao@nvidia.com> [TRTLLM-6756][feat] Add Beam Search to TorchSampler (NVIDIA#8509) Signed-off-by: Stefan Niebler <82932102+stnie@users.noreply.github.com> [None][chore] Defer exposing context parallel configs (NVIDIA#9552) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com> [TRTC-1943][feat] Env vars override support in LLM API (NVIDIA#9104) Signed-off-by: Venky Ganesh <23023424+venkywonka@users.noreply.github.com> [None][feat] AutoDeploy: Use the router gemm op for nemotron MOE (NVIDIA#9500) Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> [NVIDIA#9198][feat] Refactor dist ops in AutoDeploy (NVIDIA#9301) Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com> [None][fix] Prevent YAML partial kv_cache_config from incorrectly overriding the complete kv_cache_config (NVIDIA#9262) Signed-off-by: Yuening Li <62227368+Yuening-wa@users.noreply.github.com> [TRTLLM-9085][doc] fix math formula rendering issues in github (NVIDIA#9605) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> [None][feat] Unify nvfp4 gemm backend (NVIDIA#8963) Signed-off-by: Shijie Wang <jaywan@nvidia.com> Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> Signed-off-by: Shijie <jaywan@nvidia.com> Co-authored-by: Yukun He <23156053+hyukn@users.noreply.github.com> [None][feat] Add support for KVCache reuse for DSv32 (NVIDIA#9383) Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <90828364+tensorrt-cicd@users.noreply.github.com> [None][chroe] Polish qwen3-next modeling code. (NVIDIA#8902) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> [https://nvbugs/5703953][fix] Use random port for disagg tests (NVIDIA#9582) Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com> [None][fix] Waive gb200 (NVIDIA#9580) Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com> [FMDL-1328][feat] Add support for nano-v3 and super-v3 with pytorch backend (NVIDIA#9261) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com> [https://nvbugs/5582091][test] increase warmup times in testing for multi-gpu cases (NVIDIA#9578) Signed-off-by: Ruodi Lu <ruodil@users.noreply.github.com> Co-authored-by: Ruodi Lu <ruodil@users.noreply.github.com> [None][chore] Add failed cases into waives.txt (NVIDIA#9588) Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com> [https://nvbugs/5702793][fix] Fix uncontiguous tensor view (NVIDIA#9576) Signed-off-by: shuyix <219646547+shuyixiong@users.noreply.github.com> [None][infra] Waive failed cases for main branch (NVIDIA#9615) Signed-off-by: qqiao <qqiao@nvidia.com> [TRTLLM-9488][feat] use FlashInfer.sampling by default (NVIDIA#9545) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com> [None][infra] Update allowlist 2025/12/01 (NVIDIA#9616) Signed-off-by: Yuanjing Xue <197832395+yuanjingx87@users.noreply.github.com> [None][infra] Remove an invalid test name in waives.txt (NVIDIA#9620) Signed-off-by: qqiao <qqiao@nvidia.com> Lock the gpu clocks in L0 perf tests (NVIDIA#9585) Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com> [TRTLLM-9466][test] Evaluate helix parallelism with DSV3 Lite (NVIDIA#9597) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com> [None][fix] Extract GPU count from single-node stage names (NVIDIA#9599) Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com> [https://nvbugs/5667774][fix] Refine Piecewise Cuda Graph Condition for DP (NVIDIA#9393) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com> [TRTLLM-9144][fix] enhance RPC robustness (NVIDIA#8711) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com> Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com> Co-authored-by: Erin Ho <14718778+hchings@users.noreply.github.com> [https://nvbugs/5627710][fix] Fix synchronization bugs in KvCacheTransferManager that can cause corrupted blocks (NVIDIA#9056) Signed-off-by: thorjohnsen <41591019+thorjohnsen@users.noreply.github.com> Signed-off-by: Thor Johnsen <41591019+thorjohnsen@users.noreply.github.com> Co-authored-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com> Co-authored-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> [TRTLLM-8980][test] Clean up spec dec tests in test_llm_api_pytorch (NVIDIA#8889) Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> [NVIDIA#9150][feat] Add code for nano v3 to custom implementation in AD (NVIDIA#9465) * Why? We would like to show an alternative to monkey-patching in AutoDeploy. * What? This commit builds on the existing custom model implementation for NemotronH and adds the bits relevant for MoE layers. Part of NVIDIA#9150. Signed-off-by: William Zhang <133824995+2ez4bz@users.noreply.github.com> [NVIDIA#9150][feat] AutoDeploy: reviewer comments for NVIDIA#9150 (NVIDIA#9527) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> [https://nvbugs/5651854][fix] Fix dist-serving perf by clearing CPU affinity (NVIDIA#9549) Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> [NVIDIA#9550][feat] AutoDeploy: Add NVFP4 Cutlass MoE kernels (NVIDIA#9551) Signed-off-by: Neta Zmora <96238833+nzmora-nvidia@users.noreply.github.com> [https://nvbugs/5688388][fix] fix: Reducing num request in disagg test to speed up (NVIDIA#9598) Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com> [TRTLLM-8946][feat] Improved heuristics to detect shardable regions (NVIDIA#9200) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> Signed-off-by: greg-kwasniewski1 <213329731+greg-kwasniewski1@users.noreply.github.com> Co-authored-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> [NVIDIA#9632][feat] Support EXTRA_WHEEL_BUILD_ARGS during wheel build (NVIDIA#9633) Signed-off-by: Yu Chi Li <yuchil@nvidia.com> [None][chore] Waive test failing on pre-merge (NVIDIA#9638) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com> [None][chore] Remove traceback dump for multimodal input processor (NVIDIA#9634) Signed-off-by: Chang Liu (Enterprise Products) <9713593+chang-l@users.noreply.github.com> [None][chore] Fix trtllm-eval and move GroupedGemmInputsHelper (NVIDIA#9612) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> [https://nvbugs/5698434][fix] Use separate weight mapper for draft (NVIDIA#9607) Signed-off-by: Anurag Mukkara <134339030+amukkara@users.noreply.github.com> [TRTLLM-7101][infra] Reuse passed tests (NVIDIA#6894) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com> Co-authored-by: Yanchao Lu <yanchaol@nvidia.com> [None][test] Remove duplicate test cases (NVIDIA#9623) Signed-off-by: yufeiwu <230315618+yufeiwu-nv@users.noreply.github.com> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <90828364+tensorrt-cicd@users.noreply.github.com> [None][feat] Add RocketKV usage doc and e2e accuracy test on LongBenchV2 (NVIDIA#9572) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com> [TRTLLM-9242][doc] Add examples showcasing openai compatible APIs (NVIDIA#9520) Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com> [None][chore] AutoDeploy update cuda stream manager for multi-device (NVIDIA#9575) Signed-off-by: Suyog Gupta <41447211+suyoggupta@users.noreply.github.com> [TRTLLM-9391][chore] Automatically estimate required workspace. (NVIDIA#9535) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com> [https://nvbugs/5708475][fix] Fix e2e eval accuracy for helix parallelism (NVIDIA#9647) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com> [https://nvbugs/5561153][test] Fix log error for perf test (NVIDIA#9622) Signed-off-by: FredricZ-2007 <226039983+fredricz-20070104@users.noreply.github.com> [TRTLLM-8241][feat] Aliasing to comply to LlmArgs (NVIDIA#9586) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com> [None][chore] Add failed cases into waives.txt (NVIDIA#9593) Signed-off-by: Jie Li <lijie@nvidia.com> Co-authored-by: Jie Li <lijie@nvidia.com> [TRTLLM-6842][feat] Support Response API for general purpose (NVIDIA#9392) Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com> [None][test] Update Qwen3-next accuracy testing by setting the cuda … (NVIDIA#9613) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> [None][feat] update trtllm-gen nvfp4 kernels with better performance (NVIDIA#9510) Signed-off-by: Perkz Zheng <67892460+PerkzZheng@users.noreply.github.com> [None][doc] Replace the tensorrt icon with torch icon on overview.md (NVIDIA#9644) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> [https://nvbugs/5705197][chore] Unwaive timeout disagg tests (NVIDIA#9637) Signed-off-by: Patrice Castonguay <55748270+pcastonguay@users.noreply.github.com> [https://nvbugs/5552132][fix] Enable LoRa for GPT OSS Torch (NVIDIA#8253) Signed-off-by: Michal Guzek <mguzek@nvidia.com> [None][fix] Fix wide ep MoE error (NVIDIA#9642) Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com> [https://nvbugs/5702795][fix] Remove the warning message for aten.log. (NVIDIA#9665) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> [https://nvbugs/5693853][fix] Fix error handling when querying machin… (NVIDIA#9483) Signed-off-by: Gal Hubara Agam <96368689+galagam@users.noreply.github.com> [OMNIML-2932] [feat] nvfp4 awq support (NVIDIA#8698) Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com> [NVIDIA#9643][fix] AutoDeploy: fix nano sharding config (NVIDIA#9668) Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com> [NVIDIA#9147][feat] AutoDeploy: Draft Target Speculative Decoding (NVIDIA#9275) Signed-off-by: Govind Ramnarayan <105831528+govind-ramnarayan@users.noreply.github.com> [None][feat] Update Qwen3CodeToolParser to align tool-calling parameters (NVIDIA#9540) Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com> [TRTLLM-7181][infra] Generate test results when pytest timeout happens (NVIDIA#9396) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <90828364+tensorrt-cicd@users.noreply.github.com> [TRTLLM-9522][fix] restore `trtllm-serve mm_embedding_serve` (NVIDIA#9669) [TRTLLM-5093][infra] Write env variables to a file in the interactive debug session (NVIDIA#6792) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com> [None][fix] fix error when processing batches containing both text and mm data (NVIDIA#8381) Signed-off-by: Nekofish-L <liuxiangyang@mail.ustc.edu.cn> [TRTLLM-7073][feat] Support torch compile for PP for Llama and DeepSeekV3 (NVIDIA#7838) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com> [None][feat] Add weights initialization and context phase parser to layer-wise benchmarks (NVIDIA#9667) Signed-off-by: Tailing Yuan <yuantailing@gmail.com> [TRTLLM-8274][feat] Check if executor is shutdown in /health entrypoint (NVIDIA#9057) Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com> [NVIDIA#8733][feat] Add Llama4 MoE handling to AutoDeploy (NVIDIA#9556) Signed-off-by: Tal Cherckez <127761168+tcherckez-nvidia@users.noreply.github.com> Signed-off-by: tcherckez-nvidia <127761168+tcherckez-nvidia@users.noreply.github.com> Co-authored-by: Neta Zmora <nzmora@nvidia.com> [None][ci] unwaive tests (NVIDIA#9651) Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com> [None][feat] Add NIXL-LIBFABRIC support (NVIDIA#9225) Signed-off-by: Yoray Zack <62789610+zackyoray@users.noreply.github.com> Signed-off-by: zackyoray <yorayz@nvidia.com> [None][test] rename wide ep and disagg metric name in perf test (NVIDIA#9704) Signed-off-by: Ruodi Lu <ruodil@users.noreply.github.com> Co-authored-by: Ruodi Lu <ruodil@users.noreply.github.com> [https://nvbugs/5467531][fix] Unwaive fused_moe all to all test with … (NVIDIA#9617) Signed-off-by: Jin Li <59594262+liji-nv@users.noreply.github.com> [None][fix] Recover TRTLLM MoE Perf for DEP (NVIDIA#9562) Signed-off-by: Anthony Chang <27950904+rosenrodt@users.noreply.github.com> [None][chore] Add failed cases into waives.txt (NVIDIA#9662) Signed-off-by: Xin He (SW-GPU) <200704525+xinhe-nv@users.noreply.github.com> Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com> Signed-off-by: Yanchao Lu <yanchaol@nvidia.com> Co-authored-by: Yanchao Lu <yanchaol@nvidia.com> [None][fix] Fix TLLM_SPEC_DECODE_FORCE_NUM_ACCEPTED_TOKENS for MTP/EAGLE (NVIDIA#9608) Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com> [None][infra] Add container notices and documentation (NVIDIA#9185) Signed-off-by: Parker Drake <pdrake@nvidia.com> [TRTLLM-5312][infra] Add triton trigger rules (NVIDIA#6440) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com> [None][doc] Add feature docs for helix parallelism (NVIDIA#9684) Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com> [TRTLLM-9579][infra] Set mergeWaiveList stage UNSTABLE when there is any issue (NVIDIA#9692) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com> [None][doc] Added line about partial reuse (NVIDIA#7846) Signed-off-by: thorjohnsen <41591019+thorjohnsen@users.noreply.github.com> [TRTLLM-8920][feat] decouple disagg service from fastapi (NVIDIA#8714) Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com> [https://nvbugs/5633340][fix] start disagg workers and servers on free ports (NVIDIA#9694) Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com> [TRTLLM-9562] [doc] Add Deployment Guide for Kimi K2 Thinking on TensorRT LLM - Blackwell (NVIDIA#9711) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> [NVIDIA#9602][feat] AutoDeploy: Support TRTLLM Sampler (NVIDIA#9641) Signed-off-by: Govind Ramnarayan <105831528+govind-ramnarayan@users.noreply.github.com> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <90828364+tensorrt-cicd@users.noreply.github.com> [None] [tests] Unwaive EPLB tests (NVIDIA#9625) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> [https://nvbugs/5518713][test] Refactor core test lists by merging with llm_perf_cluster.yml (NVIDIA#9714) Signed-off-by: yufeiwu <230315618+yufeiwu-nv@users.noreply.github.com> [TRTLLM-7136][feat] Update load_weights method to include mapping parameter in checkpoint loaders (NVIDIA#9583) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> [None][refactor] Improve request processing function in sampler (NVIDIA#9671) Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com> [https://nvbugs/5670672][fix] Fix flaky KV connector tests (NVIDIA#9676) Signed-off-by: jthomson04 <jwillthomson19@gmail.com> [None][infra] Update allowed list 20251204 (NVIDIA#9718) Signed-off-by: Yuanjing Xue <197832395+yuanjingx87@users.noreply.github.com> [None][feat] AutoDeploy: Perf optimization for Attention and rmsnorm (NVIDIA#9719) Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> [None][chore] Waive flakey disagg tests (NVIDIA#9749) Signed-off-by: Mike Iovine <miovine@nvidia.com> [https://nvbugs/5601682][fix] Fix cacheTransceiver hang (NVIDIA#9311) Signed-off-by: Iman Tabrizian <10105175+tabrizian@users.noreply.github.com> Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> [TRTLLM-9199][docs] KV Connector Docs (NVIDIA#9325) Signed-off-by: jthomson04 <jwillthomson19@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> [TRTLLM-9160][doc] add doc to llm_runtime.py (NVIDIA#9482) Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com> Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> [None][doc] VDR 1.0 trtllm-serve doc enhancement (NVIDIA#9443) Signed-off-by: Pengyun Lin <81065165+LinPoly@users.noreply.github.com> Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> [TRTLLM-9086][doc] Clean up TODOs in documentation (NVIDIA#9292) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> [TRTLLM-9157][doc] Guided decoding doc improvement (NVIDIA#9359) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> [None][infra] Updated Linux installation guide (NVIDIA#9485) Signed-off-by: Yiqing Yan <yiqingy@nvidia.com> Co-authored-by: Yanchao Lu <yanchaol@nvidia.com> Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> [TRTLLM-9075][doc] refine the slurm examples (NVIDIA#9548) Signed-off-by: Yan Chunwei <328693+Superjomn@users.noreply.github.com> Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> [TRTLLM-9093][doc] update hyper links in overview (NVIDIA#9568) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> [TRTLLM-9092][doc] link to modelopt checkpoints in quick start guide (NVIDIA#9571) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com> Signed-off-by: Mike Iovine <6158008+mikeiovine@users.noreply.github.com> Signed-off-by: Mike Iovine <miovine@nvidia.com> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <90828364+tensorrt-cicd@users.noreply.github.com> [None][fix] Fix triton moe load_weight (NVIDIA#9649) Signed-off-by: shuyix <219646547+shuyixiong@users.noreply.github.com> [None][fix] fix a bug: deepseek_fp8_block_scales in TRTLLMGEN-MoE use 2D x_sf instead of 1D (NVIDIA#9658) Signed-off-by: xxi <xxi@nvidia.com> [TRTLLM-9372][feat] Enable CuteDSL MoE with Large EP (NVIDIA#9592) Signed-off-by: Enwei Zhu <21126786+syuoni@users.noreply.github.com> [TRTLLM-9522][chore] implement default `attach_multimodal_embeddings` (NVIDIA#9664) Signed-off-by: ixlmar <206748156+ixlmar@users.noreply.github.com> [TRTLLM-9660][feat] Convert cuteDSL GEMM to opt-in feature (NVIDIA#9682) Signed-off-by: Jonas Li <6110159+longlee0622@users.noreply.github.com> Co-authored-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> [None][fix] enable hmac in RPC (NVIDIA#9745) Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <90828364+tensorrt-cicd@users.noreply.github.com> [https://nvbugs/5703953][fix] Preserving ip:port for trtllm-serve before initializing llm (NVIDIA#9646) Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com> [None][infra] Waive failed cases for main branch on 12/07 (NVIDIA#9769) Signed-off-by: qqiao <qqiao@nvidia.com> [None][fix] Several minor fixes to CI setting (NVIDIA#9765) Signed-off-by: Yanchao Lu <yanchaol@nvidia.com> [OMNIML-3036][doc] Re-branding TensorRT-Model-Optimizer as Nvidia Model-Optimizer (NVIDIA#9679) Signed-off-by: Chenjie Luo <chenjiel@nvidia.com> [None][feat] Enable NCCL_SYMMETRIC as default fallback for AllReduce (NVIDIA#9314) Signed-off-by: Ludwig Schneider <lschneider@nvidia.com> [TRTLLM-9000][feat] Add multi-node Perf Tests into CI (NVIDIA#8800) Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com> [None][test] add ntp tolerance in time metrics verification (NVIDIA#9741) Signed-off-by: zhengd-nv <200704041+zhengd-nv@users.noreply.github.com> [TRTLLM-9603][feat] Enable ConfigurableMoE test in the CI (NVIDIA#9645) [https://nvbugs/5422621][test] Add GB 200 WIDEEP test case for RCCA 5422621 (NVIDIA#9506) Signed-off-by: FredricZ-2007 <226039983+fredricz-20070104@users.noreply.github.com> [None][fix] Fix two tuning cache miss issues. (NVIDIA#9743) Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> [None][infra] Check in most recent lock file from nightly pipeline Signed-off-by: TensorRT LLM <90828364+tensorrt-cicd@users.noreply.github.com> [TRTLLM-9706] [doc] Update wide EP documents (NVIDIA#9724) Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> [https://nvbugs/5666804][test] only adding sampler config for limited models (NVIDIA#9512) Signed-off-by: Ruodi Lu <ruodil@users.noreply.github.com> Co-authored-by: Ruodi Lu <ruodil@users.noreply.github.com> Co-authored-by: yufeiwu-nv <230315618+yufeiwu-nv@users.noreply.github.com> Co-authored-by: Larry Xu <197874197+LarryXFly@users.noreply.github.com> [None][infra] Waive failed cases for main on 12/08 (NVIDIA#9773) Signed-off-by: qqiao <qqiao@nvidia.com> [None][chore] Move the rocketkv e2e test to post-merge (NVIDIA#9768) Signed-off-by: Fanrong Li <23290157+lfr-0531@users.noreply.github.com> [None][chore] Enable tvm_ffi for cute dsl nvfp4_gemm to reduce host overhead. (NVIDIA#9690) Signed-off-by: Mindy Li <11663212+limin2021@users.noreply.github.com> [TRTLLM-9431][perf] Enable multistream for Linear Attention in Qwen3-… (NVIDIA#9696) Signed-off-by: nv-guomingz <137257613+nv-guomingz@users.noreply.github.com> [None][chore] Remove closed bugs (NVIDIA#9770) Signed-off-by: xinhe-nv <200704525+xinhe-nv@users.noreply.github.com> [None][infra] update mooncake in docker images (NVIDIA#9584) Signed-off-by: zhengd-nv <200704041+zhengd-nv@users.noreply.github.com> Signed-off-by: Zheng Duan <200704041+zhengd-nv@users.noreply.github.com> [None][test] Add Kimi k2 WIDEEP perf and accuracy cases (NVIDIA#9686) Signed-off-by: FredricZ-2007 <226039983+fredricz-20070104@users.noreply.github.com> Signed-off-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> Co-authored-by: Kaiyu Xie <26294424+kaiyux@users.noreply.github.com> [https://nvbugs/5527655][test] Add test case for RCCA 5527655 (NVIDIA#9511) Signed-off-by: FredricZ-2007 <226039983+fredricz-20070104@users.noreply.github.com> [http://nvbugs/5649010][fix] fix test_auto_scaling.py::test_worker_restart timeout (NVIDIA#9775) Signed-off-by: Lizhi Zhou <1432185+reasonsolo@users.noreply.github.com> [None][fix] Switch AutoDeploy's default allreduce strategy to NCCL (NVIDIA#9666) Signed-off-by: Eran Geva <19514940+MrGeva@users.noreply.github.com> [TRTLLM-9506][fix] Fix AR for DeepSeek-R1 2 model path (NVIDIA#9661) Signed-off-by: qgai <qgai@nvidia.com> ray + updatew works trtllm works in async env trtllm works in sync and async env ray + updatew works rebase to the updated verl server mode still cherry pick still cherry pick still cherry pick integrated http interface hang at RyExecutor create workers ray.remote clean code use tensorrt_llm.rlhf_utils Signed-off-by: Liwei Ma <liweim@nvidia.com> placement, asyncllm, and basic tests Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com> connect sleep and wakeup; Add support to pass None to update_weights Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com> Batching ctx for IFB scheduler Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com> accuracy WAR for TP>1: always use AllReduceStrategy.NCCL, refactored Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com> fix e2e integration Signed-off-by: Superjomn <328693+Superjomn@users.noreply.github.com> update asyncllm, other nits Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com> fix init setup Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com> Fix TRTLLMSampler logprobs perf Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com> fix and cleanup Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com> fix server Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com> Revert "Batching ctx for IFB scheduler" This reverts commit b51aac0 Signed-off-by: Yuan Tong <13075180+tongyuantongyu@users.noreply.github.com> update & address comments Signed-off-by: Erin Ho <14718778+hchings@users.noreply.github.com>
usberkeley
pushed a commit
to usberkeley/TensorRT-LLM
that referenced
this pull request
Dec 11, 2025
…9392) Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>
codego7250
pushed a commit
to codego7250/TensorRT-LLM
that referenced
this pull request
Dec 11, 2025
…9392) Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>
1 task
codego7250
pushed a commit
to codego7250/TensorRT-LLM
that referenced
this pull request
Dec 13, 2025
…9392) Signed-off-by: Junyi Xu <219237550+JunyiXu-nv@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
New Features
Improvements
✏️ Tip: You can customize this high-level summary in your review settings.
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
Details
run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]Launch build/test pipelines. All previously running jobs will be killed.
--reuse-test (optional)pipeline-id(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.--disable-reuse-test(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-PyTorch-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--test-backend "pytorch, cpp"(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".--detailed-log(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.--debug(OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-listparameter to access the appropriate container environment. Note: Does NOT update GitHub check status.For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.mdand the
scripts/test_to_stage_mapping.pyhelper.kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.