v1: Support KV events from connectors #19737

orozery · 2025-06-17T09:33:00Z

This PR adds a new scheduler-side connector API to collect KV cache events.
Additionally, we add a generic metadata field to KVCacheEvent to allow custom metadata on KV events.

Part of the work described in RFC #19854

gemini-code-assist

Summary of Changes

Hello @orozery, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the KV cache event system by extending the connector API to allow connectors to emit KV cache events. It also introduces a flexible metadata field for KVCacheEvent objects and integrates the collection of these new connector-emitted events into the scheduler's event publishing mechanism.

Highlights

KVCacheEvent Metadata: Introduced a generic metadata: Optional[dict[str, Any]] field to the KVCacheEvent base class, allowing custom data to be associated with KV cache events. The KVCacheEvent dataclass was also updated to make its fields keyword-only.
New KV Event API for Connectors: Added a new take_events() method to the KVConnector base class. This method provides an API for connectors to expose KV cache events they collect, returning an optional list of KVCacheEvent objects.
Scheduler Event Aggregation: Updated the scheduler's schedule method to collect and publish KV cache events. It now aggregates events from both the existing kv_cache_manager and the newly introduced KVConnector.take_events() method, ensuring all relevant events are published.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This PR introduces a new scheduler-side connector API to collect KV cache events, along with a generic metadata field for KVCacheEvent. The changes look good, with a suggestion for optimizing list concatenation in the scheduler and a few minor suggestions for improved code clarity.

vllm/v1/core/sched/scheduler.py

github-actions · 2025-06-17T09:37:28Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

njhill · 2025-07-31T16:58:50Z

vllm/distributed/kv_events.py

@vMaroon adding metadata here kind of defeats the purpose of having the strongly-typed event types. Could you elaborate on what it's for?

I don't see any generic use for this right now. The intention here was to add free metadata, starting with device-type. This can be explicitly added. Plus, static metadata information should be most likely discovered and not mounted per event.

I've removed the generic metadata field.
Instead, added a "medium" optional field which can be used to associate the event with a specific medium (e.g. gpu/cpu/disk).

mergify · 2025-08-03T17:47:57Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @orozery.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

njhill

LGTM

njhill · 2025-08-05T19:56:33Z

@orozery some test failures are due to the changes. It would probably be best to have medium default to None.

njhill

@orozery I think MultiConnector should also be updated to collect events from the aggregated collectors?

njhill

Thanks @orozery, just have a small nit.

njhill · 2025-08-08T18:10:23Z

vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py

nit: avoid list creation

Suggested change

for c in self._connectors:

events += c.take_events() or []

for c in self._connectors:

events.extend(c.take_events() or ())

orozery · 2025-08-10T05:11:05Z

pre-commit failed for some unknown reason

njhill · 2025-08-11T17:00:08Z

vllm/distributed/kv_transfer/kv_connector/v1/base.py

@orozery could we instead return Iterable[KVCacheEvent]? I think it may be a bit odd to have a generator function itself as part of the API.

njhill · 2025-08-12T15:42:41Z

vllm/distributed/kv_transfer/kv_connector/v1/base.py

Could we just return ()here? rather than having the interface method itself be a generator?

This commit adds a new scheduler-side connector API to collect KV cache events. Additionally, we add a medium field to KV events, to allow distinguishing KV events on different mediums (e.g. blocks stored on cpu, disk, or gpu (default)). Signed-off-by: Or Ozeri <oro@il.ibm.com>

Signed-off-by: Or Ozeri <oro@il.ibm.com>

orozery requested review from WoosukKwon, alexm-redhat, comaniac, njhill, robertgshaw2-redhat and ywang96 as code owners June 17, 2025 09:33

gemini-code-assist bot reviewed Jun 17, 2025

View reviewed changes

mergify bot added the v1 label Jun 17, 2025

gemini-code-assist bot reviewed Jun 17, 2025

View reviewed changes

vllm/v1/core/sched/scheduler.py Outdated Show resolved Hide resolved

orozery force-pushed the connector-kv-events branch from daec62d to cdd5414 Compare June 17, 2025 11:48

orozery mentioned this pull request Jun 19, 2025

[RFC]: KV cache offloading #19854

Open

1 task

vMaroon mentioned this pull request Jul 28, 2025

vLLM Native CPU Offloading Connector llm-d/llm-d-kv-cache#67

Closed

njhill reviewed Jul 31, 2025

View reviewed changes

orozery force-pushed the connector-kv-events branch from cdd5414 to c0f66eb Compare August 3, 2025 17:47

mergify bot added the needs-rebase label Aug 3, 2025

orozery force-pushed the connector-kv-events branch from c0f66eb to 393cff5 Compare August 3, 2025 17:50

mergify bot removed the needs-rebase label Aug 3, 2025

njhill approved these changes Aug 4, 2025

View reviewed changes

njhill added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 4, 2025

njhill requested changes Aug 5, 2025

View reviewed changes

orozery force-pushed the connector-kv-events branch 2 times, most recently from 1f3e942 to ab12890 Compare August 6, 2025 17:05

mergify bot added the documentation Improvements or additions to documentation label Aug 6, 2025

orozery force-pushed the connector-kv-events branch from ab12890 to 36bec03 Compare August 6, 2025 17:13

njhill approved these changes Aug 8, 2025

View reviewed changes

orozery force-pushed the connector-kv-events branch from 36bec03 to 448212f Compare August 10, 2025 04:59

njhill reviewed Aug 11, 2025

View reviewed changes

orozery force-pushed the connector-kv-events branch 5 times, most recently from 76965d6 to 39dc6c0 Compare August 11, 2025 19:12

njhill reviewed Aug 12, 2025

View reviewed changes

orozery force-pushed the connector-kv-events branch from 39dc6c0 to d87c649 Compare August 12, 2025 17:51

njhill approved these changes Aug 12, 2025

View reviewed changes

orozery force-pushed the connector-kv-events branch from d87c649 to ebca231 Compare August 14, 2025 12:36

njhill added 2 commits August 22, 2025 14:23

Merge branch 'main' into connector-kv-events

d7ec28c

Merge branch 'main' into connector-kv-events

d1a4280

njhill enabled auto-merge (squash) August 28, 2025 23:20

Merge branch 'main' into connector-kv-events

9887cae

njhill disabled auto-merge August 31, 2025 18:45

njhill enabled auto-merge (squash) August 31, 2025 18:45

njhill merged commit 14b4326 into vllm-project:main Sep 1, 2025
45 checks passed

AlpinDale mentioned this pull request Sep 2, 2025

feat: support KV events from connectors aphrodite-engine/aphrodite-engine#1458

Merged

orozery mentioned this pull request Sep 9, 2025

[KV offload][1/N] Introduce an offloading component #19848

Merged

eicherseiji pushed a commit to eicherseiji/vllm that referenced this pull request Sep 9, 2025

v1: Support KV events from connectors (vllm-project#19737)

577e68c

Signed-off-by: Or Ozeri <oro@il.ibm.com>

FeiDaLI pushed a commit to FeiDaLI/vllm that referenced this pull request Sep 25, 2025

v1: Support KV events from connectors (vllm-project#19737)

0ef0f42

Signed-off-by: Or Ozeri <oro@il.ibm.com>

-        for c in self._connectors:
-            events += c.take_events() or []
+        for c in self._connectors:
+            events.extend(c.take_events() or ())

Uh oh!

v1: Support KV events from connectors #19737

v1: Support KV events from connectors #19737

Uh oh!

Conversation

orozery commented Jun 17, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

github-actions bot commented Jun 17, 2025

Uh oh!

njhill Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

vMaroon Aug 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

orozery Aug 3, 2025

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Aug 3, 2025

Uh oh!

njhill left a comment

Choose a reason for hiding this comment

Uh oh!

njhill commented Aug 5, 2025

Uh oh!

njhill left a comment

Choose a reason for hiding this comment

Uh oh!

njhill left a comment

Choose a reason for hiding this comment

Uh oh!

njhill Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

orozery commented Aug 10, 2025

Uh oh!

njhill Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

orozery Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

njhill Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

orozery commented Jun 17, 2025 •

edited by github-actions bot

Loading

vMaroon Aug 3, 2025 •

edited

Loading