Skip to content

[connectors] input backpressure does not limit librdkafka prefetch, so a stalled parser causes OOM #7610

Open

Description

@mihaibudiu

Cause

  1. A RecvThread calls the parser, and the parser runs the preprocessor (input.rs:1475). If the preprocessor blocks, the thread stops reading its partition queues.
  2. The backpressure thread pauses a connector only when is_full() is true (controller.rs:6638). is_full() compares buffered_records and buffered_bytes with the configured maximums (stats.rs:2587). The Kafka reader increments these counters only after it parses a record (input.rs:635, input.rs:749). The 1 MB per-partition limit also counts only parsed data (input.rs:1236).
  3. InputReader::memory() gives the librdkafka queued bytes (kafka.rs:595). Its only use is the metric input_connector_extra_memory_bytes (controller.rs:2082).
  4. RSS sets the memory-pressure level (runtime.rs:772). Memory pressure changes only the DBSP spill thresholds (for example runtime.rs:1430). It does not pause input connectors.
  5. The reader splits each partition into its own queue (input.rs:550). With split queues, librdkafka applies queued.max.messages.kbytes (default 65536, 64 MiB) to each partition. For this pipeline the limit is 40 x 8 x 64 MiB = 20 GiB, which is more than the memory budget.

Activity

  1. added theissue type on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    connectorsIssues related to the adapters/connectors crate

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions