You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[connectors] input backpressure does not limit librdkafka prefetch, so a stalled parser causes OOM #7610
A RecvThread calls the parser, and the parser runs the preprocessor (input.rs:1475). If the preprocessor blocks, the thread stops reading its partition queues.
The backpressure thread pauses a connector only when is_full() is true (controller.rs:6638). is_full() compares buffered_records and buffered_bytes with the configured maximums (stats.rs:2587). The Kafka reader increments these counters only after it parses a record (input.rs:635, input.rs:749). The 1 MB per-partition limit also counts only parsed data (input.rs:1236).
InputReader::memory() gives the librdkafka queued bytes (kafka.rs:595). Its only use is the metric input_connector_extra_memory_bytes (controller.rs:2082).
RSS sets the memory-pressure level (runtime.rs:772). Memory pressure changes only the DBSP spill thresholds (for example runtime.rs:1430). It does not pause input connectors.
The reader splits each partition into its own queue (input.rs:550). With split queues, librdkafka applies queued.max.messages.kbytes (default 65536, 64 MiB) to each partition. For this pipeline the limit is 40 x 8 x 64 MiB = 20 GiB, which is more than the memory budget.
Activity