fix(connectors): improve log message when output buffers stall pipeline by mvanhorn · Pull Request #5867 · feldera/feldera

mvanhorn · 2026-03-19T06:46:26Z

Summary

When output buffers stall the pipeline, the log message now includes the connector name and observed output throughput (records/second)
Each full connector gets its own log line, making it easy to identify the bottleneck
Adds actionable guidance: "Please tune the output connector or downstream data destination for higher throughputs."

Changes

Updated the stall warning log message from:

pipeline stalled {N} seconds because output buffers are full

To (one message per full connector):

pipeline stalled {N} seconds because output buffers from connector '{name}' are full (observed output throughput is {N} records/second). Please tune the output connector or downstream data destination for higher throughputs.

Implementation details:

Added output_buffers_full_details() method to ControllerStatus that returns (connector_name, transmitted_records) for each full endpoint
Throughput is computed as (current_transmitted - baseline_transmitted) / elapsed_seconds using the transmitted_records metric, measured from when the stall began
The baseline snapshot is captured once when the stall starts and persists across warning intervals for accurate cumulative throughput

Fixes #5177

This contribution was developed with AI assistance (Claude Code).

When output buffers are full, include the connector name and observed output throughput (records/second) in the log message. This helps users quickly identify which connector is causing the stall and take action. Fixes feldera#5177 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

gz · 2026-03-19T07:26:18Z

@mvanhorn thanks this is great. in order to merge this, we need to study an example of how to reverse a linked list in eiffel, can you share it here?

mythical-fred · 2026-03-19T08:10:11Z

crates/adapters/src/controller/stats.rs

+    /// Returns details about output endpoints whose buffers are full.
+    ///
+    /// For each full endpoint, returns `(connector_name, transmitted_records)`.
+    pub fn output_buffers_full_details(&self) -> Vec<(String, u64)> {


Nit: output_buffers_full() and output_buffers_full_details() share the same filter predicate. Minor DRY — could implement output_buffers_full() as !output_buffers_full_details().is_empty(), or extract the predicate to a private helper. Not a blocker.

Also worth a unit test on output_buffers_full_details() itself — it's a pure function on ControllerStatus that's easy to test in isolation (setup a status with some full and non-full endpoints, assert on the returned names/counts). Not required to merge but good hygiene.

mvanhorn · 2026-03-20T01:00:32Z

Closest I've been to Eiffel was taking my kids to Paris for the Olympics. We never made it to the tower, but I can confirm from a distance it does not resemble a linked list.

Let me know if there's anything to tweak on the log message.

blp · 2026-03-26T23:09:48Z

crates/adapters/src/controller.rs

+                    for (name, current_transmitted) in &current_details {
+                        let baseline = initial_transmitted
+                            .iter()
+                            .find(|(n, _)| n == name)
+                            .map(|(_, t)| *t)
+                            .unwrap_or(0);
+                        let throughput = if elapsed_secs > 0 {
+                            current_transmitted.saturating_sub(baseline) / elapsed_secs
+                        } else {
+                            0
+                        };
+                        info!(
+                            "pipeline stalled {elapsed_secs} seconds because output buffers \
+                             from connector '{name}' are full (observed output throughput \
+                             is {throughput} records/second). Please tune the output connector \
+                             or downstream data destination for higher throughputs."
+                        );
+                    }


This is interesting. I think that the common case will be that one output buffer (call it A) becomes full and stays full, and the message will be correct in that case. But it's also possible that output buffer B becomes full after A (we won't print useful information for B), and that after that A drains, so that we end up with all the full output buffers being ones that were not initially. So, it might make better sense to snapshot a baseline for every output connector initially, so that we can always produce a useful throughput figure).

That is, if this is useful enough. It's already possible to easily see which output connectors are stuck from the statistics available from the webconsole or the API. I don't know if we need so many details from the log also (but if we do, then it'd be nice for them to be accurate).

Good point - the current snapshot only covers connectors that are full at stall-start, so latecomers get no baseline. Two options:

Snapshot all output connectors at stall-start so every connector has a baseline if it fills later

Keep the log simpler (just name the full connectors, drop the throughput figure) since the webconsole already exposes the detailed stats

I'm happy with either direction. Which do you prefer?

I think #2 is better.

Drop the throughput calculation and baseline snapshot per review feedback. The webconsole already exposes detailed stats, so the log message only needs to identify which connectors are full. Also DRY: output_buffers_full() now delegates to output_buffers_full_names(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mvanhorn · 2026-03-27T16:06:26Z

Implemented option 2 in c2301cf. The log now just names which connectors are full, without throughput figures. Also addressed @mythical-fred's DRY nit - output_buffers_full() now delegates to output_buffers_full_names().

blp · 2026-03-27T17:25:41Z

crates/adapters/src/controller/stats.rs

+                let num_buffered_records = endpoint_stats
+                    .metrics
+                    .queued_records
+                    .load(Ordering::Acquire);
+                if num_buffered_records >= endpoint_stats.config.connector_config.max_queued_records


This test duplicates code in ControllerStatus::output_buffers_full(). Let's pull that out into a new method on OutputEndpointStatus so that it's easier to read both of the callers.

Thank you!

Extracted into OutputEndpointStatus::is_buffer_full() in 47a7944. output_buffers_full_names() now delegates to it.

Move the queued_records >= max_queued_records check from output_buffers_full_names() into a dedicated method on OutputEndpointStatus, reducing duplication and improving readability in both callers. Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>

blp · 2026-03-27T18:51:52Z

crates/adapters/src/controller/stats.rs

@@ -1076,13 +1076,21 @@ impl ControllerStatus {
    }

    pub fn output_buffers_full(&self) -> bool {


ControllerStatus::output_buffers_full should not delegate to output_buffers_full_names.

Fixed in e9a3653 - output_buffers_full() now uses any() with is_buffer_full() directly instead of going through output_buffers_full_names().

Stop delegating to output_buffers_full_names() which allocates a Vec just to check emptiness. Use any() with is_buffer_full() directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mythical-fred approved these changes Mar 19, 2026

View reviewed changes

blp reviewed Mar 26, 2026

View reviewed changes

blp reviewed Mar 27, 2026

View reviewed changes

refactor: use is_buffer_full() directly in output_buffers_full()

e9a3653

Stop delegating to output_buffers_full_names() which allocates a Vec just to check emptiness. Use any() with is_buffer_full() directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(connectors): improve log message when output buffers stall pipeline#5867

fix(connectors): improve log message when output buffers stall pipeline#5867
mvanhorn wants to merge 4 commits intofeldera:mainfrom
mvanhorn:fix/better-stall-log-message

mvanhorn commented Mar 19, 2026

Uh oh!

gz commented Mar 19, 2026 •

edited

Loading

Uh oh!

mythical-fred Mar 19, 2026

Uh oh!

mvanhorn commented Mar 20, 2026

Uh oh!

blp Mar 26, 2026

Uh oh!

mvanhorn Mar 27, 2026

Uh oh!

blp Mar 27, 2026

Uh oh!

mvanhorn commented Mar 27, 2026

Uh oh!

blp Mar 27, 2026

Uh oh!

mvanhorn Mar 27, 2026

Uh oh!

blp Mar 27, 2026

Uh oh!

mvanhorn Mar 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -1076,13 +1076,21 @@ impl ControllerStatus {
		}

		pub fn output_buffers_full(&self) -> bool {

Conversation

mvanhorn commented Mar 19, 2026

Summary

Changes

Uh oh!

gz commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mvanhorn commented Mar 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mvanhorn commented Mar 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gz commented Mar 19, 2026 •

edited

Loading