Skip to content

fix(connectors): improve log message when output buffers stall pipeline #5867

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mvanhorn wants to merge 4 commits into
base: main
Choose a base branch
from
Open

fix(connectors): improve log message when output buffers stall pipeline #5867

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
866a343
fix(connectors): improve log message when output buffers stall pipeline
mvanhorn Mar 19, 2026
c2301cf
fix(connectors): simplify stall log to just name full connectors
mvanhorn Mar 27, 2026
47a7944
refactor: extract is_buffer_full() method on OutputEndpointStatus
mvanhorn Mar 27, 2026
e9a3653
refactor: use is_buffer_full() directly in output_buffers_full()
mvanhorn Mar 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 19 additions & 5 deletions crates/adapters/src/controller.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2526,7 +2526,7 @@ impl CircuitThread {
};
let _ = init_status_sender.send(Ok(self.controller.clone()));

let mut output_backpressure_warning = None;
let mut output_backpressure_warning: Option<LongOperationWarning> = None;
loop {
// Run received commands. Commands can initiate checkpoint
// requests, so attempt to execute those afterward. Executing a
Expand Down Expand Up @@ -2558,11 +2558,21 @@ impl CircuitThread {
debug!("circuit thread: park waiting for output buffer space");
let warning = output_backpressure_warning
.get_or_insert_with(|| LongOperationWarning::new(Duration::from_secs(1)));
let controller = &self.controller;
warning.check(|elapsed| {
info!(
"pipeline stalled {} seconds because output buffers are full",
elapsed.as_secs()
)
let full_names = controller.output_buffers_full_names();
let elapsed_secs = elapsed.as_secs();
if full_names.is_empty() {
info!(
"pipeline stalled {elapsed_secs} seconds because output buffers are full"
);
} else {
let names = full_names.join(", ");
info!(
"pipeline stalled {elapsed_secs} seconds because output buffers \
are full: {names}"
);
}
});
self.parker.park_deadline(warning.next_warning());
continue;
Expand Down Expand Up @@ -6375,6 +6385,10 @@ impl ControllerInner {
self.status.output_buffers_full()
}

fn output_buffers_full_names(&self) -> Vec<String> {
self.status.output_buffers_full_names()
}

fn warn_restoring() -> ControllerError {
static RATE_LIMIT: LazyLock<DefaultDirectRateLimiter> =
LazyLock::new(|| RateLimiter::direct(Quota::per_minute(nonzero!(10u32))));
Expand Down
29 changes: 22 additions & 7 deletions crates/adapters/src/controller/stats.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1076,13 +1076,23 @@ impl ControllerStatus {
}

pub fn output_buffers_full(&self) -> bool {
Copy link
Copy Markdown
Member

@blp blp Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ControllerStatus::output_buffers_full should not delegate to output_buffers_full_names.

Copy link
Copy Markdown
Author

@mvanhorn mvanhorn Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e9a3653 - output_buffers_full() now uses any() with is_buffer_full() directly instead of going through output_buffers_full_names().

self.output_status().values().any(|endpoint_stats| {
let num_buffered_records = endpoint_stats
.metrics
.queued_records
.load(Ordering::Acquire);
num_buffered_records >= endpoint_stats.config.connector_config.max_queued_records
})
self.output_status()
.values()
.any(|endpoint_stats| endpoint_stats.is_buffer_full())
}

/// Returns the names of output connectors whose buffers are currently full.
pub fn output_buffers_full_names(&self) -> Vec<String> {
self.output_status()
.values()
.filter_map(|endpoint_stats| {
if endpoint_stats.is_buffer_full() {
Some(endpoint_stats.endpoint_name.clone())
} else {
None
}
})
.collect()
}

pub fn parse_error(&self, endpoint_id: EndpointId, tag: Option<&str>, error: &ParseError) {
Expand Down Expand Up @@ -2411,6 +2421,11 @@ impl OutputEndpointStatus {
}
}

pub fn is_buffer_full(&self) -> bool {
self.metrics.queued_records.load(Ordering::Acquire)
>= self.config.connector_config.max_queued_records
}

pub fn is_busy(&self) -> bool {
self.metrics.buffered_records.load(Ordering::Relaxed) != 0
|| self.metrics.queued_records.load(Ordering::Relaxed) != 0
Expand Down