Skip to content

Reset pane offsets when a pane is respawned (fixes server SIGSEGV under control mode) - #5498

Open
rattray-ant wants to merge 1 commit into
tmux:masterfrom
rattray-ant:respawn-reset-offsets
Open

Reset pane offsets when a pane is respawned (fixes server SIGSEGV under control mode)#5498
rattray-ant wants to merge 1 commit into
tmux:masterfrom
rattray-ant:respawn-reset-offsets

Conversation

@rattray-ant

@rattray-ant rattray-ant commented Aug 15, 2026

Copy link
Copy Markdown

Fixes #5499.

Issue description

respawn-pane and respawn-window can leave the server one read away from a segfault. The precondition is a control-mode client (-C or -CC) attached that has not consumed all of the target pane's output. The respawn returns 0; when the new process's output is next read, the server dies in input_parse() and every session goes with it.

It applies with or without -k, and to a dead remain-on-exit pane. Two ways a client gets into that state:

  • it is not reading its stdout (any amount of unread pane output is enough);
  • the pane's window was move-window'd out of the client's session, after which the client's offset for that pane never advances. This is how we hit it: iTerm2 attached with tmux -CC, long-lived CLI processes parked in a second session, one of them respawned.

It reproduces every time on 3.6a and on master (next-3.8 at 851c5a9); with this patch every variant listed below leaves the server running, and the patch adds regress/respawn-pane-control-lag.sh, which fails on master and passes patched for both cases.

Steps to reproduce:

#!/bin/sh
# Stock tools, no config, a private socket per run. TMUX_BIN=./tmux tests a build.
unset TMUX; B=${TMUX_BIN:-tmux}; T="$B -L bugtest$$ -f /dev/null"
trap '$T kill-server 2>/dev/null' 0
$T new-session -d -s s1 -x 80 -y 24 \
  "sh -c 'while :; do printf \"%0720d\n\" 0; sleep 0.01; done'"
PID=$($T display -p '#{pid}'); echo "server pid: $PID"
# control client: the first sleep holds its stdin open, the second never
# reads its stdout, so it falls behind on %0
sleep 30 | $B -L bugtest$$ -C attach -t s1 2>/dev/null | sleep 30 &
sleep 4
$T respawn-pane -k -t s1:0 'printf respawned; sleep 600'
echo "respawn-pane exit: $?"
sleep 3; ps -o pid= -p $PID >/dev/null && echo "server alive" || echo "server gone"
$T list-sessions; echo "list-sessions exit: $?"

Expected: %0 restarts and list-sessions lists s1. Actual on unpatched master:

server pid: 992786
respawn-pane exit: 0
server gone
no server running on /tmp/tmux-<uid>/bugtest992647
list-sessions exit: 1

On 3.6a the server is sometimes still alive at the ps check and dies when list-sessions connects: with its only client blocked the pane is not read until then.

move-window variant (client reading normally)

The server dies within milliseconds of the respawn; the client prints %exit server exited unexpectedly and exits 1.

#!/bin/sh
# Stock tools, no config, a private socket per run. TMUX_BIN=./tmux tests a build.
unset TMUX; B=${TMUX_BIN:-tmux}; T="$B -L bugtest$$ -f /dev/null"
trap '$T kill-server 2>/dev/null' 0
$T new-session -d -s s1 -x 80 -y 24 'sleep 600'
$T new-window -d -t s1: -n chatty \
  "sh -c 'while :; do printf \"%072d\n\" 0; sleep 0.05; done'"
PID=$($T display -p '#{pid}'); echo "server pid: $PID"
# control client attached to s1 and reading normally; sleep holds its stdin
sleep 30 | $B -L bugtest$$ -C attach -t s1 >client.out 2>&1 &
sleep 2                                 # client receives %output for %1
$T new-session -d -s s2 'sleep 600'
$T move-window -d -s s1:chatty -t s2:   # window leaves the client's session
sleep 3                                 # %1 keeps writing; the client's offset is stuck
$T respawn-pane -k -t s2:chatty 'printf respawned; sleep 600'
echo "respawn-pane exit: $?"
sleep 3; ps -o pid= -p $PID >/dev/null && echo "server alive" || echo "server gone"
$T list-sessions; echo "list-sessions exit: $?"
tail -n 2 client.out

Results on master 851c5a9 (3.6a matches the unpatched column):

Variant Unpatched Patched
first script: client not reading, respawn-pane -k SIGSEGV ok
move-window variant: -C client, respawn-pane -k SIGSEGV ok
same with -CC (under script(1) for a tty) SIGSEGV ok
same with respawn-window -k SIGSEGV ok
same, pane exits under remain-on-exit on, plain respawn-pane SIGSEGV ok

Both builds keep running with no control client attached, with the client keeping up on a window still in its session, and when the pane is replaced by new-window -d -k instead of respawned; that last form is also the workaround until a fix is released. Patched, a client viewing the window still receives %output %1 respawned… after the respawn, and moving the window back into s1 afterwards delivers the new process's output from its first byte.

Backtrace from gdb -p on the unpatched master server (--enable-debug) running the move-window variant, where the chatty pane is %1; libevent frames 5–7 and client_main/main trimmed:

Program received signal SIGSEGV, Segmentation fault.
input_parse (ictx=0x55c41bef8820, buf=0x55c41bea6af0 "", len=18446744073709547481) at input.c:980
980			ictx->ch = buf[off++];
#1  input_parse_buffer (wp=0x55c41bf04120, buf=0x55c41bea6af0 "", len=18446744073709547481) at input.c:1072
#2  input_parse_pane (wp=0x55c41bf04120) at input.c:1038
#3  window_pane_read_callback (bufev=<optimized out>, data=0x55c41bf04120) at window.c:1515
#4  bufferevent_run_readcb_ () from libevent_core-2.1.so.7
#8  proc_loop (…, loopcb=server_loop) at proc.c:227
#9  server_start (…) at server.c:257
(gdb) print wp->offset
$1 = {used = 12580}
(gdb) print wp->base_offset
$2 = 8436

len is 9 - 4144 as a size_t: 9 new bytes (respawned) minus the 4144 (12580 - 8436) still pending from the old buffer. A tmux -vv run of the same variant on 3.6a (a separate run; the script's fixed output rate reproduces the same 4144-byte lag run to run) logs server_client_check_pane_buffer: /dev/pts/68 has 0 bytes used and 0 left for %1 immediately after spawn_pane: the control client's stale offset, read against the new, empty buffer. Its last line before the log ends is input_parse_buffer: %1 ground, 18446744073709547481 bytes: …, ~95,000 parser lines later.

Cause, reading master at 851c5a9: on respawn, spawn_pane() frees the pane's bufferevent and window_pane_set_event() creates an empty one, while wp->offset, wp->base_offset, wp->pipe_offset and each control client's control_pane offsets keep their old values. Those agree with an empty buffer only if every consumer had caught up before the respawn.

In either case above the lagging client pins wp->base_offset, so the pane's own parser offset is L = wp->offset.used - wp->base_offset > 0 bytes into a buffer that is about to be emptied. In the move-window case the pin is permanent. control_write_output() returns at its winlink_find_by_window() check, so the client's offset stops advancing; control_pane_offset() reports that stale offset to server_client_check_pane_buffer(), which never drains past it.

On the first read of N < L bytes from the new process, window_pane_get_new_data() returns EVBUFFER_DATA + L with size N - L, which wraps, and input_parse_pane() walks off the buffer: the backtrace above.

The change resets the offsets in spawn_pane()'s respawn branch, where the old buffer is discarded. The pane's three offsets go to zero, and a new control_reset_pane() does for each control client what cae229c (#5054) did in control_set_pane_off(): discard the blocks still queued for the pane, then copy the pane's offset into cp->offset and cp->queued. It only touches a control_pane entry the client already has (control_get_pane() looks one up, never creates it), leaves the pane's CONTROL_PANE_OFF and PAUSED flags alone, and skips clients that have not finished identifying.

It is in the respawn branch rather than in window_pane_set_event() so it runs only when an old buffer existed; either place works. The regress test drives both cases with plain -C clients: one whose output goes down a fifo that is never read while a second client keeps the pane being read, then a window moved out of the session both clients view.

Not addressed here: in the move-window case the parked pane's buffer is never drained, so server memory grows with its output until the window returns to a viewed session.

Required information

  • tmux version: tmux 3.6a (nixpkgs build; libevent 2.1.12, utf8proc 2.11.3) and tmux next-3.8 built from master at 851c5a9 with --enable-debug
  • Platform: Linux x86_64 (kernel 6.12, Ubuntu 24.04 userland in a container)
  • Terminal: iTerm2 (tmux -CC over ssh) in the incident; none for the scripts, whose control clients run on pipes
  • $TERM: tmux-256color inside and out (the scripts ran from a pane of an unrelated tmux server with TMUX unset); not recorded for the iTerm2 incident
  • Logs: excerpt above from tmux -vv -L bugtest -f /dev/null; the full server log is left out because it records the pane's environment, and the script regenerates it in seconds. No core file: core_pattern names a directory this container cannot create, hence the gdb -p backtrace

respawn-pane and respawn-window free the pane's bufferevent and create a
new, empty one, but wp->offset, wp->base_offset, wp->pipe_offset and the
offsets control clients keep for the pane were left pointing into the
old buffer. If a control client had not consumed all of the old output
(it was not reading, or the pane's window had been moved out of its
session), the next read from the new process made
window_pane_get_new_data() return a pointer past the end of the new
buffer with a wrapped size, and the server crashed in input_parse().

Discard anything control clients still had queued for the pane and reset
all of the offsets to zero before the new event is created. A regress
test covers both cases.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Not Started

Development

Successfully merging this pull request may close these issues.

Server SIGSEGV after respawn-pane when a control client is behind on the pane

1 participant