Reset pane offsets when a pane is respawned (fixes server SIGSEGV under control mode) - #5498
Open
rattray-ant wants to merge 1 commit into
Open
Reset pane offsets when a pane is respawned (fixes server SIGSEGV under control mode)#5498rattray-ant wants to merge 1 commit into
rattray-ant wants to merge 1 commit into
Conversation
respawn-pane and respawn-window free the pane's bufferevent and create a new, empty one, but wp->offset, wp->base_offset, wp->pipe_offset and the offsets control clients keep for the pane were left pointing into the old buffer. If a control client had not consumed all of the old output (it was not reading, or the pane's window had been moved out of its session), the next read from the new process made window_pane_get_new_data() return a pointer past the end of the new buffer with a wrapped size, and the server crashed in input_parse(). Discard anything control clients still had queued for the pane and reset all of the offsets to zero before the new event is created. A regress test covers both cases.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5499.
Issue description
respawn-paneandrespawn-windowcan leave the server one read away from a segfault. The precondition is a control-mode client (-Cor-CC) attached that has not consumed all of the target pane's output. The respawn returns 0; when the new process's output is next read, the server dies ininput_parse()and every session goes with it.It applies with or without
-k, and to a deadremain-on-exitpane. Two ways a client gets into that state:move-window'd out of the client's session, after which the client's offset for that pane never advances. This is how we hit it: iTerm2 attached withtmux -CC, long-lived CLI processes parked in a second session, one of them respawned.It reproduces every time on 3.6a and on
master(next-3.8at 851c5a9); with this patch every variant listed below leaves the server running, and the patch addsregress/respawn-pane-control-lag.sh, which fails onmasterand passes patched for both cases.Steps to reproduce:
Expected:
%0restarts andlist-sessionslistss1. Actual on unpatchedmaster:On 3.6a the server is sometimes still alive at the
pscheck and dies whenlist-sessionsconnects: with its only client blocked the pane is not read until then.move-window variant (client reading normally)
The server dies within milliseconds of the respawn; the client prints
%exit server exited unexpectedlyand exits 1.Results on
master851c5a9 (3.6a matches the unpatched column):respawn-pane -k-Cclient,respawn-pane -k-CC(under script(1) for a tty)respawn-window -kremain-on-exit on, plainrespawn-paneBoth builds keep running with no control client attached, with the client keeping up on a window still in its session, and when the pane is replaced by
new-window -d -kinstead of respawned; that last form is also the workaround until a fix is released. Patched, a client viewing the window still receives%output %1 respawned…after the respawn, and moving the window back intos1afterwards delivers the new process's output from its first byte.Backtrace from
gdb -pon the unpatchedmasterserver (--enable-debug) running the move-window variant, where the chatty pane is%1; libevent frames 5–7 andclient_main/maintrimmed:lenis9 - 4144as asize_t: 9 new bytes (respawned) minus the 4144 (12580 - 8436) still pending from the old buffer. Atmux -vvrun of the same variant on 3.6a (a separate run; the script's fixed output rate reproduces the same 4144-byte lag run to run) logsserver_client_check_pane_buffer: /dev/pts/68 has 0 bytes used and 0 left for %1immediately afterspawn_pane: the control client's stale offset, read against the new, empty buffer. Its last line before the log ends isinput_parse_buffer: %1 ground, 18446744073709547481 bytes: …, ~95,000 parser lines later.Cause, reading
masterat 851c5a9: on respawn,spawn_pane()frees the pane's bufferevent andwindow_pane_set_event()creates an empty one, whilewp->offset,wp->base_offset,wp->pipe_offsetand each control client'scontrol_paneoffsets keep their old values. Those agree with an empty buffer only if every consumer had caught up before the respawn.In either case above the lagging client pins
wp->base_offset, so the pane's own parser offset is L =wp->offset.used - wp->base_offset> 0 bytes into a buffer that is about to be emptied. In themove-windowcase the pin is permanent.control_write_output()returns at itswinlink_find_by_window()check, so the client's offset stops advancing;control_pane_offset()reports that stale offset toserver_client_check_pane_buffer(), which never drains past it.On the first read of N < L bytes from the new process,
window_pane_get_new_data()returnsEVBUFFER_DATA + Lwith sizeN - L, which wraps, andinput_parse_pane()walks off the buffer: the backtrace above.The change resets the offsets in
spawn_pane()'s respawn branch, where the old buffer is discarded. The pane's three offsets go to zero, and a newcontrol_reset_pane()does for each control client what cae229c (#5054) did incontrol_set_pane_off(): discard the blocks still queued for the pane, then copy the pane's offset intocp->offsetandcp->queued. It only touches acontrol_paneentry the client already has (control_get_pane()looks one up, never creates it), leaves the pane'sCONTROL_PANE_OFFandPAUSEDflags alone, and skips clients that have not finished identifying.It is in the respawn branch rather than in
window_pane_set_event()so it runs only when an old buffer existed; either place works. The regress test drives both cases with plain-Cclients: one whose output goes down a fifo that is never read while a second client keeps the pane being read, then a window moved out of the session both clients view.Not addressed here: in the
move-windowcase the parked pane's buffer is never drained, so server memory grows with its output until the window returns to a viewed session.Required information
tmux 3.6a(nixpkgs build; libevent 2.1.12, utf8proc 2.11.3) andtmux next-3.8built frommasterat 851c5a9 with--enable-debugLinux x86_64(kernel 6.12, Ubuntu 24.04 userland in a container)tmux -CCover ssh) in the incident; none for the scripts, whose control clients run on pipes$TERM:tmux-256colorinside and out (the scripts ran from a pane of an unrelated tmux server withTMUXunset); not recorded for the iTerm2 incidenttmux -vv -L bugtest -f /dev/null; the full server log is left out because it records the pane's environment, and the script regenerates it in seconds. No core file:core_patternnames a directory this container cannot create, hence thegdb -pbacktrace