Stuck-thread watchdog self-dump segfaults on NULL stconn->sedesc during an httpclient L7 retry #3490

Detailed Description of the Problem

haproxy crashes with SIGSEGV (exit code 139) under load, when using
core.httpclient() from Lua against a backend that sometimes responds
slower than the client-side timeout.

The crash is in HAProxy's own stuck-thread diagnostic self-dump, not in
Lua or application code. Backtraces from a debug build consistently show:

sc_conn() (include/haproxy/stconn.h)
  <- __strm_dump_to_buffer() (src/stream.c, the "conn = sc_conn(strm->scb);"
     line right after printing "backend=...")
  <- strm_dump_to_buffer()
  <- ha_task_dump() (src/debug.c)
  <- ha_thread_dump_one()
  <- ha_stuck_warning()
  <- wdt_handler() (src/wdt.c)

sc_conn() calls sc_ep_test(), which dereferences sc->sedesc
unconditionally. At the moment of the crash, the backend stream connector
(strm->scb) is in state SC_ST_ASS (server assigned, connection not yet
attached) with sedesc == NULL — it's caught mid-teardown inside
do_l7_retry() (src/http_ana.c) -> sc_reset_endp() -> sc_detach_endp()
-> the H1 mux's detach path, at the exact instant the per-thread CPU-time
watchdog (governed by warn-blocked-traffic-after) decides the thread has
been unresponsive too long and fires its own diagnostic dump on itself
(is_caller=1 in every observed case -- the thread interrupts itself).

This is specifically reachable via core.httpclient() because HAProxy's
internal <HTTPCLIENT> proxy hardcodes L7 retries on for connection
failure, disconnection, and response timeout
(src/http_client.c: px->retry_type |= PR_RE_CONN_FAILED | PR_RE_DISCONNECTED | PR_RE_TIMEOUT;), so any httpclient call whose
backend is slower than its configured timeout goes through
do_l7_retry() -- landing it in exactly this window.

Expected Behavior

The stuck-thread self-diagnostic should log its warning and continue
running, never crash the process.

Steps to Reproduce the Behavior

Fully self-contained: the official haproxy:3.4.3 Docker image, Python 3
(stdlib only), and curl. No other dependencies.

1. Create the files:

mkdir -p /tmp/hap-repro && cd /tmp/hap-repro

cat > haproxy.cfg <<'EOF'
global
    nbthread 1
    lua-load /usr/local/etc/haproxy/trigger.lua
    # Minimum allowed value: makes the stuck-thread self-diagnostic dump run
    # on the smallest possible stall, to reproduce quickly under load.
    warn-blocked-traffic-after 1ms

defaults
    mode http
    timeout client  5s
    timeout connect 1s
    timeout server  5s
    retries 1

frontend fe
    bind *:8080
    http-request lua.trigger
    http-request return status 204
EOF

cat > trigger.lua <<'EOF'
-- core.httpclient()'s internal "<HTTPCLIENT>" proxy hardcodes
-- retry_type = PR_RE_CONN_FAILED|PR_RE_DISCONNECTED|PR_RE_TIMEOUT
-- (src/http_client.c, httpclient_precreate_proxy). Setting a request
-- timeout far shorter than the backend's response time guarantees every
-- call hits PR_RE_TIMEOUT and runs do_l7_retry() (src/http_ana.c).
core.register_action("trigger", {"http-req"}, function(txn)
    local hc = core.httpclient()
    hc:get{url = "http://127.0.0.1:9099/", timeout = 5}
end)
EOF

cat > slow_backend.py <<'EOF'
#!/usr/bin/env python3
"""Synthetic backend that always responds slower than the client's timeout,
so every request to it triggers HAProxy's httpclient L7-retry path."""
import http.server
import time

RESPONSE_DELAY_S = 0.05  # 50ms; must exceed trigger.lua's 5ms client timeout


class SlowHandler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        time.sleep(RESPONSE_DELAY_S)
        self.send_response(200)
        self.send_header("Content-Length", "2")
        self.end_headers()
        self.wfile.write(b"ok")

    def log_message(self, fmt, *args):
        pass  # keep stdout quiet during the stress run


if __name__ == "__main__":
    http.server.ThreadingHTTPServer(("127.0.0.1", 9099), SlowHandler).serve_forever()
EOF

cat > load.sh <<'EOF'
#!/usr/bin/env bash
# Hammers the repro frontend with concurrent requests for DURATION seconds.
# Only needs curl. Higher CONCURRENCY reproduces the crash faster.
set -u
DURATION=${1:-60}
CONCURRENCY=${2:-50}
END=$(( $(date +%s) + DURATION ))

worker() {
    while [ "$(date +%s)" -lt "$END" ]; do
        curl -s -o /dev/null "http://127.0.0.1:8080/"
    done
}

for _ in $(seq 1 "$CONCURRENCY"); do
    worker &
done
wait
EOF
chmod +x slow_backend.py load.sh

2. Start the synthetic slow backend:

python3 slow_backend.py &

3. Start HAProxy (official image, host networking so it can reach the backend):

docker run -d --name haproxy-repro --network host \
  -v "$PWD/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro" \
  -v "$PWD/trigger.lua:/usr/local/etc/haproxy/trigger.lua:ro" \
  haproxy:3.4.3

4. Generate load (4 minutes, 400 concurrent requesters):

./load.sh 240 400

5. Check the outcome:

docker inspect haproxy-repro --format 'exitcode={{.State.ExitCode}}'
docker logs haproxy-repro | tail -20

Observed Result

Within the load window, docker logs haproxy-repro shows repeated
WARNING! thread 1 has stopped processing traffic for N milliseconds blocks
(the stuck-thread self-diagnostic firing as designed), followed by:

[ALERT]    (1) : Current worker (8) exited with code 139 (Segmentation fault)
[WARNING]  (1) : A worker process unexpectedly died and this can only be
explained by a bug in haproxy or its dependencies.
Please check that you are running an up to date and maintained version of
haproxy and open a bug report.
[ALERT]    (1) : exit-on-failure: killing every processes with SIGTERM

Verified reproducing this locally: the crash landed within a single
4-minute, 400-concurrency run (docker inspect showed exitcode=139);
~100 stuck-thread warnings were logged before the fatal one. Raising
concurrency and/or duration further should only make it land faster/more
reliably -- it's a timing race between the L7-retry detach window and the
watchdog's async signal, not something that needs exact tuning.

Do you have any idea what may have caused this?

Yes -- see the analysis above. sc_conn()'s caller in
__strm_dump_to_buffer() (src/stream.c) doesn't guard against a NULL
sedesc on the backend stream connector, which is a legitimate transient
state during do_l7_retry()'s teardown/re-attach sequence.

This looks like the same general defect class as two prior Coverity
reports in this codebase that also flagged unguarded sc_conn()/sedesc
access (#2156 in sc_detach_endp(), #3046 in
cli_io_handler_dump_sess()) -- both were fixed at their own call sites
only; this one (inside the stuck-thread dump path) doesn't appear to have
been addressed.

Do you have an idea how to solve the issue?

Guard the conn = sc_conn(strm->scb); call (and its scf counterpart a
few lines above) in __strm_dump_to_buffer() against a NULL sedesc,
mirroring the NULL check already present a few lines later in the same
function for strm->txn->http->uri. Alternatively/additionally, harden
sc_conn()/sc_ep_test() themselves.

What is your configuration?

See haproxy.cfg inlined above (11 lines, fully synthetic).

Output of haproxy -vv

HAProxy version 3.4.3-80ea565fd 2026/07/29 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2031.
Running on: Linux 6.8.0-138-generic x86_64
(official docker.io/library/haproxy:3.4.3 image)