Skip to content

use_backend routing does not get applied (acl based) / default_backend hit instead / also shows wrong and odd last_rule_line #3497

Description

@amsnek

Detailed Description of the Problem

i am using "use_backend" to route to backends, see configuration below. sometimes, possibly since the update from 3.2.x -> 3.4.3 / 3.4.4 i am seeing an odd behaviour: the use_backend rule does not always apply and without changes or reload, requests sometimes hit the default_backend instead.
The "last_line" logged is also very odd in such a case -> it points to the middle of a totally unrelated backend (to a timeout server line in that backend). its espeically odd since this has always the same value: 1893 (even with config changes in haproxy.cfg)

Interestingly enough, this can happen in different browsers differntly at the same time (chrome gets to the correct backend, while firefox at the same time does not an vice versa). waiting a bit and pressing reload will result in a working routing or can result from a working to default_backend

There are multiple use_backend rules -> this seems to only happen for this (new) backend (BE-TESTDOMAINS)

This seems to happen only on a single frontend, on a single LB from many. It makes very little sense. I did check logs to look for other occurences (this is not entirely trivial to spot) but it seems it does not occur anywhere else.

I am not sure howto debug this or properly reproduce.

Expected Behavior

always routed to the correct backend based on the host header given.

Steps to Reproduce the Behavior

  1. call the domain into question in (different) browser

Do you have any idea what may have caused this?

No response

Do you have an idea how to solve the issue?

No response

What is your configuration?

defaults http-fe-defaults
         mode   http
         log    global
         option forwardfor except 127.0.0.1/8
         option http-keep-alive
         option abortonclose
         option splice-auto
         option tcp-smart-accept
         option tcp-smart-connect

         timeout http-request     10s
         timeout http-keep-alive  60s
         timeout client           60s
         timeout client-fin       10s
         timeout tarpit 3s


defaults http-be-defaults
         mode    http
         log     global
         http-reuse safe
         balance leastconn
         option  http-keep-alive
         option  redispatch
         option  abortonclose
         option  tcp-smart-connect
         default-server no-sni-auto
         default-server no-check-sni-auto

         timeout http-request     10s
         timeout http-keep-alive  60s
         timeout connect           5s
         timeout check             5s
         timeout queue            15s
         timeout server           60s


frontend TEST from http-fe-defaults
    guid d27bc276-477b-4ec0-9686-0baa809c0cba
    bind 172.27.10.169:80
    bind 172.27.10.169:443 ssl crt-list /data/crt-list_TESTDOMAINS ktls on
    bind quic4@172.27.10.169:443 ssl crt-list /data/crt-list_TESTDOMAINS alpn h3 ktls on 

    acl testdomains -i -f /data/testdomains.txt
    acl testdomains2 -i -f /data/testdomains2.txt
    acl testdomains3 -i -f /data/testdomains3.txt
    acl testdomains4 -i -f /data/testdomains4.txt
    acl allowed_ips      -f /data/allowed_ips.txt
    acl haproxy_host_ip src 172.27.10.10

    # for tests with curl from haproxy itself
    http-request allow if haproxy_host_ip
    http-request allow if allowed_ips 
    http-request deny 

    http-after-response add-header alt-svc 'h3=":443"; ma=2592000' if { ssl_fc }

    redirect scheme https if !{ ssl_fc }

    use_backend BE-TESTDOMAINS4  if testdomains4 
    use_backend BE-TESTDOMAINS3  if testdomains3 
    use_backend BE-TESTDOMAINS2  if testdomains2 
    use_backend BE-TESTDOMAINS  if testdomains 
    default_backend BE-404


backend BE-TESTDOMAINS from http-be-defaults
        guid bbc7b830-74dc-482b-bff0-5e23615805ac
        cookie T insert indirect nocache
        server testserver 10.10.10.100:443 weight 1 cookie C ssl verify required ca-file bundle-ca.crt alpn h2,http/1.1 ktls on


backend BE-404 from http-be-defaults
        guid 443e1490-77a8-4bfa-afb6-68efb05831e3
        http-request return status 404 content-type "text/html" hdr "cache-control" "no-cache" hdr "connection" "close" file /data/404.txt



/data/testdomain.txt contains a single domain like so:
test-domain-ts.domain.tld

Output of haproxy -vv

HAProxy version 3.4.4-7f03ae6 2026/08/27 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2031.
Known bugs: http://www.haproxy.org/bugs/bugs-3.4.4.html
Running on: Linux 4.18.0-553.154.1.el8_10.x86_64 #1 SMP Sun Aug 9 20:57:29 EDT 2026 x86_64
Build options : 
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -Werror=format-security
  OPTIONS = USE_OPENSSL_AWSLC=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_NS=1 USE_QUIC=1 USE_PROMEX=1 USE_STATIC_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = 

Feature list : -51DEGREES +ACCEPT4 +ACME +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO +HAVE_TCP_MD5SIG -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL +OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE +STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY +TRACE -WURFL -ZLIB
Detected feature list : +HAVE_WORKING_TCP_MD5SIG

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=2).
Built with SSL library version : OpenSSL 1.1.1 (compatible; AWS-LC 5.8.0)
Running on SSL library version : AWS-LC 5.8.0
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library FIPS mode : no
SSL library default verify directory : /etc/ssl/certs
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
QUIC: connection sock-per-conn mode support : yes
QUIC: GSO emission support : no
Built with Lua version : Lua 5.5.1
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.48 2026-08-31
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 8.5.0 20210514 (Red Hat 8.5.0-28)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       qmux : mode=HTTP  side=FE|BE  mux=QMUX  flags=HTX|NO_UPG
       quic : mode=HTTP  side=FE|BE  mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
       spop : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] comp-req
	[COMP] comp-res
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace

Last Outputs and Backtraces


Additional Information

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    status: needs-triageThis issue needs to be triaged.type: bugThis issue describes a bug.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions