[Bug]: connect_to_this_ip46() stops iterating resolved addresses after the first connect timeout #24392

Bug description

In connect_to_this_ip46() (src/libnetdata/socket/connect-to.c), the address-iteration loop advances to the next resolved address only when the connect error is ND_SOCK_ERR_CONNECTION_REFUSED. After a timeout, the function gives up on the remaining addresses of the same host — even though other A/AAAA records may be reachable.

The loop condition is fd == -1 (connect-to.c:125):

for (ai = ai_head; ai != NULL && fd == -1; ai = ai->ai_next) {

Whether it iterates is decided by the numeric coincidence that ND_SOCK_ERR_CONNECTION_REFUSED == 1. There is no explicit switch/if on the error code and no comment documenting the intended policy anywhere in connect-to.c or nd-sock.h.

Error assignments in the same function (ND_SOCK_ERROR enum: src/libnetdata/socket/nd-sock.h:8-25):

Three related observations in the same function:

• The immediate-error branch does not inspect errno. errno_clear() runs at :175 and errno is examined only at :177 (EALREADY/EINPROGRESS). Every other immediate connect() failure — ENETUNREACH, EHOSTUNREACH, EACCES, a genuine ECONNREFUSED — falls into the else at :223-230 and is mapped to CONNECTION_REFUSED, so it does continue. The inverse inconsistency: "network unreachable" iterates, "timeout" does not. The log line at :224-226 carries neither errno nor strerror().
• fallback_ipv4 cannot help within an attempt. It is set on an IPv6 timeout at :208-209 and consumed by the skip at :128-129, but the IPv6 timeout has already exited the loop.
• getaddrinfo() is called once per attempt (:104) with no Netdata-side caching, so successive attempts may use different records for the same host. Expected, but worth knowing when reading logs.

Observed in agent logs during a partial network outage (agent → mqtt.netdata.cloud, 3 A + 3 AAAA records): exactly one Timed out while connecting to '<ip>' line per reconnect attempt, with a different address each attempt — the remaining IPv4 records are never tried within an attempt.

Timed out while connecting to '44.196.50.41', port '443'.
Timed out while connecting to '54.198.178.11', port '443'.   (~20 s later — new attempt, different record)
Timed out while connecting to '44.207.131.212', port '443'.

Expected behavior

On a connect timeout, the loop should try the remaining resolved addresses of the same host, the same way it does today for connection-refused — ideally under an aggregate per-attempt budget so worst-case latency stays bounded.

If stopping after the first timeout is intentional (to bound connect latency), then it should be explicit and commented rather than an emergent property of CONNECTION_REFUSED == 1, and it should be consistent with the immediate-error path, which currently iterates for any immediate failure regardless of errno.

Steps to reproduce

1. On a Linux host running the Netdata agent, resolve mqtt.netdata.cloud and pick the first A record.
2. Blackhole traffic to that single address:

   sudo iptables -I OUTPUT -d <first-resolved-A-record> -j DROP
   

3. Force an ACLK reconnect (restart the agent, or wait for the next reconnect attempt).
4. Read the agent log.

Observed: one Timed out while connecting to '<dropped ip>' line per attempt, and no attempt against the remaining A records within that attempt — even though they are reachable.

Counter-check: repeat with a rule that yields connection-refused instead of a drop. The loop then does iterate the remaining addresses, which isolates the behaviour to the timeout path.

Installation method

kickstart.sh

System info

Both hosts where the behaviour was observed in production agent logs:

# Host A
Linux <redacted> 5.14.0-570.41.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux 9.6 (Plow)
x86_64, 4 cores, Intel Xeon Gold 6530, ~19.3 GiB RAM
Standalone agent (no parent/child relationship)

# Host B
Linux <redacted> 5.14.0-570.21.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux 9.6 (Plow)
x86_64, 8 cores, Intel Xeon Gold 6530, ~31.1 GiB RAM
Netdata Parent with 1 child, itself a child of another Parent (hops=1)

Netdata build info

agent v2.12.1, release channel stable, on both hosts

Additional info

Scope. connect_to_this_ip46() is the legacy connect path. Confirmed callers:

• ACLK MQTT: src/aclk/mqtt_websockets/mqtt_wss_client.c:451-452 — passes a 10 s timeout
• ACLK HTTPS: src/aclk/https_client.c:940-941 — same 10 s pattern, also passes fallback_ipv4
• ND_SOCK reaches it indirectly: nd_sock_connect_to_this() (src/libnetdata/socket/nd-sock.c:66) calls connect_to_this() at :115, which forwards to connect_to_this_ip46(..., NULL) at connect-to.c:412 — i.e. without fallback_ipv4

Honest caveat on impact. In the incident these logs come from, all resolved IPv4 addresses were unreachable, so iterating all records within an attempt would not have changed the outcome. The issue is about partial outages — one record blackholed or slow while the others are fine — where today's behaviour abandons the host after a single record times out.

Secondary observation (separate concern, same function): one attempt takes ~20 s although the caller passes a 10 s timeout.

Attempts in the logs are ~20 s apart while mqtt_wss_client.c:451 passes .tv_sec = 10. Working hypothesis — the Netdata-side steps are code-verified, the kernel-behaviour step is not:

1. The socket is created blocking: socket() at :165 with DEFAULT_SOCKET_FLAGS = SOCK_CLOEXEC only (src/libnetdata/socket/socket.h:49-53), no SOCK_NONBLOCK.
2. SO_SNDTIMEO = 10 s is set at :168; a blocking connect() to a blackholed destination appears to consume ~10 s in-kernel before returning.
3. The code then polls for a fresh, full 10 s (timeval_to_poll_timeout_ms() at :182, wait_on_socket_or_cancel_with_timeout() at :184) ⇒ ~10 + 10 = 20 s per attempt, matching the logs.

Falsifiable check for maintainers: enable debug logging and compare the timestamp of the Waiting for connection to ip ... to be established line against the attempt start — expected ≈10 s in if the hypothesis holds. Happy to split this into its own issue if it has different ownership; kept here because it is the same function and the same log lines.

Line numbers are from master as inspected 2026-10-21.