Skip to content

net: Happy Eyeballs cancels every candidate when TCP RTT exceeds the 250ms attempt timeout — fetch fails with an empty-message ETIMEDOUT against healthy hosts #67375

Open

Description

@KenAng-tech

Summary

With Happy Eyeballs on (the default since v20), fetch() / https.get() fails outright against any destination whose TCP connect RTT exceeds the attempt timeout, because the same value is used both as the stagger interval between candidates and as the deadline that cancels each candidate. If RTT > attemptTimeout, every candidate is cancelled before its own SYN/ACK can arrive — including the last one — so the algorithm cannot succeed at all, while connecting without Happy Eyeballs succeeds on the first address.

Measured on v24.22.1 / darwin-arm64 against public hosts. curl and /usr/bin/python3 (SecureTransport) on the same machine at the same second succeed every time; Node is the only client that fails.

Reproduction (no accounts, no setup)

// repro.mjs
import net from 'node:net';
import dns from 'node:dns/promises';

const host = process.argv[2] ?? 'docker.io';
const { address } = await dns.lookup(host, { verbatim: true });

const t0 = Date.now();
const s = net.connect({ host: address, port: 443, family: 4 });   // Happy Eyeballs bypassed: explicit IP
s.on('connect', () => {
  console.log(`tcp connect to one IP directly : OK in ${Date.now() - t0} ms`);
  s.destroy();
  const t1 = Date.now();
  fetch(`https://${host}/`)
    .then((r) => console.log(`fetch(https://${host}/)              : ${r.status} in ${Date.now() - t1} ms`))
    .catch((e) => console.log(`fetch(https://${host}/)              : FAILED ${e.message}`
      + ` | cause=${e.cause?.code} message="${e.cause?.message}"`
      + ` | cause.errors=${e.cause?.errors?.length}`
      + ` (${e.cause?.errors?.[0]?.code} ${e.cause?.errors?.[0]?.address}) in ${Date.now() - t1} ms`));
});
$ node repro.mjs docker.io
tcp connect to one IP directly : OK in 384 ms
fetch(https://docker.io/)      : FAILED fetch failed | cause=ETIMEDOUT message="" | cause.errors=6 | ETIMEDOUT 2600:9000:...:443 in 2798 ms

$ node --no-network-family-autoselection repro.mjs docker.io
tcp connect to one IP directly : OK in 386 ms
fetch(https://docker.io/)      : 302 in 1255 ms

The failure is not a property of the destination. Run the same command twice a minute apart and the outcome tracks the current RTT across the 250 ms line:

target measured TCP RTT fetch() at default budget
docker.io 384, 386 ms FAIL, FAIL (2/2)
docker.io 255, 263 ms 302, 302
quay.io 369, 388 ms FAIL, FAIL (2/2)
quay.io 237, 265 ms 200, 200
arxiv.org (A-only edge, 4 ms) 4 ms 200 (never affected)

Budget sweep, one variable

Same host, same minute, only --network-family-autoselection-attempt-timeout changes (RTT sampled in the same run):

budget RTT sample result
250 (default) 465 ms FAIL
250 (default) 365 ms FAIL
300 368 ms FAIL
350 457 ms 200 in 2970 ms
350 349 ms FAIL
400 372 ms 200 in 2178 ms
500 383, 362 ms 200, 200
600 389 ms 200

The knee sits exactly at the measured RTT. Note the successful marginal runs take ~2.1–3.0 s — i.e. the winner is the 6th–9th candidate, which is what you get when candidates are launched one attemptTimeout apart and only the last ones are still alive when RTT ≈ budget.

The reported error actively misleads

https.get() rejects with the AggregateError itself (no cause); fetch() wraps it in TypeError: fetch failed:

{ "ctor": "AggregateError", "name": "AggregateError", "message": "", "code": "ETIMEDOUT",
  "hasCause": false, "nErrors": 18 }
tally: {"ETIMEDOUT@v4": 9, "EHOSTUNREACH@v6": 9}
sample: {"message": "connect ETIMEDOUT 221.194.164.123:443", "code": "ETIMEDOUT",
         "address": "221.194.164.123", "port": 443, "syscall": "connect"}
keys: ["errno","code","syscall","address","port"]

Three problems for anyone debugging this in production:

  1. err.message is the empty string at the top level — log lines that print err.message print nothing.
  2. Every v4 sub-error says connect ETIMEDOUT <ip>:443, yet a plain net.connect to that exact IP completes in ~370 ms. The addresses were never given a chance to finish; they were cancelled at 250 ms. Reporting them as ETIMEDOUT sends people hunting for a firewall/blackhole that isn't there (I did — see below).
  3. Nothing in the error mentions the attempt timeout, the number of candidates tried, or that --no-network-family-autoselection would have worked. ECONNREFUSED/EHOSTUNREACH per candidate is fine, but "we cancelled all of them because the budget expired" is a distinct, actionable cause.

What I ruled out first (so you don't have to)

On the same machine, same second: curl 8.7.1 200 · system python3 urllib 200 · openssl s_client handshake OK · Node against pypi/npm/github/google/eastmoney all 200 · 9 v4 addresses in parallel 9/9 OK · tls.connect to each IP individually + hand-written HTTP request 10/10 200 OK. net.setDefaultAutoSelectFamily(false) fixes it. So it is not the network, the certificate, TLS version, ALPN, UA, rate limiting, or the post-quantum ClientHello.

Expected behaviour

Fall back to a working address instead of failing the whole attempt. Options that would each fix my case, in rough order of preference:

  1. Once all candidates have been cancelled without a completed handshake, retry at least one round without the per-candidate cancellation (curl/SecureTransport effectively do this and never fail here). Or: treat attemptTimeout only as the stagger delay, and let an in-flight candidate live until the socket-level timeout.
  2. Raise the default. RFC 8305's 250 ms default assumes intra-domain reachability; the public internet from many networks is routinely 250–500 ms, which makes the default configuration strictly worse than disabling Happy Eyeballs.
  3. Make the failure self-describing: non-empty message, and a cause that says "all N candidates exceeded the X ms attempt timeout (last RTT-class: v4/v6)".

Environment / workarounds in use

Node v24.22.1, darwin-arm64, macOS 15, no IPv6 egress on the host. We unblocked production with NODE_OPTIONS=--network-family-autoselection-attempt-timeout=1000 (verified: 8/8 previously-failing call sites now 200, and RTT to unrelated hosts unchanged at 32–192 ms, so the raised budget costs nothing when addresses are healthy). --no-network-family-autoselection also works but gives up fast-fail for genuinely dual-stacked hosts.

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions