Skip to content

--use-system-ca: process exit waits for the off-thread system CA load, even when no certificate is ever used (macOS: seconds with a slow keychain/smart card) #67510

Open

Description

@giladben-dor-alt

Version

v26.13.1 (also v26.7.0; the code is unchanged on main)

Platform

macOS 27.2.1, Darwin 27.2.0 arm64

Subsystem

crypto, tls

What steps will reproduce the bug?

This report was drafted by an AI assistant (GitHub Copilot) from investigation and measurements on my machine, at my request; the reproduction below was run as written.

The real-world trigger is a slow keychain query (below), which is hard to reproduce on demand, so this reproduction makes SecItemCopyMatching take 10 s with an interposed library. No smart card is needed.

slow_keychain.c:

// Simulates a slow keychain query (e.g. a smart-card token that is slow to answer).
#include <Security/Security.h>
#include <unistd.h>

static OSStatus slow_SecItemCopyMatching(CFDictionaryRef query, CFTypeRef *result) {
  sleep(10);
  return SecItemCopyMatching(query, result);
}

__attribute__((used, section("__DATA,__interpose")))
static struct { const void *replacement, *original; } interpose[] = {
  {(const void *)slow_SecItemCopyMatching, (const void *)SecItemCopyMatching},
};
clang -dynamiclib -framework Security -o slow_keychain.dylib slow_keychain.c
export DYLD_INSERT_LIBRARIES=$PWD/slow_keychain.dylib   # the official binaries allow it

time node --use-system-ca -e "require('tls')"                          # 10.12 s
time node --use-system-ca -e "require('tls'); process.exit(0)"         # 10.15 s
time node --use-system-ca --input-type=module -e "import 'node:http'"  # 10.13 s
time node -e "require('tls')"                                          #  0.10 s

The scripts do nothing with certificates, yet each process lives until the keychain query returns.

How often does it reproduce? Is there a required condition?

Always with the steps above. Required: --use-system-ca (or NODE_USE_SYSTEM_CA=1), and tls loaded.

In real life the condition is a slow keychain query. On my Mac, with a PIV smart card (a USB security key) plugged in, a plain NODE_OPTIONS=--use-system-ca node --input-type=module -e "import 'node:http'" took between 0.1 s and 31 s, depending on the token's state. macOS sample of the lingering process showed:

  • main thread: node::DefaultProcessExitHandlerInternal → node::crypto::CleanupCachedRootCertificates() → uv_thread_join;
  • the joined thread: node::crypto::LoadSystemCACertificates → ReadMacOSKeychainCertificates → SecItemCopyMatching → SecTokenSessionCreate → -[TKExtensionClientToken withError:invoke:] → +[NSThread sleepForTimeInterval:] (CryptoTokenKit retrying the token's extension).

A separate direct tls.getCACertificates('system') run, sampled at about 2, 10 and 20 seconds, stayed in the same CryptoTokenKit sleep frame.

What is the expected behavior? Why is that the expected behavior?

The process exits as soon as the script is done (~0.1 s), as it does without --use-system-ca. The off-thread load is an optimization for a later TLS connection. When the process is exiting and nothing has waited for the result, waiting for it only delays exit. Short-lived CLI tools that load tls at startup and never connect pay the full keychain read on every run.

What do you see instead?

Exit blocks until the system CA thread finishes:

  1. lib/tls.js calls startLoadingCertificatesOffThread() at module load. With --use-system-ca this starts LoadSystemCACertificates on system_cert_loading_thread (src/crypto/crypto_context.cc, StartLoadingCertificatesOffThread).
  2. At exit, CleanupCachedRootCertificates() calls uv_thread_join(&system_cert_loading_thread). process.exit() takes the same path, so an explicit exit does not help.
  3. Under ESM, import 'node:http' alone is enough: building the builtin's ESM facade reads every export, including the lazy WebSocket/CloseEvent/MessageEvent getters, which load undici and therefore tls. With require('http') or process.getBuiltinModule('node:http'), tls is not loaded.

Additional information

Possible directions, not tested fixes:

  • Could exit avoid waiting for a preload nobody has used, while keeping the loader's shared state and teardown safe? (CleanupCachedRootCertificates() joins before it clears the caches, so simply detaching is probably not enough.)
  • ReadMacOSKeychainCertificates queries kSecClassCertificate with kSecMatchLimitAll and no keychain restriction, and the wait happens during token enumeration inside SecItemCopyMatching. A query-level way to exclude token-backed items might avoid it, provided the existing trusted-certificate behaviour is preserved. I have not tested such a change.

Related, but neither covers the exit join: #63313 (macOS trust evaluation during the system CA load can stall on network fetches) and its open fix #65765. That fix makes the load faster in one case; this report is about exit waiting for the load at all.

Workaround for affected tools: load tls/https lazily (process.getBuiltinModule('node:https') where a connection is made), so processes that make no connection never start the load.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions