feat(auth): add _MtlsCertManager for mTLS client certificate rotation - #18925

The except block currently catches a specific list of exceptions (ClientCertError, MutualTLSChannelError, OSError, ValueError). However, when a custom client_cert_source callback is provided, it may raise other unexpected or custom exceptions (e.g., network errors, custom HSM/KMS exceptions, or even TypeError if the callback returns an unexpected format).

If such an exception is raised during a certificate reload (when self._generation_id > 0), it will propagate and crash the application/request instead of being gracefully handled as a transient reload error. To ensure maximum robustness and successfully retain the last known good certificate, we should catch all standard exceptions (Exception) during reload.

            except Exception as caught_exc:
                if self._generation_id > 0:
                    _LOGGER.debug(
                        "Failed to reload mTLS client certificate; retaining "
                        "previous certificate (generation %d): %s",
                        self._generation_id,
                        caught_exc,
                    )
                    return False
                if isinstance(caught_exc, exceptions.ClientCertError):
                    raise
                raise exceptions.ClientCertError(caught_exc) from caught_exc

Should we re-export _MtlsCertManager from google.auth.transport.mtls here or in a follow-up PR so google-api-core doesn't have to import private _mtls_helper directly?

Does get_credential_snapshot() need to block on self._lock? While refresh() holds the lock across disk reads, mismatch retries, callbacks, or cert_provider_command, snapshot callers on other threads or an async event loop have to wait behind that I/O instead of reading an atomic snapshot tuple.

get_client_ssl_credentials falls back to ~/.secureConnect/context_aware_metadata.json whenever workload discovery returns (None, None). If an explicit certificate_config_path or workload config is briefly missing during rotation on a host with ~/.secureConnect, refresh() swaps to the SecureConnect cert instead of keeping the cached cert, and a missing explicit certificate_config_path on initial load returns False instead of raising ClientCertError. Should default discovery here also check GOOGLE_API_USE_CLIENT_CERTIFICATE?

_load_cert_and_fingerprint parses cert_bytes here, but never parses key_bytes or checks that it matches cert_obj. #19043 only runs inside _read_cert_and_key_files, so client_cert_source callbacks, GKE credential bundles, and SecureConnect bypass it. Even on workload files with #19043, an unparseable or encrypted key makes _has_cert_key_mismatch return False, which overwrites the working snapshot and latches _cert_fingerprint to the new cert so _key stays broken after a valid key file arrives.

Should this catch Exception instead of listing five exception types? cryptography.x509.InvalidVersion doesn't inherit from ValueError, malformed context_aware_metadata.json can raise TypeError or IndexError, and custom client_cert_source callbacks can raise RuntimeError or custom errors, all of which crash live reloads at generation_id > 0 and leave _completed_reload_id behind _reload_id. Watch out for the gemini-code-assist suggestion below too, since it's from an earlier commit and drops self._completed_reload_id = self._reload_id.

Should suppressed reload failures log at warning instead of debug? Otherwise a broken cert path or permission error during rotation stays silent at default log levels until the cached cert expires hours later.

Should changed check cert_bytes != self._cert or key_bytes != self._key instead of only the leaf fingerprint? Right now if a PEM bundle renews an intermediate CA while keeping the same leaf cert, or updates key_bytes for the same leaf, refresh() returns False and keeps serving the stale bytes.

Rotating only cert_file to OTHER_CERT_BYTES while leaving key_file untouched doesn't test that _key updates on rotation. Also other_cert.pem doesn't match privatekey.pem, so this test will fail once #19043 lands and _read_cert_and_key_files rejects mismatched pairs.

side_effect=[reload_result] * 2 only holds two items, and a second thread already consumes the second item on normal runs because the first thread bumps _reload_id before time.sleep(0.1). If any of the remaining four threads wakes up 100ms late after barrier.wait() on a busy CI runner, reload_source() raises StopIteration inside the worker thread and leaves results[idx] as None.