feat: Add mTLS support to remote registry gRPC client by jacob-bush-shopify · Pull Request #6474 · feast-dev/feast

jacob-bush-shopify · 2026-06-04T20:45:13Z

What this PR does / why we need it:

Feast currently supports TLS, but does not support mTLS.

For an example of why this is needed: our feature registry is hosted in GCP and sits behind an internal Application Load Balancer that requires mutual TLS for all connections. Additionally, since the feature registry is not exposed to the internet, we need to use an IAP tunnel to connect to the registry during development. The three new configuration options added in this PR solve both of these issues: client_cert, client_key, and authority.

Which issue(s) this PR fixes:

Checks

I've made sure the tests are passing.
My commits are signed off (git commit -s)
My PR title follows conventional commits format

Testing Strategy

Unit tests
Integration tests
Manual tests
Testing is not required for this change

Manual 🎩

I've redacted some of this to remove specific names, and to keep logs shorter.

# Creating ca.crt, tls.crt, tls.key files
❯ certloader [ ... ]

# Creating the tunnel
❯ gcloud compute start-iap-tunnel <REMOTE FEATURE STORE URL> 443 \
  --local-host-port=localhost:8443 \
  [ ... ]

# feature-store.yml
registry:
    registry_type: remote
    path: localhost:8443
    authority: <REMOTE FEATURE STORE URL>
    cert: ./ca.crt
    client_cert: ./tls.crt
    client_key: ./tls.key
    cache_ttl_seconds: 60

On feast 0.61.0 we get an auth failure as expected (the client certs aren't passed through)

❯ feast projects list
[ ... ]
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8443: Custom verification check failed with error: UNAUTHENTICATED: Hostname Verification Check failed."
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8443: Custom verification check failed with error: UNAUTHENTICATED: Hostname Verification Check failed.", grpc_status:14}"
>

On this branch, we we can connect via the iap tunnel 🎉

❯ uvx '~/tmp/feast[gcp]' projects list
NAME           DESCRIPTION                  TAGS    OWNER
<PROJECT NAME>  <PROJECT DESCRIPTION>  {}

Note that setting the authority is necessary in this case because the IAP tunnel causes a host name mismatch otherwise. If we remove it we can see the behaviour of that failure:

❯ uvx '/Users/jacobbush/tmp/feast[gcp]' projects list
[...]
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8443: Custom verification check failed with error: UNAUTHENTICATED: Hostname Verification Check failed."
        debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8443: Custom verification check failed with error: UNAUTHENTICATED: Hostname Verification Check failed.", grpc_status:14}"

Misc

Copilot

Pull request overview

This PR adds mutual TLS (mTLS) client authentication support to the Python Remote Registry gRPC client, including an authority override to support proxy/tunnel workflows (e.g., IAP tunnels) where the connection address differs from the service hostname.

Changes:

Extend RemoteRegistryConfig with client_cert, client_key, and authority, and load these into grpc.ssl_channel_credentials.
Add unit tests that spin up a local mTLS-enforced gRPC server to validate success and failure cases, including the “SAN mismatch without authority” tunnel scenario.
Update documentation to describe mTLS configuration and the authority setting for tunnel/proxy hostname verification.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`sdk/python/feast/infra/registry/remote.py`	Adds mTLS config fields and uses client cert/key when creating the secure gRPC channel.
`sdk/python/tests/utils/ssl_certifcates_util.py`	Adds helper to generate CA/server/client cert chain for mTLS-focused tests.
`sdk/python/tests/unit/infra/registry/test_remote_registry_mtls.py`	New unit tests covering direct mTLS, tunnel-style authority override, and rejection/validation cases.
`docs/reference/registries/remote.md`	Documents new `client_cert`, `client_key`, and `authority` remote registry options with an example.
`docs/how-to-guides/starting-feast-servers-tls-mode.md`	Adds a client-facing mTLS section plus guidance for tunnel/proxy (`authority`) usage.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Add client certificate authentication (mutual TLS) to RemoteRegistry so it can connect to gRPC servers that require client certificates. New config fields on RemoteRegistryConfig: - client_cert: path to the client certificate (PEM) - client_key: path to the client private key (PEM) - authority: override the gRPC :authority header for TLS hostname verification (required when connecting through a tunnel or proxy) The existing cert field continues to serve as the CA certificate for server verification. Signed-off-by: Jacob Bush <jacob.bush@shopify.com> Co-authored-by: AI (Pi/Claude Opus 4.6 [250k]) <noreply@pi.dev>

# [0.64.0](v0.63.0...v0.64.0) (2026-06-13) ### Bug Fixes * Add async_supported property to RedisOnlineStore ([9b088fe](9b088fe)) * Add missing feast init templates to operator CRD and enhance persistence documentation ([1941d4d](1941d4d)) * Allow to publish from reference branch ([5458ec8](5458ec8)) * API calls list ([4203eb7](4203eb7)) * **bigquery:** Enable list inference for parquet loads in offline_write_batch ([9243497](9243497)), closes [#5845](#5845) * Bump grpcio dependencies ([07b4782](07b4782)) * **compute-engine/local:** Honor field_mapping on join keys in dedup + join nodes ([#6395](#6395)) ([bd01824](bd01824)) * **dynamodb:** Avoid tag race condition by using diff-based tag updates ([#6479](#6479)) ([bad2b7d](bad2b7d)), closes [#6418](#6418) * **dynamodb:** Fix mypy type for _build_projection_expression return ([217b4da](217b4da)) * Fix intermittent async test failures for DynamoDB and Redis ([63c5eb1](63c5eb1)) * Fix mongodb blog title ([57d28d4](57d28d4)) * Fix shared SQL registry crash - avoid unnecessary UDF deserialization in proto cache building ([ac588d7](ac588d7)) * Fix SparkRetrievalJob.persist() failing for SparkSource ([209d7cd](209d7cd)) * Fixed formatting and image for mongo blog ([#6377](#6377)) ([f8389fb](f8389fb)) * Fixes for ray source ([7f592a4](7f592a4)) * **go:** skip registry refresh when cache_ttl_seconds <= 0 ([97ed40c](97ed40c)) * Handle array of strings columns in Athena materialization ([#6324](#6324)) ([4ed0278](4ed0278)) * make milvus VARCHAR max_length configurable, remove hardcoded 512 limit ([3b98c22](3b98c22)) * **operator:** Set appProtocol: grpc on registry gRPC Service ([#6367](#6367)) ([c9ae2b4](c9ae2b4)) * PyJWT 2.10+ added validation that rejects empty HMAC keys ([e756ffe](e756ffe)) * RemoteOnlineStore sends all features in a single HTTP request ([8f187dd](8f187dd)) * Remove registry proto dump to enforce RBAC and add permission checks to Commit/Refresh RPCs ([328431f](328431f)) * Remove selector migration job - no longer needed ([51c325e](51c325e)) * replace broken .claude skill symlink with correct relative path ([4541690](4541690)) * Replace selector label strip patch with migration Job for upgrade-safe selector uniqueness ([00dea50](00dea50)) * Scope feature view name conflict check to current project in file-based registry ([#6369](#6369)) ([a4fde83](a4fde83)), closes [#6209](#6209) * **snowflake:** Stop double-quoting connection identifiers ([#6462](#6462)) ([e914d59](e914d59)) * **spark:** S3/GCS PyArrow filesystem resolution for staging paths ([#6442](#6442)) ([ae50414](ae50414)) * **trino:** Clean up temporary entity tables after retrieval ([#6381](#6381)) ([d86b13d](d86b13d)), closes [#6306](#6306) * Update go-feature-server base image to Go 1.25 and fix operator Dockerfile COPY permissions ([86ef0bc](86ef0bc)) ### Features * [Backend] Data Quality Monitoring with native compute, multi-backend support, REST API, CLI ([#6202](#6202)) ([5458c37](5458c37)) * Add apache flink compute engine ([#6476](#6476)) ([9636d6a](9636d6a)) * Add demo noteboooks for users ([e362173](e362173)) * Add enabled/disabled toggle for feature views ([#6401](#6401)) ([5f1fa0d](5f1fa0d)), closes [#6395](#6395) * Add Label View to init template ([ec272d5](ec272d5)) * Add mTLS support to remote registry gRPC client ([#6474](#6474)) ([c9602d8](c9602d8)) * Add Prometheus gauges for FeatureStore installation telemetry ([#6354](#6354)) ([1b681b7](1b681b7)) * Adds registry REST API endpoints for managing entities, data sources, and feature views ([#6413](#6413)) ([f77bd1d](f77bd1d)) * Allow CRUD on entities, data sources, and feature views from UI ([#6412](#6412)) ([2321c07](2321c07)) * Allow default openlineage configuration ([#6467](#6467)) ([276b6df](276b6df)) * **bigquery:** Support DATE-type event timestamp columns ([#6362](#6362)) ([753dee5](753dee5)), closes [#2530](#2530) * **cli:** Add `feast projects delete` command (closes [#5095](#5095)) ([#6318](#6318)) ([1a4b96c](1a4b96c)) * Data Quality Monitoring added in feast UI ([#6422](#6422)) ([fa271be](fa271be)) * **dynamodb:** Use ProjectionExpression when requested_features is set ([0adc906](0adc906)), closes [#6058](#6058) * Enhance DataSource and FeatureView modals with error handling and submission states ([96d7169](96d7169)) * Expose registry endpoints on feature server for MCP access ([f77981c](f77981c)) * Feast First-Class LabelView Implementation ([#6292](#6292)) ([c0e7e5d](c0e7e5d)) * Feast-MLflow Integration ([#6235](#6235)) ([7279c75](7279c75)) * Operational metrics for offline store and SOX metrics for both ([#6340](#6340)) ([65b1b80](65b1b80)) * Pre-compute feature service ([8011550](8011550)) * REST API-backed UI for RBAC compatibility and per-page lazy loading ([#6414](#6414)) ([6ae80af](6ae80af)) * Support non-string map key types ([#6382](#6382)) ([#6383](#6383)) ([728aa2e](728aa2e)) * Update FeatureStore CRD with DRA Fields ([01241e4](01241e4)) ### Performance Improvements * Cache feature view resolution in get_online_features to reduce per-request overhead ([55c2f18](55c2f18)) * Optimize feature serving latency with batched async Redis, cached checks fix ([103809a](103809a)) * Replace MessageToDict with optimized custom dict builder ([#6015](#6015)) ([9902064](9902064))

jacob-bush-shopify requested a review from a team as a code owner June 4, 2026 20:45

jacob-bush-shopify commented Jun 4, 2026

View reviewed changes

Comment thread sdk/python/feast/infra/registry/remote.py

jacob-bush-shopify force-pushed the feat/remote-registry-mtls branch from 56dfa26 to 3d85ddb Compare June 4, 2026 20:57

shuchu requested a review from Copilot June 5, 2026 03:51

Copilot started reviewing on behalf of shuchu June 5, 2026 03:51 View session

shuchu added the ok-to-test label Jun 5, 2026

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Comment thread sdk/python/feast/infra/registry/remote.py

Comment thread sdk/python/feast/infra/registry/remote.py Outdated

Comment thread sdk/python/tests/unit/infra/registry/test_remote_registry_mtls.py Outdated

Comment thread sdk/python/tests/utils/ssl_certifcates_util.py

jacob-bush-shopify force-pushed the feat/remote-registry-mtls branch from 3d85ddb to 88c58fd Compare June 5, 2026 13:31

jacob-bush-shopify force-pushed the feat/remote-registry-mtls branch from 88c58fd to e2e0b91 Compare June 5, 2026 13:46

franciscojavierarceo approved these changes Jun 9, 2026

View reviewed changes

Merge branch 'master' into feat/remote-registry-mtls

b03e870

ntkathole approved these changes Jun 9, 2026

View reviewed changes

ntkathole merged commit c9602d8 into feast-dev:master Jun 9, 2026
30 checks passed

jacob-bush-shopify deleted the feat/remote-registry-mtls branch June 10, 2026 16:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add mTLS support to remote registry gRPC client#6474

feat: Add mTLS support to remote registry gRPC client#6474
ntkathole merged 2 commits into
feast-dev:masterfrom
jacob-bush-shopify:feat/remote-registry-mtls

jacob-bush-shopify commented Jun 4, 2026 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

jacob-bush-shopify commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Checks

Testing Strategy

Manual 🎩

Misc

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jacob-bush-shopify commented Jun 4, 2026 •

edited

Loading