Skip to content

Conversation

@u-sheikh
Copy link

@u-sheikh u-sheikh commented Dec 28, 2025

This fix addresses a fork-safety issue that caused the Feast feature server to crash when launched with multiple Gunicorn workers and a SQL Registry backend.

Root Cause:

  • SQLAlchemy's connection pools are not fork-safe
  • Background refresh threads inherited from parent become invalid
  • Forked workers share corrupted connection pool state

Solution:

  • Add on_worker_init() hook to BaseRegistry for post-fork cleanup
  • Add reinitialize_engines() to SqlRegistry for engine disposal/recreation
  • Implement Gunicorn post_fork hook in FeastServeApplication
  • Each worker now properly reinitializes DB connections after fork

Changes:

  • sdk/python/feast/infra/registry/base_registry.py: Add on_worker_init()
  • sdk/python/feast/infra/registry/caching_registry.py: Override to restart refresh thread
  • sdk/python/feast/infra/registry/sql.py: Add reinitialize_engines() and on_worker_init()
  • sdk/python/feast/feature_server.py: Add post_fork hook to Gunicorn app

Fixes: #5784

This fix addresses a fork-safety issue that caused the Feast feature
server to crash when launched with multiple Gunicorn workers and a
SQL Registry backend.

Root Cause:
- SQLAlchemy's connection pools are not fork-safe
- Background refresh threads inherited from parent become invalid
- Forked workers share corrupted connection pool state

Solution:
- Add `on_worker_init()` hook to BaseRegistry for post-fork cleanup
- Add `reinitialize_engines()` to SqlRegistry for engine disposal/recreation
- Implement Gunicorn `post_fork` hook in FeastServeApplication
- Each worker now properly reinitializes DB connections after fork

Changes:
- sdk/python/feast/infra/registry/base_registry.py: Add on_worker_init()
- sdk/python/feast/infra/registry/caching_registry.py: Override to restart refresh thread
- sdk/python/feast/infra/registry/sql.py: Add reinitialize_engines() and on_worker_init()
- sdk/python/feast/feature_server.py: Add post_fork hook to Gunicorn app

Fixes: feast-dev#5784
@u-sheikh u-sheikh requested a review from a team as a code owner December 28, 2025 14:34
…dev#5597)

This adds support for configuring DuckDB's HTTPFS extension S3 settings,
enabling compatibility with S3-compatible storage providers like MinIO,
LocalStack, and other providers that require path-style URLs.

New configuration options in DuckDBOfflineStoreConfig:
- s3_url_style: 'path' or 'vhost' - URL style for S3 requests
- s3_endpoint: Custom S3 endpoint URL
- s3_access_key_id: AWS access key ID
- s3_secret_access_key: AWS secret access key
- s3_region: AWS region
- s3_use_ssl: Whether to use SSL for S3 connections

Example usage in feature_store.yaml:
```yaml
offline_store:
    type: duckdb
    s3_url_style: path
    s3_endpoint: localhost:9000
    s3_access_key_id: minioadmin
    s3_secret_access_key: minioadmin
    s3_region: us-east-1
    s3_use_ssl: false
```

Implementation details:
- S3 configuration is applied via DuckDB's HTTPFS extension
- Settings are cached to avoid reconfiguring on each operation
- Graceful error handling if configuration fails
- Type validation ensures only valid values are accepted

Fixes: feast-dev#5597
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The feast server crashes when starting multiple workers with SQL registry

2 participants