Skip to content

Commit f45ed4d

Browse files
nikolauspschuetznikolauspschuetz
committed
fix: Use LONGBLOB for SQL registry proto columns on MySQL
The SQL registry stores each Feast object as a serialized protobuf in a binary column. On MySQL/MariaDB, SQLAlchemy's LargeBinary maps to BLOB, which caps at 64 KB. A single FeatureView proto routinely exceeds that, so MySQL silently truncates the write and the registry later fails to load with a protobuf DecodeError (e.g. `feast serve` failing to start). PostgreSQL and SQLite were never affected. Introduce a dialect-aware `ProtoBytes` type that emits LONGBLOB on MySQL and MariaDB while keeping LargeBinary's default mapping on every other dialect, and apply it to all binary proto/metadata columns. The variants are chained (not variadic) so the expression also works on SQLAlchemy 1.4.x, which Feast still supports. `metadata.create_all` only creates missing tables, so existing MySQL registries are not migrated automatically. Add a best-effort startup warning that names any columns still typed BLOB and points operators at the documented ALTER TABLE migration, and document the migration (with metadata-lock / online-schema-change guidance) in the SQL registry reference. Tests assert the compiled DDL emits LONGBLOB on MySQL and MariaDB and BLOB on SQLite, and cover the startup-warning paths. Signed-off-by: Nikolaus Schuetz <nikolauspschuetz@gmail.com>
1 parent 0d02614 commit f45ed4d

8 files changed

Lines changed: 548 additions & 20 deletions

File tree

‎.claude/rules/feast-components.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ For testing patterns and debugging, also read `skills/feast-testing/SKILL.md`.
2424

2525
- **Unit tests**: add or update tests in `sdk/python/tests/unit/infra/<subsystem>/`
2626
- **Integration tests**: run `make test-python-integration-local`; add a universal test case in `sdk/python/tests/integration/` if the change affects retrieval or materialization behavior
27+
- **SQL registry binary columns**: in `infra/registry/sql.py`, a new column that stores a serialized proto or blob metadata must use `ProtoBytes`, not `LargeBinary` directly — `LargeBinary` maps to MySQL `BLOB` (64 KB cap) and silently truncates large protos
2728
- **Protos**: if you add a field to a proto message, recompile with `make protos` and update serialization helpers in `proto_registry_utils.py`
2829
- **Both SDKs**: if the change affects online serving, check whether the Go server (`go/`) also needs updating
2930
- **Skills/Rules**: if the change introduces new patterns, interfaces, or conventions that agents should follow, update the relevant section in `skills/feast-architecture/SKILL.md` (and `skills/feast-testing/SKILL.md` if testing patterns changed)

‎.cursor/rules/feast-components.mdc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ For testing patterns and debugging, also read `skills/feast-testing/SKILL.md`.
2020

2121
- **Unit tests**: add or update tests in `sdk/python/tests/unit/infra/<subsystem>/`
2222
- **Integration tests**: run `make test-python-integration-local`; add a universal test case in `sdk/python/tests/integration/` if the change affects retrieval or materialization behavior
23+
- **SQL registry binary columns**: in `infra/registry/sql.py`, a new column that stores a serialized proto or blob metadata must use `ProtoBytes`, not `LargeBinary` directly — `LargeBinary` maps to MySQL `BLOB` (64 KB cap) and silently truncates large protos
2324
- **Protos**: if you add a field to a proto message, recompile with `make protos` and update serialization helpers in `proto_registry_utils.py`
2425
- **Both SDKs**: if the change affects online serving, check whether the Go server (`go/`) also needs updating
2526
- **Skills/Rules**: if the change introduces new patterns, interfaces, or conventions that agents should follow, update the relevant section in `skills/feast-architecture/SKILL.md` (and `skills/feast-testing/SKILL.md` if testing patterns changed)

‎docs/reference/registries/sql.md‎

Lines changed: 78 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,84 @@ If you are running Feast in Kubernetes, set the `image.repository` and
8383
There are some things to note about how the SQL registry works:
8484
- Once instantiated, the Registry ensures the tables needed to store data exist, and creates them if they do not.
8585
- Upon tearing down the feast project, the registry ensures that the tables are dropped from the database.
86-
- The schema for how data is laid out in tables can be found . It is intentionally simple, storing the serialized protobuf versions of each Feast object keyed by its name.
86+
- The schema for how data is laid out in tables can be found in the table definitions in [`sdk/python/feast/infra/registry/sql.py`](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/registry/sql.py). It is intentionally simple, storing the serialized protobuf versions of each Feast object keyed by its name.
87+
88+
## MySQL: serialized-proto columns use `LONGBLOB`
89+
90+
The registry stores each Feast object as a serialized protobuf in a binary
91+
column. On MySQL these columns are created as `LONGBLOB` (up to 4 GB). Earlier
92+
versions created them as `BLOB`, which caps at 64 KB — a single `FeatureView`
93+
proto routinely exceeds that, so MySQL would silently truncate the write and the
94+
registry would later fail to load with a protobuf `DecodeError` (for example,
95+
`feast serve` failing to start). Other dialects (PostgreSQL, SQLite) were never
96+
affected.
97+
98+
New deployments get the correct schema automatically — the registry creates its
99+
tables as `LONGBLOB` on first use. When an existing MySQL/MariaDB registry still
100+
has `BLOB` columns, the registry logs an error at startup listing the affected
101+
columns (it does not refuse to start — a registry whose protos all fit in 64 KB
102+
is unaffected). **Existing deployments are not migrated automatically**: the
103+
registry only creates tables that do not already exist, and it has no
104+
schema-migration step, so previously created `BLOB` columns remain `BLOB`. To
105+
upgrade an existing MySQL registry, alter each serialized-proto column to
106+
`LONGBLOB`, for example:
107+
108+
> ⚠️ **Run the migration carefully on a live registry.** A `BLOB`→`LONGBLOB`
109+
> change is a column *data-type* change, which MySQL InnoDB performs with
110+
> `ALGORITHM=COPY` — a full table rebuild under a metadata lock that blocks
111+
> readers and writers for the duration (potentially minutes on a large table
112+
> such as `feature_view_version_history`). `ALGORITHM=INPLACE` is **not**
113+
> generally supported for this change and is rejected with
114+
> `ER_ALTER_OPERATION_NOT_SUPPORTED_REASON` on most builds — do not rely on it.
115+
>
116+
> **Before running any `ALTER TABLE`:**
117+
>
118+
> 1. **Stop all `feast apply` and materialization jobs.** This is required, not
119+
> optional — a write of a `>64 KB` proto to a not-yet-widened `BLOB` column
120+
> truncates silently with no error, and concurrent writes also extend the
121+
> `ALTER`'s lock duration.
122+
> 2. Confirm there are no active writers (e.g. `SHOW PROCESSLIST`).
123+
> 3. Verify you have a backup of the registry database.
124+
>
125+
> Then, to minimize the lock window:
126+
>
127+
> - On large tables, or on managed MySQL (AWS RDS, Aurora) without shell access,
128+
> use an online schema-change tool —
129+
> [`pt-online-schema-change`](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html)
130+
> (Percona Toolkit) or [`gh-ost`](https://github.com/github/gh-ost) — which
131+
> rebuild the table without a long-held lock. For small tables a plain
132+
> `ALTER TABLE` in the maintenance window is fine.
133+
> - Apply one table at a time so a failure is easy to isolate and re-run.
134+
> - Resume jobs only after all `ALTER TABLE` statements complete successfully.
135+
> - Rollback is safe (revert `MODIFY ... BLOB`) **only** while no stored proto
136+
> exceeds 64 KB; otherwise a revert re-introduces truncation.
137+
138+
```sql
139+
ALTER TABLE projects MODIFY project_proto LONGBLOB NOT NULL;
140+
ALTER TABLE entities MODIFY entity_proto LONGBLOB NOT NULL;
141+
ALTER TABLE data_sources MODIFY data_source_proto LONGBLOB NOT NULL;
142+
ALTER TABLE feature_views MODIFY materialized_intervals LONGBLOB,
143+
MODIFY feature_view_proto LONGBLOB NOT NULL,
144+
MODIFY user_metadata LONGBLOB;
145+
ALTER TABLE stream_feature_views MODIFY feature_view_proto LONGBLOB NOT NULL,
146+
MODIFY user_metadata LONGBLOB;
147+
ALTER TABLE on_demand_feature_views MODIFY feature_view_proto LONGBLOB NOT NULL,
148+
MODIFY user_metadata LONGBLOB;
149+
ALTER TABLE label_views MODIFY feature_view_proto LONGBLOB NOT NULL,
150+
MODIFY user_metadata LONGBLOB;
151+
ALTER TABLE feature_services MODIFY feature_service_proto LONGBLOB NOT NULL;
152+
ALTER TABLE saved_datasets MODIFY saved_dataset_proto LONGBLOB NOT NULL;
153+
ALTER TABLE validation_references MODIFY validation_reference_proto LONGBLOB NOT NULL;
154+
ALTER TABLE managed_infra MODIFY infra_proto LONGBLOB NOT NULL;
155+
ALTER TABLE permissions MODIFY permission_proto LONGBLOB NOT NULL;
156+
-- LARGE TABLE: one row per versioned apply — likely the slowest ALTER. Use
157+
-- pt-online-schema-change or gh-ost if this registry has significant history.
158+
ALTER TABLE feature_view_version_history MODIFY feature_view_proto LONGBLOB NOT NULL;
159+
```
160+
161+
Any object whose proto already exceeded 64 KB before the upgrade may have been
162+
stored truncated; re-run `feast apply` for those objects after altering the
163+
columns so the full proto is rewritten.
87164

88165
## Example Usage: Concurrent materialization
89166
The SQL Registry should be used when materializing feature views concurrently to ensure correctness of data in the registry. This can be achieved by simply running feast materialize or feature_store.materialize multiple times using a correctly configured feature_store.yaml. This will make each materialization process talk to the registry database concurrently, and ensure the metadata updates are serialized.

0 commit comments

Comments
 (0)