Skip to content

Commit 10341e4

Browse files
zerafachrisclaude
zerafachris
and
claude
authored
fix: Bump decommissioned Snowflake Python UDF runtime from 3.9 to 3.10 (#6606) (#6608)
* fix: Bump decommissioned Snowflake Python UDF runtime from 3.9 to 3.10 (#6606) Snowflake decommissioned the Python 3.9 UDF runtime, so any Snowflake offline store `feast apply` that deploys the Feast materialization UDFs now fails with "Python runtime version 3.9 is decommissioned". The RUNTIME_VERSION = '3.9' value was hardcoded in the executed UDF creation SQL template (snowflake_python_udfs_creation.sql) and mirrored in the reference docstrings of snowflake_udfs.py. Both are bumped to '3.10', matching Feast's own minimum supported Python version (requires-python = ">=3.10.0" in pyproject.toml). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: zerafachris <christopher.zerafa@blocklabs.io> * feat: Make Snowflake Python UDF runtime version user-configurable Addresses maintainer feedback from @ntkathole on #6608 ("is it possible to make it user configurable?") following the RUNTIME_VERSION 3.9 -> 3.10 bump for decommissioned Snowflake Python UDF runtimes (#6606). Adds `python_udf_runtime_version` (default "3.10") to SnowflakeComputeEngineConfig, validated as a Python version string, and plumbs it through SnowflakeComputeEngine.update() via a new RUNTIME_VERSION_HOLDER placeholder in snowflake_python_udfs_creation.sql, following the same substitution pattern already used for STAGE_HOLDER and PROJECT_NAME. This way, the next time Snowflake decommissions a Python runtime, users can override the config instead of waiting on a Feast release with another hardcoded bump. Also updates snowflake_udfs.py with a note clarifying its RUNTIME_VERSION docstrings are static reference documentation of the default (not the templated value actually deployed), documents the new field in docs/reference/compute-engine/snowflake.md, and adds/updates tests covering the config default, custom override, validation, and the rendered SQL template. Signed-off-by: zerafachris <christopher.zerafa@blocklabs.io> * fix: Redeploy Snowflake UDFs on every apply so runtime version changes take effect Addresses maintainer feedback from @ntkathole on #6608 (#6608 (comment)): once materialization UDFs were deployed for a project, SnowflakeComputeEngine.update() short-circuited on a `SHOW USER FUNCTIONS` check, and the SQL template used `CREATE FUNCTION IF NOT EXISTS`. Together, this meant an already-deployed project's UDFs were never redeployed, so changing `python_udf_runtime_version` (e.g. after Snowflake decommissions a runtime, as in #6606) silently left the stale runtime version in place. Removes the early-return guard and switches every UDF creation statement in snowflake_python_udfs_creation.sql from `CREATE FUNCTION IF NOT EXISTS` to `CREATE OR REPLACE FUNCTION`, so `update()` always (re)deploys the UDFs using whatever runtime version is currently configured. Adds unit tests covering that update() no longer short-circuits and that a redeploy always uses the current python_udf_runtime_version. Signed-off-by: zerafachris <christopher.zerafa@blocklabs.io> --------- Signed-off-by: zerafachris <christopher.zerafa@blocklabs.io> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
1 parent f487b37 commit 10341e4

7 files changed

Lines changed: 413 additions & 68 deletions

File tree

‎docs/reference/compute-engine/snowflake.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,5 +24,10 @@ batch_engine:
2424
role: sysadmin
2525
warehouse: demo_wh
2626
database: FEAST
27+
python_udf_runtime_version: "3.10"
2728
```
2829
{% endcode %}
30+
31+
## Configuration
32+
33+
* `python_udf_runtime_version` *(optional, default: `"3.10"`)* -- The Snowflake Python UDF `RUNTIME_VERSION` used when Feast deploys its materialization UDFs. Snowflake periodically decommissions old Python UDF runtimes (for example, the 3.9 runtime was decommissioned, requiring Feast to bump its default to 3.10 -- see [#6606](https://github.com/feast-dev/feast/issues/6606)). If Snowflake decommissions the 3.10 runtime in the future, set this field to a still-supported version (e.g. `"3.11"`) instead of waiting for a new Feast release.

‎sdk/python/feast/infra/compute_engines/snowflake/snowflake_engine.py‎

Lines changed: 51 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import os
2+
import re
23
import shutil
34
from datetime import timezone
45
from typing import Literal, Optional, Sequence, Union
@@ -7,7 +8,7 @@
78
import pandas as pd
89
import pyarrow as pa
910
from colorama import Fore, Style
10-
from pydantic import ConfigDict, Field, StrictStr
11+
from pydantic import ConfigDict, Field, StrictStr, field_validator
1112
from tqdm import tqdm
1213

1314
import feast
@@ -87,8 +88,41 @@ class SnowflakeComputeEngineConfig(FeastConfigBaseModel):
8788

8889
schema_: Optional[str] = Field("PUBLIC", alias="schema")
8990
""" Snowflake schema name """
91+
92+
python_udf_runtime_version: StrictStr = "3.10"
93+
"""
94+
Snowflake Python UDF RUNTIME_VERSION used when Feast deploys its materialization
95+
UDFs (see `snowflake_python_udfs_creation.sql`).
96+
97+
Snowflake periodically decommissions old Python UDF runtimes -- e.g. the 3.9
98+
runtime was decommissioned, which required Feast to bump its default runtime
99+
from 3.9 to 3.10 (see https://github.com/feast-dev/feast/issues/6606). Rather
100+
than hardcoding a version that will eventually go stale again, this field is
101+
user-configurable so you are not blocked on a new Feast release the next time
102+
Snowflake deprecates a runtime.
103+
104+
Defaults to "3.10", matching Feast's own minimum supported Python version
105+
(`requires-python` in `pyproject.toml`). If Snowflake decommissions the 3.10
106+
runtime in the future, override it in your `feature_store.yaml`, e.g.:
107+
108+
batch_engine:
109+
type: snowflake.engine
110+
...
111+
python_udf_runtime_version: "3.11"
112+
"""
113+
90114
model_config = ConfigDict(populate_by_name=True, extra="allow")
91115

116+
@field_validator("python_udf_runtime_version")
117+
@classmethod
118+
def validate_python_udf_runtime_version(cls, v: str) -> str:
119+
if not re.fullmatch(r"\d+\.\d+(\.\d+)?", v):
120+
raise ValueError(
121+
"python_udf_runtime_version must be a valid Python version string "
122+
f"such as '3.10' or '3.11.2', got {v!r}"
123+
)
124+
return v
125+
92126

93127
class SnowflakeComputeEngine(ComputeEngine):
94128
def get_historical_features(
@@ -110,25 +144,18 @@ def update(
110144
entities_to_delete: Sequence[Entity],
111145
entities_to_keep: Sequence[Entity],
112146
):
113-
stage_context = f'"{self.repo_config.batch_engine.database}"."{self.repo_config.batch_engine.schema_}"'
114-
stage_path = f'{stage_context}."feast_{project}"'
147+
stage_path = f'"{self.repo_config.batch_engine.database}"."{self.repo_config.batch_engine.schema_}"."feast_{project}"'
115148
with GetSnowflakeConnection(self.repo_config.batch_engine) as conn:
116-
query = f"SHOW USER FUNCTIONS LIKE 'FEAST_{project.upper()}%' IN SCHEMA {stage_context}"
117-
cursor = execute_snowflake_statement(conn, query)
118-
function_list = pd.DataFrame(
119-
cursor.fetchall(),
120-
columns=[column.name for column in cursor.description],
121-
)
122-
123-
# if the SHOW FUNCTIONS query returns results,
124-
# assumes that the materialization functions have been deployed
125-
if len(function_list.index) > 0:
126-
click.echo(
127-
f"Materialization functions for {Style.BRIGHT + Fore.GREEN}{project}{Style.RESET_ALL} already detected."
128-
)
129-
click.echo()
130-
return None
131-
149+
# Always (re)deploy the materialization functions, using
150+
# `CREATE OR REPLACE FUNCTION` below, instead of skipping deployment
151+
# when functions already exist. Previously, an early return here
152+
# (triggered by a `SHOW USER FUNCTIONS` check) combined with
153+
# `CREATE FUNCTION IF NOT EXISTS` in the SQL template meant that
154+
# once UDFs were deployed for a project, they were never
155+
# redeployed -- so a config change to `python_udf_runtime_version`
156+
# (e.g. after Snowflake decommissions a runtime) would silently
157+
# keep using the stale, already-deployed runtime version. See
158+
# https://github.com/feast-dev/feast/pull/6608#discussion_r3602461959.
132159
click.echo(
133160
f"Deploying materialization functions for {Style.BRIGHT + Fore.GREEN}{project}{Style.RESET_ALL}"
134161
)
@@ -151,7 +178,11 @@ def update(
151178
sqlCommands = sqlFile.split(";")
152179
for command in sqlCommands:
153180
command = command.replace("STAGE_HOLDER", f"{stage_path}")
154-
query = command.replace("PROJECT_NAME", f"{project}")
181+
command = command.replace("PROJECT_NAME", f"{project}")
182+
query = command.replace(
183+
"RUNTIME_VERSION_HOLDER",
184+
self.repo_config.batch_engine.python_udf_runtime_version,
185+
)
155186
execute_snowflake_statement(conn, query)
156187

157188
return None

‎sdk/python/feast/infra/utils/snowflake/snowpark/snowflake_python_udfs_creation.sql‎

Lines changed: 32 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,127 +1,127 @@
1-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_snowflake_binary_to_bytes_proto(df BINARY)
1+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_snowflake_binary_to_bytes_proto(df BINARY)
22
RETURNS BINARY
33
LANGUAGE PYTHON
4-
RUNTIME_VERSION = '3.9'
4+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
55
PACKAGES = ('protobuf', 'pandas')
66
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_snowflake_binary_to_bytes_proto'
77
IMPORTS = ('@STAGE_HOLDER/feast.zip');
88

9-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_snowflake_varchar_to_string_proto(df VARCHAR)
9+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_snowflake_varchar_to_string_proto(df VARCHAR)
1010
RETURNS BINARY
1111
LANGUAGE PYTHON
12-
RUNTIME_VERSION = '3.9'
12+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
1313
PACKAGES = ('protobuf', 'pandas')
1414
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_snowflake_varchar_to_string_proto'
1515
IMPORTS = ('@STAGE_HOLDER/feast.zip');
1616

17-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_snowflake_array_bytes_to_list_bytes_proto(df ARRAY)
17+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_snowflake_array_bytes_to_list_bytes_proto(df ARRAY)
1818
RETURNS BINARY
1919
LANGUAGE PYTHON
20-
RUNTIME_VERSION = '3.9'
20+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
2121
PACKAGES = ('protobuf', 'pandas')
2222
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_snowflake_array_bytes_to_list_bytes_proto'
2323
IMPORTS = ('@STAGE_HOLDER/feast.zip');
2424

25-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_snowflake_array_varchar_to_list_string_proto(df ARRAY)
25+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_snowflake_array_varchar_to_list_string_proto(df ARRAY)
2626
RETURNS BINARY
2727
LANGUAGE PYTHON
28-
RUNTIME_VERSION = '3.9'
28+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
2929
PACKAGES = ('protobuf', 'pandas')
3030
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_snowflake_array_varchar_to_list_string_proto'
3131
IMPORTS = ('@STAGE_HOLDER/feast.zip');
3232

33-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_snowflake_array_number_to_list_int32_proto(df ARRAY)
33+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_snowflake_array_number_to_list_int32_proto(df ARRAY)
3434
RETURNS BINARY
3535
LANGUAGE PYTHON
36-
RUNTIME_VERSION = '3.9'
36+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
3737
PACKAGES = ('protobuf', 'pandas')
3838
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_snowflake_array_number_to_list_int32_proto'
3939
IMPORTS = ('@STAGE_HOLDER/feast.zip');
4040

41-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_snowflake_array_number_to_list_int64_proto(df ARRAY)
41+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_snowflake_array_number_to_list_int64_proto(df ARRAY)
4242
RETURNS BINARY
4343
LANGUAGE PYTHON
44-
RUNTIME_VERSION = '3.9'
44+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
4545
PACKAGES = ('protobuf', 'pandas')
4646
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_snowflake_array_number_to_list_int64_proto'
4747
IMPORTS = ('@STAGE_HOLDER/feast.zip');
4848

49-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_snowflake_array_float_to_list_double_proto(df ARRAY)
49+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_snowflake_array_float_to_list_double_proto(df ARRAY)
5050
RETURNS BINARY
5151
LANGUAGE PYTHON
52-
RUNTIME_VERSION = '3.9'
52+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
5353
PACKAGES = ('protobuf', 'pandas')
5454
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_snowflake_array_float_to_list_double_proto'
5555
IMPORTS = ('@STAGE_HOLDER/feast.zip');
5656

57-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_snowflake_array_boolean_to_list_bool_proto(df ARRAY)
57+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_snowflake_array_boolean_to_list_bool_proto(df ARRAY)
5858
RETURNS BINARY
5959
LANGUAGE PYTHON
60-
RUNTIME_VERSION = '3.9'
60+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
6161
PACKAGES = ('protobuf', 'pandas')
6262
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_snowflake_array_boolean_to_list_bool_proto'
6363
IMPORTS = ('@STAGE_HOLDER/feast.zip');
6464

65-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_snowflake_array_timestamp_to_list_unix_timestamp_proto(df ARRAY)
65+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_snowflake_array_timestamp_to_list_unix_timestamp_proto(df ARRAY)
6666
RETURNS BINARY
6767
LANGUAGE PYTHON
68-
RUNTIME_VERSION = '3.9'
68+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
6969
PACKAGES = ('protobuf', 'pandas')
7070
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_snowflake_array_timestamp_to_list_unix_timestamp_proto'
7171
IMPORTS = ('@STAGE_HOLDER/feast.zip');
7272

73-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_snowflake_number_to_int32_proto(df NUMBER)
73+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_snowflake_number_to_int32_proto(df NUMBER)
7474
RETURNS BINARY
7575
LANGUAGE PYTHON
76-
RUNTIME_VERSION = '3.9'
76+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
7777
PACKAGES = ('protobuf', 'pandas')
7878
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_snowflake_number_to_int32_proto'
7979
IMPORTS = ('@STAGE_HOLDER/feast.zip');
8080

81-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_snowflake_number_to_int64_proto(df NUMBER)
81+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_snowflake_number_to_int64_proto(df NUMBER)
8282
RETURNS BINARY
8383
LANGUAGE PYTHON
84-
RUNTIME_VERSION = '3.9'
84+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
8585
PACKAGES = ('protobuf', 'pandas')
8686
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_snowflake_number_to_int64_proto'
8787
IMPORTS = ('@STAGE_HOLDER/feast.zip');
8888

89-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_snowflake_float_to_double_proto(df DOUBLE)
89+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_snowflake_float_to_double_proto(df DOUBLE)
9090
RETURNS BINARY
9191
LANGUAGE PYTHON
92-
RUNTIME_VERSION = '3.9'
92+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
9393
PACKAGES = ('protobuf', 'pandas')
9494
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_snowflake_float_to_double_proto'
9595
IMPORTS = ('@STAGE_HOLDER/feast.zip');
9696

97-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_snowflake_boolean_to_bool_proto(df BOOLEAN)
97+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_snowflake_boolean_to_bool_proto(df BOOLEAN)
9898
RETURNS BINARY
9999
LANGUAGE PYTHON
100-
RUNTIME_VERSION = '3.9'
100+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
101101
PACKAGES = ('protobuf', 'pandas')
102102
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_snowflake_boolean_to_bool_boolean_proto'
103103
IMPORTS = ('@STAGE_HOLDER/feast.zip');
104104

105-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_snowflake_timestamp_to_unix_timestamp_proto(df NUMBER)
105+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_snowflake_timestamp_to_unix_timestamp_proto(df NUMBER)
106106
RETURNS BINARY
107107
LANGUAGE PYTHON
108-
RUNTIME_VERSION = '3.9'
108+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
109109
PACKAGES = ('protobuf', 'pandas')
110110
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_snowflake_timestamp_to_unix_timestamp_proto'
111111
IMPORTS = ('@STAGE_HOLDER/feast.zip');
112112

113-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_serialize_entity_keys(names ARRAY, data ARRAY, types ARRAY)
113+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_serialize_entity_keys(names ARRAY, data ARRAY, types ARRAY)
114114
RETURNS BINARY
115115
LANGUAGE PYTHON
116-
RUNTIME_VERSION = '3.9'
116+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
117117
PACKAGES = ('protobuf', 'pandas')
118118
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_serialize_entity_keys'
119119
IMPORTS = ('@STAGE_HOLDER/feast.zip');
120120

121-
CREATE FUNCTION IF NOT EXISTS feast_PROJECT_NAME_entity_key_proto_to_string(names ARRAY, data ARRAY, types ARRAY)
121+
CREATE OR REPLACE FUNCTION feast_PROJECT_NAME_entity_key_proto_to_string(names ARRAY, data ARRAY, types ARRAY)
122122
RETURNS BINARY
123123
LANGUAGE PYTHON
124-
RUNTIME_VERSION = '3.9'
124+
RUNTIME_VERSION = 'RUNTIME_VERSION_HOLDER'
125125
PACKAGES = ('protobuf', 'pandas')
126126
HANDLER = 'feast.infra.utils.snowflake.snowpark.snowflake_udfs.feast_entity_key_proto_to_string'
127127
IMPORTS = ('@STAGE_HOLDER/feast.zip')

0 commit comments

Comments
 (0)