Skip to content

Commit 79b33ce

Browse files
addenergyxclaude
addenergyx
and
claude
authored
feat: Add opt-in filter_by_created_timestamp cutoff to get_historical_features (#6617)
* feat: Add opt-in at_event_time created_timestamp cutoff to get_historical_features When a feature view has a created_timestamp_column, it is currently used only as a dedup tiebreaker in point-in-time joins, so retrieval can serve feature values whose created_timestamp is after the entity row's event timestamp (backfills, late corrections). This leaks future information into training data and makes training sets non-reproducible. Add an opt-in at_event_time flag (default False) to get_historical_features that adds a created_timestamp <= entity_timestamp predicate to the point-in-time join, so retrieval reflects what was known as of each entity row's timestamp. Implemented for the SQL template stores (BigQuery, Redshift, Spark, Trino, Athena, Postgres, ClickHouse, Couchbase), the ibis-based stores (DuckDB, MSSQL, Oracle) and the dask store. Snowflake, Remote and Ray raise NotImplementedError when the flag is set. Fixes #6615 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: David <david-adeniji@hotmail.co.uk> * test: Add universal integration test and docs for at_event_time Adds a universal offline store integration test covering at_event_time (default returns backfilled values, at_event_time=True only returns values created at or before the entity timestamp, stores without support skip via NotImplementedError) and documents the flag on the point-in-time joins concept page. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: David <david-adeniji@hotmail.co.uk> * refactor: Rename at_event_time to filter_by_created_timestamp and address review feedback Rename the flag to filter_by_created_timestamp to match the repo's mechanical flag naming and sit alongside created_timestamp_column. Review fixes: - HybridOfflineStore now forwards optional kwargs to the delegated store instead of raising TypeError for supported underlying stores. - The dask filter now excludes feature rows with a null created timestamp (consistent with the SQL predicate) while still keeping unmatched entity rows from the left join. - The integration test only treats NotImplementedError from get_historical_features itself as an unsupported-store skip. - Add template-render tests for all SQL dialects and a dask null created-timestamp test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: David <david-adeniji@hotmail.co.uk> * refactor: Centralize filter_by_created_timestamp support behind a store capability flag Simplification pass over the feature, reviewed again by Codex: - Declare support via OfflineStore.supports_filter_by_created_timestamp (default False) and check it once in the passthrough provider via ensure_filter_by_created_timestamp_supported, replacing the three per-store NotImplementedError guards. Unsupported stores can no longer silently ignore the flag. - The hybrid store re-checks the resolved child store before delegating. - Standardize every supporting store on an explicit filter_by_created_timestamp parameter instead of kwargs.get lookups. - Document the flag in the OfflineStore.get_historical_features docstring alongside start_date/end_date. - Return the dask created-timestamp filter lazily so the mask fuses into the following _drop_duplicates persist instead of forcing an extra materialization. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: David <david-adeniji@hotmail.co.uk> * chore: Trim comments to the non-obvious constraints Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: David <david-adeniji@hotmail.co.uk> * docs: Tighten filter_by_created_timestamp docstring Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: David <david-adeniji@hotmail.co.uk> * docs: Condense filter_by_created_timestamp caveats into a hint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: David <david-adeniji@hotmail.co.uk> * fix: Keep entity rows whose candidate versions are all future-created The Dask cutoff filtered rows after the left join, so an entity whose every candidate version was created after its timestamp lost all of its rows and disappeared from the result. get_historical_features must preserve entity-dataframe cardinality and return null features instead. Blank the feature-view columns rather than dropping the row, which leaves it indistinguishable from an unmatched left join. _drop_duplicates already sorts nulls first and keeps the last row, so a valid version still wins where one exists and a blanked row survives only when nothing else does. The unit tests now set fv.entity_columns. Without it the derived join keys are empty and _merge silently cross joins, which hid the per-entity behaviour because the existing cases all used a single entity row. Signed-off-by: David <david-adeniji@hotmail.co.uk> * refactor: Normalize created timestamp on read, not in the join predicate The cutoff predicate cast created_timestamp to UTC while the event-timestamp comparison beside it did not. read_fv normalizes the event timestamp when it reads the source and left the created timestamp alone, so the predicate was compensating for a missing normalization at the one comparison site. Normalize both on read instead. The predicate then needs no cast and matches its neighbour. deduplicate() orders by created_timestamp_column regardless of the cutoff flag, so the normalization is unconditional rather than gated on it; casting a column that is already tz-aware compiles away, so this leaves the emitted query unchanged for tz-aware sources and retires the "mutate only if tz-naive" TODO. Signed-off-by: David <david-adeniji@hotmail.co.uk> * Shorten comments in the created timestamp cutoff Signed-off-by: David <david-adeniji@hotmail.co.uk> --------- Signed-off-by: David <david-adeniji@hotmail.co.uk> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent faf85e0 commit 79b33ce

22 files changed

Lines changed: 618 additions & 8 deletions

File tree

‎docs/getting-started/concepts/point-in-time-joins.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,3 +62,26 @@ Below is the resulting joined training dataframe. It contains both the original
6262

6363
Three feature rows were successfully joined to the entity dataframe rows. The first row in the entity dataframe was older than the earliest feature rows in the feature view and could not be joined. The last row in the entity dataframe was outside of the TTL window \(the event happened 11 hours after the feature row\) and also couldn't be joined.
6464

65+
## Retrieving features as of the event time
66+
67+
By default, point-in-time joins only constrain the feature's event timestamp. If a data source also has a `created_timestamp_column`, it is used to deduplicate rows that share an event timestamp \(the row with the highest created timestamp wins\), but it is not otherwise filtered. This means a value that was backfilled or corrected *after* an entity dataframe timestamp can still be returned for it.
68+
69+
To restrict retrieval to feature values that were already available at each entity row's timestamp, pass `filter_by_created_timestamp=True`:
70+
71+
```python
72+
training_df = store.get_historical_features(
73+
entity_df=entity_df,
74+
features = [
75+
'driver_hourly_stats:trips_today',
76+
'driver_hourly_stats:earnings_today'
77+
],
78+
filter_by_created_timestamp=True,
79+
)
80+
```
81+
82+
This adds a `created_timestamp <= entity_timestamp` condition to the join, so each entity dataframe row only sees feature values whose created timestamp is at or before its own timestamp. This is useful to keep backfilled values from leaking into training data, and to reproduce what the online store would have served at each event time \(assuming the created timestamp reflects when the value became available online\).
83+
84+
{% hint style="info" %}
85+
Rows with a NULL created timestamp are excluded when the flag is enabled, so the column should be non-null. Not all offline stores support this flag yet; unsupported stores raise an error rather than silently ignoring it.
86+
{% endhint %}
87+

‎sdk/python/feast/feature_store.py‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1940,6 +1940,7 @@ def get_historical_features(
19401940
full_feature_names: bool = False,
19411941
start_date: Optional[datetime] = None,
19421942
end_date: Optional[datetime] = None,
1943+
filter_by_created_timestamp: bool = False,
19431944
) -> RetrievalJob:
19441945
"""Enrich an entity dataframe with historical feature values for either training or batch scoring.
19451946
@@ -1971,6 +1972,11 @@ def get_historical_features(
19711972
Required when entity_df is not provided.
19721973
end_date (Optional[datetime]): End date for the timestamp range when retrieving features without entity_df.
19731974
Required when entity_df is not provided. By default, the current time is used.
1975+
filter_by_created_timestamp (bool): If True, exclude feature values whose created timestamp
1976+
(the batch source's ``created_timestamp_column``) is later than the entity row's event
1977+
timestamp, so retrieval only reflects what was known at the event time and backfilled
1978+
values cannot leak into training data. Feature views without a
1979+
``created_timestamp_column`` are unaffected. Defaults to False.
19741980
19751981
Returns:
19761982
RetrievalJob which can be used to materialize the results.
@@ -2072,11 +2078,13 @@ def get_historical_features(
20722078
provider = self._get_provider()
20732079

20742080
# Optional kwargs
2075-
kwargs = {}
2081+
kwargs: Dict[str, Any] = {}
20762082
if start_date is not None:
20772083
kwargs["start_date"] = start_date
20782084
if end_date is not None:
20792085
kwargs["end_date"] = end_date
2086+
if filter_by_created_timestamp:
2087+
kwargs["filter_by_created_timestamp"] = filter_by_created_timestamp
20802088

20812089
_retrieval_start = time.monotonic()
20822090

‎sdk/python/feast/infra/offline_stores/bigquery.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,8 @@ def project_id_exists(cls, v, values, **kwargs):
137137

138138

139139
class BigQueryOfflineStore(OfflineStore):
140+
supports_filter_by_created_timestamp = True
141+
140142
@staticmethod
141143
def pull_latest_from_table_or_query(
142144
config: RepoConfig,
@@ -273,6 +275,7 @@ def get_historical_features(
273275
registry: BaseRegistry,
274276
project: str,
275277
full_feature_names: bool = False,
278+
filter_by_created_timestamp: bool = False,
276279
**kwargs: Any,
277280
) -> RetrievalJob:
278281
# TODO: Add entity_df validation in order to fail before interacting with BigQuery
@@ -388,6 +391,7 @@ def query_generator() -> Iterator[str]:
388391
entity_df_columns=entity_schema_keys,
389392
query_template=MULTIPLE_FEATURE_VIEW_POINT_IN_TIME_JOIN,
390393
full_feature_names=full_feature_names,
394+
filter_by_created_timestamp=filter_by_created_timestamp,
391395
)
392396

393397
try:
@@ -1734,6 +1738,10 @@ def arrow_schema_to_bq_schema(arrow_schema: pyarrow.Schema) -> List[SchemaField]
17341738
AND subquery.event_timestamp >= Timestamp_sub(entity_dataframe.entity_timestamp, interval {{ featureview.ttl }} second)
17351739
{% endif %}
17361740
1741+
{% if filter_by_created_timestamp and featureview.created_timestamp_column %}
1742+
AND subquery.created_timestamp <= entity_dataframe.entity_timestamp
1743+
{% endif %}
1744+
17371745
{% for entity in featureview.entities %}
17381746
AND subquery.{{ entity }} = entity_dataframe.{{ entity }}
17391747
{% endfor %}

‎sdk/python/feast/infra/offline_stores/contrib/athena_offline_store/athena.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,8 @@ class AthenaOfflineStoreConfig(FeastConfigBaseModel):
6666

6767

6868
class AthenaOfflineStore(OfflineStore):
69+
supports_filter_by_created_timestamp = True
70+
6971
@staticmethod
7072
def pull_latest_from_table_or_query(
7173
config: RepoConfig,
@@ -195,6 +197,7 @@ def get_historical_features(
195197
registry: BaseRegistry,
196198
project: str,
197199
full_feature_names: bool = False,
200+
filter_by_created_timestamp: bool = False,
198201
) -> RetrievalJob:
199202
assert isinstance(config.offline_store, AthenaOfflineStoreConfig)
200203
for fv in feature_views:
@@ -252,6 +255,7 @@ def query_generator() -> Iterator[str]:
252255
entity_df_columns=entity_schema.keys(),
253256
query_template=MULTIPLE_FEATURE_VIEW_POINT_IN_TIME_JOIN,
254257
full_feature_names=full_feature_names,
258+
filter_by_created_timestamp=filter_by_created_timestamp,
255259
)
256260

257261
try:
@@ -651,6 +655,10 @@ def _get_entity_df_event_timestamp_range(
651655
AND subquery.event_timestamp >= entity_dataframe.entity_timestamp - {{ featureview.ttl }} * interval '1' second
652656
{% endif %}
653657
658+
{% if filter_by_created_timestamp and featureview.created_timestamp_column %}
659+
AND subquery.created_timestamp <= entity_dataframe.entity_timestamp
660+
{% endif %}
661+
654662
{% for entity in featureview.entities %}
655663
AND subquery.{{ entity }} = entity_dataframe.{{ entity }}
656664
{% endfor %}

‎sdk/python/feast/infra/offline_stores/contrib/clickhouse_offline_store/clickhouse.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ class ClickhouseOfflineStoreConfig(ClickhouseConfig):
3939

4040

4141
class ClickhouseOfflineStore(OfflineStore):
42+
supports_filter_by_created_timestamp = True
43+
4244
@staticmethod
4345
def get_historical_features(
4446
config: RepoConfig,
@@ -48,6 +50,7 @@ def get_historical_features(
4850
registry: BaseRegistry,
4951
project: str,
5052
full_feature_names: bool = False,
53+
filter_by_created_timestamp: bool = False,
5154
**kwargs,
5255
) -> RetrievalJob:
5356
assert isinstance(config.offline_store, ClickhouseOfflineStoreConfig)
@@ -123,6 +126,7 @@ def query_generator() -> Iterator[str]:
123126
entity_df_columns=entity_schema.keys(),
124127
query_template=MULTIPLE_FEATURE_VIEW_POINT_IN_TIME_JOIN,
125128
full_feature_names=full_feature_names,
129+
filter_by_created_timestamp=filter_by_created_timestamp,
126130
)
127131
yield query
128132
finally:
@@ -539,6 +543,10 @@ def _append_alias(field_names: List[str], alias: str) -> List[str]:
539543
{% if featureview.ttl == 0 %}{% else %}
540544
AND subquery.event_timestamp >= entity_dataframe.entity_timestamp - interval {{ featureview.ttl }} second
541545
{% endif %}
546+
547+
{% if filter_by_created_timestamp and featureview.created_timestamp_column %}
548+
AND subquery.created_timestamp <= entity_dataframe.entity_timestamp
549+
{% endif %}
542550
),
543551
544552
/*

‎sdk/python/feast/infra/offline_stores/contrib/couchbase_offline_store/couchbase.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,8 @@ class CouchbaseColumnarOfflineStoreConfig(FeastConfigBaseModel):
6464

6565

6666
class CouchbaseColumnarOfflineStore(OfflineStore):
67+
supports_filter_by_created_timestamp = True
68+
6769
@staticmethod
6870
def pull_latest_from_table_or_query(
6971
config: RepoConfig,
@@ -136,6 +138,7 @@ def get_historical_features(
136138
registry: BaseRegistry,
137139
project: str,
138140
full_feature_names: bool = False,
141+
filter_by_created_timestamp: bool = False,
139142
) -> RetrievalJob:
140143
"""
141144
Retrieve historical features using point-in-time joins.
@@ -197,6 +200,7 @@ def query_generator() -> Iterator[str]:
197200
entity_df_columns=entity_schema.keys(),
198201
query_template=MULTIPLE_FEATURE_VIEW_POINT_IN_TIME_JOIN,
199202
full_feature_names=full_feature_names,
203+
filter_by_created_timestamp=filter_by_created_timestamp,
200204
)
201205
yield query
202206
finally:
@@ -481,6 +485,7 @@ def build_point_in_time_query(
481485
entity_df_columns: KeysView[str],
482486
query_template: str,
483487
full_feature_names: bool = False,
488+
filter_by_created_timestamp: bool = False,
484489
) -> str:
485490
"""Build point-in-time query between each feature view table and the entity dataframe for Couchbase Columnar"""
486491
template = Environment(loader=BaseLoader()).from_string(source=query_template)
@@ -507,6 +512,7 @@ def build_point_in_time_query(
507512
"featureviews": feature_view_query_contexts,
508513
"full_feature_names": full_feature_names,
509514
"final_output_feature_names": final_output_feature_names,
515+
"filter_by_created_timestamp": filter_by_created_timestamp,
510516
}
511517

512518
query = template.render(template_context)
@@ -620,6 +626,9 @@ def _get_entity_schema(
620626
{% if featureview.ttl == 0 %}{% else %}
621627
AND date_diff_str(entity_dataframe.entity_timestamp, subquery.event_timestamp, "second") <= {{ featureview.ttl }}
622628
{% endif %}
629+
{% if filter_by_created_timestamp and featureview.created_timestamp_column %}
630+
AND subquery.created_timestamp <= entity_dataframe.entity_timestamp
631+
{% endif %}
623632
{% for entity in featureview.entities %}
624633
AND subquery.`{{ entity }}` = entity_dataframe.`{{ entity }}`
625634
{% endfor %}

‎sdk/python/feast/infra/offline_stores/contrib/mssql_offline_store/mssql.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,8 @@ class MsSqlServerOfflineStoreConfig(FeastConfigBaseModel):
118118

119119

120120
class MsSqlServerOfflineStore(OfflineStore):
121+
supports_filter_by_created_timestamp = True
122+
121123
@staticmethod
122124
def pull_latest_from_table_or_query(
123125
config: RepoConfig,
@@ -151,6 +153,7 @@ def get_historical_features(
151153
registry: BaseRegistry,
152154
project: str,
153155
full_feature_names: bool = False,
156+
filter_by_created_timestamp: bool = False,
154157
) -> RetrievalJob:
155158
# TODO avoid this conversion
156159
if type(entity_df) == str:
@@ -168,6 +171,7 @@ def get_historical_features(
168171
data_source_reader=_build_data_source_reader(config),
169172
data_source_writer=_build_data_source_writer(config),
170173
event_expire_timestamp_fn=mssql_event_expire_timestamp_fn,
174+
filter_by_created_timestamp=filter_by_created_timestamp,
171175
)
172176

173177
@staticmethod

‎sdk/python/feast/infra/offline_stores/contrib/oracle_offline_store/oracle.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -486,6 +486,8 @@ def _oracle_try_execute_ddl(con, ddl: str) -> None:
486486

487487

488488
class OracleOfflineStore(OfflineStore):
489+
supports_filter_by_created_timestamp = True
490+
489491
@staticmethod
490492
def pull_latest_from_table_or_query(
491493
config: RepoConfig,
@@ -521,6 +523,7 @@ def get_historical_features(
521523
registry: BaseRegistry,
522524
project: str,
523525
full_feature_names: bool = False,
526+
filter_by_created_timestamp: bool = False,
524527
**kwargs,
525528
) -> RetrievalJob:
526529
if not feature_views:
@@ -554,6 +557,7 @@ def get_historical_features(
554557
full_feature_names=full_feature_names,
555558
data_source_reader=_build_data_source_reader(config, con=con),
556559
data_source_writer=_build_data_source_writer(config, con=con),
560+
filter_by_created_timestamp=filter_by_created_timestamp,
557561
)
558562

559563
@staticmethod

‎sdk/python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,8 @@ class PostgreSQLOfflineStoreConfig(PostgreSQLConfig):
7575

7676

7777
class PostgreSQLOfflineStore(OfflineStore):
78+
supports_filter_by_created_timestamp = True
79+
7880
@staticmethod
7981
def pull_latest_from_table_or_query(
8082
config: RepoConfig,
@@ -135,6 +137,7 @@ def get_historical_features(
135137
registry: BaseRegistry,
136138
project: str,
137139
full_feature_names: bool = False,
140+
filter_by_created_timestamp: bool = False,
138141
**kwargs,
139142
) -> RetrievalJob:
140143
assert isinstance(config.offline_store, PostgreSQLOfflineStoreConfig)
@@ -222,6 +225,7 @@ def query_generator() -> Iterator[str]:
222225
use_cte=use_cte,
223226
start_date=start_date,
224227
end_date=end_date,
228+
filter_by_created_timestamp=filter_by_created_timestamp,
225229
)
226230
finally:
227231
# Only cleanup if we created a table
@@ -693,6 +697,7 @@ def build_point_in_time_query(
693697
use_cte: bool = False,
694698
start_date: Optional[datetime] = None,
695699
end_date: Optional[datetime] = None,
700+
filter_by_created_timestamp: bool = False,
696701
) -> str:
697702
"""Build point-in-time query between each feature view table and the entity dataframe for PostgreSQL"""
698703
template = Environment(loader=BaseLoader()).from_string(source=query_template)
@@ -723,6 +728,7 @@ def build_point_in_time_query(
723728
"use_cte": use_cte,
724729
"start_date": start_date,
725730
"end_date": end_date,
731+
"filter_by_created_timestamp": filter_by_created_timestamp,
726732
}
727733

728734
query = template.render(template_context)
@@ -965,6 +971,10 @@ def _get_entity_schema(
965971
AND subquery.event_timestamp >= entity_dataframe.entity_timestamp - {{ featureview.ttl }} * interval '1' second
966972
{% endif %}
967973
974+
{% if filter_by_created_timestamp and featureview.created_timestamp_column %}
975+
AND subquery.created_timestamp <= entity_dataframe.entity_timestamp
976+
{% endif %}
977+
968978
{% for entity in featureview.entities %}
969979
AND subquery."{{ entity }}" = entity_dataframe."{{ entity }}"
970980
{% endfor %}

‎sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,8 @@ class SparkFeatureViewQueryContext(offline_utils.FeatureViewQueryContext):
9393

9494

9595
class SparkOfflineStore(OfflineStore):
96+
supports_filter_by_created_timestamp = True
97+
9698
@staticmethod
9799
def pull_latest_from_table_or_query(
98100
config: RepoConfig,
@@ -172,6 +174,7 @@ def get_historical_features(
172174
registry: BaseRegistry,
173175
project: str,
174176
full_feature_names: bool = False,
177+
filter_by_created_timestamp: bool = False,
175178
**kwargs,
176179
) -> RetrievalJob:
177180
assert isinstance(config.offline_store, SparkOfflineStoreConfig)
@@ -332,6 +335,7 @@ def get_historical_features(
332335
entity_df_columns=entity_schema_keys,
333336
query_template=MULTIPLE_FEATURE_VIEW_POINT_IN_TIME_JOIN,
334337
full_feature_names=full_feature_names,
338+
filter_by_created_timestamp=filter_by_created_timestamp,
335339
)
336340

337341
return SparkRetrievalJob(
@@ -1792,6 +1796,10 @@ def _cast_data_frame(
17921796
AND subquery.event_timestamp >= entity_dataframe.entity_timestamp - {{ featureview.ttl }} * interval '1' second
17931797
{% endif %}
17941798
1799+
{% if filter_by_created_timestamp and featureview.created_timestamp_column %}
1800+
AND subquery.created_timestamp <= entity_dataframe.entity_timestamp
1801+
{% endif %}
1802+
17951803
{% for entity in featureview.entities %}
17961804
AND subquery.{{ entity }} = entity_dataframe.{{ entity }}
17971805
{% endfor %}

0 commit comments

Comments
 (0)