-
Notifications
You must be signed in to change notification settings - Fork 108
GC for AsOf joins #2010
Copy link
Copy link
Closed
Labels
DBSP coreRelated to the core DBSP libraryRelated to the core DBSP librarySQL compilerRelated to the SQL compilerRelated to the SQL compiler
Description
Consider an asof-join query T1 ASOF JOIN T2 MATCH_CONDITION(T1.ts1 >= T2.ts2), where ts1 and ts2 are timestamp columns in T1 and T2 with LATENESS values L1 and L2 respectively.
Let TS1 and TS2 be the latest observed timestamps in T1 and T2.
Let WL = min(TS1 - L1, TS2 - L2), i.e., the timestamp before which neither of the two streams can see any updates.
-
We can GC
T1usingintegrate_trace_retain_valuesand the waterline value equal toWL. -
To GC the right side, we need the mechanism described in RFC: Generalizing the GC API to support group operators #1975. Specifically, we need to preserve the latest value preceding
WLin each group and all newer values.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
DBSP coreRelated to the core DBSP libraryRelated to the core DBSP librarySQL compilerRelated to the SQL compilerRelated to the SQL compiler