-
Notifications
You must be signed in to change notification settings - Fork 108
RFC: Append-only tables #2040
Copy link
Copy link
Open
1 / 11 of 1 issue completedLabels
DBSP coreRelated to the core DBSP libraryRelated to the core DBSP librarySQL compilerRelated to the SQL compilerRelated to the SQL compiler
Description
There are several situations where the compiler + runtime could optimize operations on append-only tables:
-
min,max, -
argmax,argminaggregates can be evaluated by storing the current max only. -
joinwith foreign keys
We need a mechanism by which the compiler can request the DBSP layer to use these operators. One solution is to build custom operators for this case.
Related to the above, but separate, the compiler can use the fact that max grows monotonically to GC relations that get filtered by this max, e.g., select * from my_table where ts > select(max(...)).
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
DBSP coreRelated to the core DBSP libraryRelated to the core DBSP librarySQL compilerRelated to the SQL compilerRelated to the SQL compiler