Skip to content

RFC: Append-only tables #2040

@ryzhyk

Description

@ryzhyk

There are several situations where the compiler + runtime could optimize operations on append-only tables:

  • min, max,
  • argmax, argmin aggregates can be evaluated by storing the current max only.
  • join with foreign keys

We need a mechanism by which the compiler can request the DBSP layer to use these operators. One solution is to build custom operators for this case.

Related to the above, but separate, the compiler can use the fact that max grows monotonically to GC relations that get filtered by this max, e.g., select * from my_table where ts > select(max(...)).

Metadata

Metadata

Labels

DBSP coreRelated to the core DBSP librarySQL compilerRelated to the SQL compiler

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions