Skip to content

Row lineage: Read _last_updated_sequence_number - #4413

Open
rambleraptor wants to merge 2 commits into
apache:mainfrom
rambleraptor:row-lineage-read-seq-number
Open

rambleraptor wants to merge 2 commits into
apache:mainfrom
rambleraptor:row-lineage-read-seq-number

Conversation

@rambleraptor

Copy link
Copy Markdown
Collaborator

Rationale for this change

I can break this up into a couple separate PRs, but I wanted to finish up row lineage.

This does two things:

  • Puts first_row_id in inspect.files()
  • Lets users read _last_updated_sequence_number

Are these changes tested?

Included unit test + updated our Spark test (from the last PR)

Are there any user-facing changes?

Carry the data sequence number of each data file on FileScanTask, and fill
null _last_updated_sequence_number values with it, as the Java readers do.
The value is null when the file has no first row ID or no data sequence
number, such as for files planned by a REST server.

Part of apache#4080.
Add the first_row_id column to the files metadata tables, matching the
Java files table schema.

Part of apache#4080.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant