Repository navigation
Deprecate implicit non-spec type changes in update_column - #4410
Open
rahulsmahadev wants to merge 2 commits into
Open
rahulsmahadev wants to merge 2 commits into
rahulsmahadev wants to merge 2 commits into
Conversation
update_column applied string <-> binary and fixed(16) -> uuid silently, using the read-side promote(). Check the spec's promotion list instead. Those three pairs now warn with a DeprecationWarning, and are rejected in a later release unless allow_incompatible_changes=True is set. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
@geruh, could you review the compatibility transition for these previously accepted schema changes? @Anurag-S1ngh, tagging you as the latest author of |
Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
update_columnchecks a type change withpromote(), the read-side rule that the Avro and Parquet readers use. That rule is more lenient than the table spec: it also acceptsstring -> binary,binary -> stringandfixed(16) -> uuid. So a populatedstringcolumn can be changed tobinarywithout any warning, and the commit succeeds:The spec's "Valid primitive type promotions" table does not list these pairs. Java's
SchemaUpdate.updateColumnrejects them throughTypeUtil.isPromotionAllowed, which allows equal types,int -> long,float -> doubleand decimal widening with the same scale.This PR validates primitive type changes using Java's v1/v2 promotion rules, while keeping PyIceberg's existing
unknown -> primitivebehaviour. It does not add v3date -> timestampordate -> timestamp_nssupport, and it does not broaden unknown-type conversions. Validation runs whenfield_typeis supplied andallow_incompatible_changesis false.promote()is unchanged, because the Avro resolver, the Arrow projection, theunion_by_nameno-op check and append compatibility (_check_schema_compatible) still rely on its leniency.Behaviour change: without
allow_incompatible_changes=True,update_columnwarns before applyingstring -> binary,binary -> stringorfixed[16] -> uuid. The warning is aDeprecationWarningthat says the change is deprecated in 0.13.0 and will be removed in 0.14.0. A subsequent release will require the explicit opt-in. Every other invalid change still raisesValidationError, as onmain.allow_incompatible_changes=Trueapplies these three changes without a warning.union_by_namewarns when an incoming UUID field replaces an existingfixed[16]field, including nested struct fields, list elements and map values. String/binary unions remain unchanged, becauseunion_by_namechecks them withpromote()before it callsupdate_column. An incomingfixed[16]field over an existing UUID field also remains a no-op.Precedent: Fokko added the string <-> binary allowed-update tests in apache/iceberg#8374 (September 2023), and #207 only moved that test file. #986 added
test_table_scan_default_to_large_types, which changed a string column to binary withupdate_column. #1669 later changed that test to create the column as binary, so the update became a no-op.Alternative for maintainers: reject these three changes immediately instead of warning first. #3613 rejected decimal promotions that change the scale without a deprecation cycle, and Fokko approved it. This PR chooses a warning cycle, because these changes were tested as supported (apache/iceberg#8374), and AGENTS.md asks for a deprecation cycle before breaking public API. Choosing immediate rejection means replacing the warning branch in
update_columnwith aValidationError.Other engines: in a local DuckDB 2.0.0
iceberg_scanreproduction, promoting a populated string column to binary made a non-ASCII value fail to read, and changed the four-character string\x41to the single byteA. PyIceberg continued to read the original UTF-8 bytes.Are these changes tested?
Yes.
tests/test_schema.pycovers the following.test_update_column_rejects_invalid_promotions: the exact error message for each invalid change (for exampleint -> float, a decimal scale change andfixed[15] -> uuid), unchanged table metadata after the rejections, and the valid promotions committed and read back after a reload.test_update_column_warns_for_non_spec_promotions: the three deprecated changes warn with the exact message, commit, and read back after a reload.test_update_column_no_warning_with_allow_incompatible_changes: the explicit opt-in applies the same three changes without a warning.test_union_by_name_fixed_uuid_and_string_binary:fixed(16)and UUID in both directions, string/binary unchanged, and a list element.test_update_column_promotes_unknown_to_primitive_only: the existingunknownbehaviour, on a v3 table.On
main, the three warning tests fail withDID NOT WARNand the union case fails because no warning is emitted. With this change they all pass. Intests/integration/test_rest_schema.py,test_allowed_updatesandtest_disallowed_updatesexpect the deprecation warning for string <-> binary, andtest_alter_map_keystill opts in withallow_incompatible_changes=Trueto reach the map key check.Are there any user-facing changes?
Yes. Without
allow_incompatible_changes=True,update_columnemits aDeprecationWarningbefore applyingstring -> binary,binary -> stringorfixed[16] -> uuid. Without this opt-in,union_by_namewarns when incoming UUID fields replace existing fixed[16] fields. String/binary unions and incoming fixed[16] fields over existing UUID fields remain type no-ops. The deprecated implicit changes are scheduled to require the explicit opt-in in 0.14.0. Please add thechangeloglabel.This pull request and its description were written by Isaac.