Repository navigation
Reject colliding names in case-insensitive Schema lookups - #4389
rahulsmahadev wants to merge 2 commits into
Conversation
The lower-case name index kept only the last of two columns whose names differ only by case (ID and id), so case-insensitive lookups silently used it. Raise ValueError when two field IDs share a lower-case name, as Java does, and let the schema update and metadata column checks propagate that error instead of reading it as a missing name. Exact-case lookups are unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
|
@rambleraptor, this updates the metadata-column lookup you added in #4375 so schema name collisions propagate. @kevinjqliu, you recently worked on Schema behavior in #3958. Could you take a look at the Java-compatible lookup behavior and the caller error handling when you have a chance? |
Fokko
left a comment
There was a problem hiding this comment.
Thanks for this, the alignment with Java's TypeUtil.indexByLowerCaseName and Schema.caseInsensitiveFindField looks right to me, including failing for unambiguous names when the schema has a collision.
One correction to the description: it says StructType.field_by_name is outside this change and can still return the first matching field, which reads as if that matches Java. Java's Types.StructType.caseInsensitiveField builds its index with ImmutableMap.Builder.build(), which throws on duplicate keys, so a case-insensitive unionByNameWith on a struct with ID and id fails in Java. Could you reword that part to call it out as a remaining divergence? Fine to handle it in a follow-up issue.
| raise ValueError(f"Could not find field with id: {name_or_id}") | ||
| return self._lazy_id_to_field[name_or_id] | ||
|
|
||
| field = self._find_field_or_none(name_or_id, case_sensitive) |
There was a problem hiding this comment.
Since this is the user facing change, could we also document the collision in the public docstrings? find_field and select still only say ValueError is raised when the field cannot be found. Something like ValueError: When case_sensitive is False and two fields have names that differ only by case. in both.
There was a problem hiding this comment.
Done in 6397704: find_field and select now list the collision under Raises.
Review feedback: the public docstrings now list the ValueError raised when case_sensitive is False and two fields have names that differ only by case. Co-authored-by: Isaac <no-reply@databricks.com>
|
Thanks @Fokko! I reworded the |
Rationale for this change
A case-insensitive lookup on a schema with two columns whose names differ only by case uses the last one.
Schema._lazy_name_to_id_lowerbuilds the lower-case index with a dict comprehension, soidoverwritesID, andSchema.find_fieldandSchema.selectuse that index for case-insensitive name lookups:The same wrong column is used by sort order updates and by schema updates (
delete_column("ID")dropsid,rename_column("ID", "key")renamesid).Java does not pick a column here.
TypeUtil.indexByLowerCaseNamethrowsCannot build lower case index: ID and id collide, andSchema.caseInsensitiveFindFieldandSchemaUpdateuse that index, so the lookup fails.This change makes the index raise
ValueError: Cannot build lower case index: ID and id collide(the same message as Java) when two different field IDs share a lower-case name. As in Java'sSchemalookup, building the lower-case index fails for any requested name when the schema contains a collision, including when the requested name is unambiguous. Names that map to the same field ID, such as the short name of a list element struct field, are not collisions. Case-sensitive lookups do not change.StructType.field_by_nameuses a separate lookup and is outside this change, so it remains a divergence from Java: it can still return the first matching field (it servesunion_by_name). Java'sTypes.StructType.caseInsensitiveFieldbuilds its index withImmutableMap.Builder.build(), which throws on duplicate keys, so a case-insensitiveunionByNameWithon a struct withIDandidfails in Java. I will follow up with a separate PR for that.Callers that treated any
ValueErrorfrom a lookup as "not found" (UpdateSchema.add_column, the move methods, the identifier field check on commit, and the metadata column check in scan projection) now use a private lookup that returnsNonefor a missing name and still raises for an ambiguous one. Without updating the add-column check, an index-only fix would lettable.update_schema(case_sensitive=False).add_column("Id", LongType()).commit()add a third case variant.Are these changes tested?
Yes.
test_case_insensitive_operations_raise_on_case_variant_columnsintests/table/test_init.pyruns a delete, a scan, a sort order update, and schema updates withcase_sensitive=Falseon a table withIDandid, for the memory, sql and sql_without_rowcount catalogs. Each must raise, and the table must be unchanged afterwards. It fails on main, where the delete does not raise.test_case_sensitive_and_unambiguous_operations_are_unchangedchecks exact-case lookups on the same table and a case-insensitive delete on a table without a collision; it passes on main and with this change. Intests/test_schema.py,test_schema_case_insensitive_lookup_of_case_variant_names_raisesandtest_schema_case_insensitive_lookup_of_nested_case_variant_names_raisesfail on main withDID NOT RAISE, andtest_schema_case_insensitive_lookup_of_list_element_struct_fieldcovers the same-ID short names and passes on main and with this change.test_schema_case_insensitive_lookup_rejects_collision_for_any_name,test_metadata_column_projection_propagates_case_collisionandtest_identifier_field_lookup_propagates_case_collisioncover unambiguous and missing names, the metadata column projection and the identifier field check; they fail on main.Are there any user-facing changes?
Yes: case-insensitive name lookups through
Schema.find_fieldandSchema.selectnow raiseValueErrorwhen the schema has names that differ only by case, instead of silently using the last column. Case-sensitive lookups are unchanged.This pull request and its description were written by Isaac.