fix: Widen Athena integer type mapping for unsigned ints - #6739
Draft
nikolauspschuetz wants to merge 1 commit into
Draft
fix: Widen Athena integer type mapping for unsigned ints#6739nikolauspschuetz wants to merge 1 commit into
nikolauspschuetz wants to merge 1 commit into
Conversation
pa_to_athena_value_type mapped every unsigned Arrow int (uint8/16/32/64) to Athena tinyint, a signed 8-bit type (-128..127). Any unsigned value above 127 (e.g. a uint32 column) overflows the type in the generated CREATE TABLE DDL built by aws_utils.py. Widen each unsigned type to the next-larger signed Athena type, matching the widening already used for Postgres in arrow_to_pg_type in this module (uint8->smallint, uint16->int, uint32/uint64->bigint). Signed inputs are unchanged. Adds a regression test for both the widened and signed cases. Signed-off-by: Nikolaus Schuetz <nikolauspschuetz@gmail.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6739 +/- ##
=======================================
Coverage 46.80% 46.81%
=======================================
Files 415 415
Lines 50395 50395
Branches 7214 7214
=======================================
+ Hits 23588 23591 +3
+ Misses 25155 25145 -10
- Partials 1652 1659 +7
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
pa_to_athena_value_typeinsdk/python/feast/type_map.pymaps everyunsigned Arrow int (
uint8/uint16/uint32/uint64) to Athenatinyint.Athena
tinyintis a signed 8-bit type (-128..127), so any unsigned valueabove 127 — e.g. a
uint32column — overflows the type in the generated DDL.This feeds real Athena
CREATE TABLEDDL:aws_utils.pyemits each column as`{name}` {pa_to_athena_value_type(field.type)}when uploading a pyarrowtable.
Fix
Widen each unsigned type to the next-larger signed Athena type that holds its
full range (Athena has no unsigned integer types):
uint8tinyintsmallintuint16tinyintintuint32tinyintbigintuint64tinyintbigintSame widening already used for Postgres by
arrow_to_pg_typein this module —the right sibling since both targets use signed integers. (
pa_to_mssql_typekeeps
uint8->tinyintonly because SQL Server'stinyintis unsigned.)Signed inputs are unchanged.
Test
Regression test in
sdk/python/tests/unit/test_type_map.py. Before the fix itfails (
assert 'tinyint' == 'smallint'); after, both new cases pass (2 passed).drafted with AI assistance; verified locally (red -> green on the added test).