Repository navigation
Conversation
Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
| elif IntegerType.min > number: | ||
| return IntBelowMin() | ||
| return LongLiteral(int(number)) | ||
| return LongLiteral(_to_integral(number, type_var)) |
There was a problem hiding this comment.
The range check runs before the fractional check. So "2147483647.5" to IntegerType returns IntAboveMax, but DecimalLiteral(Decimal("2147483647.5")).to(IntegerType()) raises. Intentional?
There was a problem hiding this comment.
Good catch, it was not intentional. I moved the fractional check before the range check in both StringLiteral.to(IntegerType) and .to(LongType), mirroring DecimalLiteral, so "2147483647.5" now raises ValueError instead of returning IntAboveMax. Added test_fractional_string_near_integral_type_bounds_raises covering both bounds for both types. Fixed in 472b861.
…iteral.to Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
Fokko
left a comment
There was a problem hiding this comment.
I know that we're changing behavior here, but since it might lead to unexpected behavior, I think we should do it.
Left one comment around testing, apart from that it looks good to me
| pi_string_lit = StringLiteral("3.141") | ||
| pi_double_lit = literal(3.141).to(DoubleType()) | ||
|
|
||
| assert literal("3.141").to(IntegerType()) == literal(3) |
There was a problem hiding this comment.
I think we still want to test the happy flow
There was a problem hiding this comment.
Thanks, done. I restored the happy-flow asserts in test_string_literal: literal("3").to(IntegerType()) and literal("3").to(LongType()) both equal literal(3). The fractional cases still raise as expected.
…eral Signed-off-by: Shahzeb Ali <126076773+Shahzeb-DataEngineer@users.noreply.github.com>
Closes #4382
Rationale for this change
StringLiteral.to(IntegerType)andStringLiteral.to(LongType)parsed withDecimalthen truncated with plainint(), so "2.6" became 2 and a bound predicate such asx < "2.6"on an int column silently becamex < 2, wrongly dropping rows with x = 2 and feeding the same wrong literal into partition pruning. The siblingDecimalLiteralpath already rejects fractional values via_to_integral, and the Java reference implementation refuses the conversion entirely. This change reuses_to_integralin both StringLiteral paths.Are these changes tested?
Yes. New test
test_fractional_string_to_integral_type_raisesmirrorstest_fractional_decimal_to_integral_type_raises; it fails on unpatched code and passes with the fix.test_string_literalupdated to no longer assert the truncating behavior.tests/expressions/test_literals.py: 180 passed.tests/expressions/: 594 passed, 2 failed (pre-existing pyparsing-version failures on clean main, verified viagit stash).ruff checkandruff format --checkclean on both changed files.Are there any user-facing changes?
Yes, behavior change for the better: converting a fractional string such as "2.6" to an int/long column now raises ValueError at bind time instead of silently truncating. Integral strings ("2.0", "1e3") still convert exactly.