Repository navigation
Conversation
A multi-value IN list on an INT32 or INT64 field converted its literals without the range and decimal handling that "=" already has. Out-of-range integers were narrowed with an unchecked int cast and matched unrelated rows, and decimal literals were cast to LongLiteral, which failed with a ClassCastException. Both the pushed-down filter and the projection evaluator were affected, and "x = a OR x = b" hit the same code after it is normalized into IN. Convert each literal through a shared helper that keeps integers inside the column range and integral decimals, and drops literals that can never match. An impossible list becomes an empty IN set, so NOT IN still excludes NULL rows. Closes apache#19276
JackieTien97
left a comment
There was a problem hiding this comment.
I ran the focused unit tests and the standalone table integration suite on this revision. All 27 tests passed. Additional SQL checks exposed two wrong-result cases that need to be addressed before merge.
| // Like "=", drop literals that can never equal an INT32/INT64 value instead of narrowing them | ||
| Long value = getIntegralInListValue(literal, dataType); | ||
| if (value == null) { | ||
| continue; |
There was a problem hiding this comment.
When every literal is dropped here, constructInFilter passes an empty set to TsFile's ValueIn. Its allSatisfy treats a page whose target column is entirely NULL as a match for an empty candidate set, so the table page reader can skip row-level filtering.
I reproduced this with an INT32 field i that is NULL in two rows with a populated payload field. WHERE i IN (4294967298, 2.5) returned zero rows before FLUSH and both NULL rows afterward. The same issue occurs for INT64.
Please handle the empty set explicitly and add a flushed-data regression case. Keep the NOT IN behavior that excludes NULLs.
There was a problem hiding this comment.
Confirmed. constructInFilter now returns FalseLiteralFilter when every literal is dropped, which is what = already does for a literal that can never match, so an empty candidate set no longer reaches ValueIn. NOT IN is untouched: PredicateCombineIntoTableScanChecker rejects a NotExpression, so it stays in the column transformer and still drops NULL rows.
Added testImpossibleInListAndFlushedAllNullColumn in IoTDBPredicateConversionTableIT, on a second table whose null_dev device never writes the field and with a FLUSH in setup; it covers INT32 and INT64 and fails on the extra NULL rows without the main-code change. TablePredicateConversionTest asserts the filter directly. Both suites pass (10 unit, 7 IT).
| String.format(CalcMessages.CANNOT_CAST_TO_TYPE, value, type.getTypeEnum())); | ||
| } | ||
| // (double) max + 1 is exact for INT32 and rounds to 2^63 for INT64, so the bound stays exact. | ||
| return v == Math.floor(v) && v >= min && v < (double) max + 1 ? (long) v : null; |
There was a problem hiding this comment.
The optimizer can deduplicate an IN list and rewrite it to =, bypassing this bound check. With an INT64 field l set to 9223372036854775807, I verified that l IN (9223372036854775808.0, 9223372036854775808.0) matches in WHERE and evaluates to true in a projection. The list (9223372036854775808.0, 0) correctly misses the same row.
In the pushed-down equality path, converting Long.MAX_VALUE to double rounds it to 2^63, and converting the out-of-range literal back to long saturates to Long.MAX_VALUE. The projected equality also rounds the field to 2^63 when comparing as doubles. Please make the INT64 boundary behavior consistent after deduplication and cover both WHERE and SELECT in the regression tests.
There was a problem hiding this comment.
Confirmed. The cause is the > in constructCompareFilter: (double) Long.MAX_VALUE rounds up to 2^63, so the literal passed the range check and the cast back to long saturated onto Long.MAX_VALUE. It now compares against 0x1p63, which is that bound exactly, so WHERE agrees before and after deduplication. getIntegralInListValue uses the same exclusive bound.
The projection still reports true for l = 9223372036854775808.0, but that is not specific to IN. CompareBinaryColumnTransformer evaluates every numeric comparison through getDouble, so on master SELECT l = 9007199254740993 is already true for l = 9007199254740992. Making projected comparisons exact affects all numeric types and operators, so I left it out of this fix and can file it separately.
Covered by testInt64ColumnAndUpperBoundDoubleLiteral and testInt64UpperBoundInList, which exercise WHERE, the deduplicated single-value form, plain =, and the IN projection. Without the change the unit test reports measurements[0] = 9223372036854775807.
An IN list on an INT32/INT64 field whose literals were all dropped produced an empty TsFile ValueIn. Its statistics shortcut reports allSatisfy() for an empty candidate set on a page whose column is entirely NULL, so once the data was flushed the pushed-down filter returned those NULL rows. Return FalseLiteralFilter instead, which is what "=" already does for a literal that can never match. NOT IN is unaffected because a NotExpression is never pushed into the scan and keeps excluding NULL rows in the column transformer. The pushed-down "=" also accepted a double literal of exactly 2^63 for an INT64 column, because (double) Long.MAX_VALUE rounds up to 2^63 and the cast back to long saturated onto Long.MAX_VALUE. The optimizer deduplicates an IN list and rewrites a single remaining value to "=", so the same literal matched in "IN (2^63, 2^63)" while "IN (2^63, 0)" correctly missed. Compare against 0x1p63 so the bound is exact.
Description
Closes #19276
Bug
A multi-value
INlist on an INT32 or INT64 field converted its literals without the range and decimal handling that=already has. The bug only shows up when the list has two or more values; a single value is rewritten to=.On a field
i INT32holding 1, 2, 3, INT32_MIN, INT32_MAX and NULL:where i in (1, 4294967298)where i in (1, -4294967293)where i in (1, 2147483648)where i = 1 or i = 4294967298(normalized to IN)where i not in (1, 4294967298)where i in (2, 2.5),where l in (2, 2.5)(INT64)301: DoubleLiteral cannot be cast to LongLiteralselect i in (2, 2.5),select l in (2, 2.5)ClassCastExceptiontruefor 2,nullfor NULLwhere i not in (2.5, 4294967298)ClassCastExceptionThe "Before" column comes from a standalone server built from master. The "After" column comes from a standalone server built from this branch, running the same statements as #19276.
Cause
There were two paths with the same problem:
ConvertPredicateToFilterVisitor.constructInSetconverted every literal throughRELATIONAL_CONVERT_PREDICATE_VALUE_PARSER_SERVICE. That is(int) ((LongLiteral) v).getParsedValue()for INT32. Out-of-range values wrapped, and anyDoubleLiteralfailed the cast.calc-commonsTypeServices.IN_MULTI_COLUMN_TRANSFORMER_SERVICE, already skipped out-of-range INT32 values. It still cast every literal toLongLiteral, though, so decimals failed for both INT32 and INT64.The analyzer does not insert casts for IN lists, so these literals reach both paths unchanged.
Fix
TypeServices.getIntegralInListValue(Literal, Type)in calc-commons converts each literal for an INT32/INT64 column. It keeps the value when it is an integer within the column range, or an integral decimal within that range. It returnsnullwhen the literal can never match. This is the same ruleconstructCompareFilteruses for=.constructInSetuses it for INT32/INT64 columns and drops literals that can never match. Other column types keep the existing conversion.IN_MULTI_COLUMN_TRANSFORMER_SERVICEuse the same helper. The INT32try/catch IllegalArgumentExceptionaroundgetParsedValue()is removed; that method cannot throw, so the block was dead code.constructInFilterreturnsFalseLiteralFilter, the same as=for a literal that can never match. An empty IN set cannot be used here: TsFile'sValueIn.allSatisfyreports a match for an empty candidate set on a page whose column is entirely NULL, so a flushed all-NULL chunk would be returned.NOT INis unaffected becausePredicateCombineIntoTableScanCheckerrejects aNotExpression, so it is evaluated by the column transformer and still excludes NULL rows.(double) Long.MAX_VALUErounds up to 2^63, so both the IN helper and the pushed-down=compare against the exact exclusive bound0x1p63. Previously=usedfloatPointValue > Long.MAX_VALUE, which let9223372036854775808.0through and saturated it ontoLong.MAX_VALUE; because the optimizer deduplicates an IN list and rewrites a single remaining value to=,IN (2^63, 2^63)matched whileIN (2^63, 0)did not.SELECT l = 9223372036854775808.0still evaluates totrue.CompareBinaryColumnTransformerruns every numeric comparison throughgetDouble, so this is pre-existing and not specific to IN (on masterSELECT l = 9007199254740993is alsotrueforl = 9007199254740992). Making projected comparisons exact would affect all numeric types and operators and is left for a separate change.CANNOT_CAST_TO_TYPEmessage instead of aClassCastException.Tests
TablePredicateConversionTest(new tests:testMultiValueInPredicateAndInt32Column,testMultiValueInPredicateAndInt64Column,testInt64ColumnAndUpperBoundDoubleLiteral):2147483648at the boundary);TypeServicesTest.testIntegralInListValue: the shared conversion used by the projection path.IoTDBPredicateConversionTableIT(new tests:testMultiValueInPredicateAndInt32Column,testMultiValueInPredicateAndInt64Column,testImpossibleInListAndFlushedAllNullColumn,testInt64UpperBoundInList). The last two use a second table whosenull_devdevice never writes the fields, with aFLUSHin setup, so the all-NULL chunk statistics are exercised:Local runs (JDK 25, macOS arm64):
Without the main-code change, the two multi-value
TablePredicateConversionTestcases fail withClassCastException: ... DoubleLiteral cannot be cast to ... LongLiteral, the impossible-list assertion reportsmeasurements[0] IN [],testInt64ColumnAndUpperBoundDoubleLiteralreportsmeasurements[0] = 9223372036854775807, and both new IT cases fail on the rows that should not have been returned.Not run locally:
TableClusterIT(I relied on the CI run for the cluster environment) and the full unit-test suites of the two modules.This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
org.apache.iotdb.calc.utils.TypeServicesorg.apache.iotdb.db.queryengine.plan.relational.analyzer.predicate.ConvertPredicateToFilterVisitor