Repository navigation
Conversation
|
The CI workflows for this PR are waiting for maintainer approval since it comes from a fork. Could someone approve them when convenient? Locally the new DateTimeUtilsTest cases and the related parser/analyzer tests pass. |
fe161ea to
2ef8fcc
Compare
|
Rebased onto current master with no code changes. On the previous run, the only red checks were the TODO check, which picked up a TODO line in PageIOChannel that only moved on master, and the 1C1D1A AINode job, whose runner was shut down mid-run. Could someone approve the workflows again when convenient? |
|
All checks pass on the rebased commit except one Unit-Test job: CompactionTaskManagerTest.testRepeatedSubmitBeforeExecution failed on ubuntu with expected:<0> but was:<1> at line 93, which looks like the compaction task getting picked up before the count is checked. It passed on Windows and on the previous run, and this PR doesn't touch compaction. Could someone re-run that job when convenient? |
A date expression such as `2024-03-15 - 1mo` computed the length of the month after the given time and subtracted it, giving 2024-02-13 instead of 2024-02-15 (and `2024-03-31 - 1mo` gave 2024-03-01). Apply the subtracted duration backwards from the original time, so that `- Nmo` goes back N calendar months the same way `+ Nmo` goes forward. This affects both the tree and table date-expression parsers. Addition, other units, and the context-free duration path are unchanged. Fixes apache#19210
2ef8fcc to
56f8c83
Compare
|
Rebased onto current master again to get a fresh run, since I can't re-run the failed job from a fork. The code is unchanged; the only failure last time was the CompactionTaskManagerTest timing issue mentioned above. The workflows need approval again when someone has a moment. |
JackieTien97
left a comment
There was a problem hiding this comment.
Please fix the two date-expression regressions below and add regression tests for them. I reproduced both against the PR head and its base.
| convertDurationStrToLong( | ||
| currentTime == -1 ? -1 : Math.addExact(currentTime, total), | ||
| temp, | ||
| negative ? -temp : temp, |
There was a problem hiding this comment.
[P2] Preserve calendar-month subtraction at microsecond and nanosecond precision. In UTC with timestamp_precision=us or ns, 2024-01-16T00:00:00+00:00 - 1mo evaluates to December 16 on the base revision and December 17 on this PR in both SQL models. Calendar.setTimeInMillis interprets the scaled timestamp as milliseconds, so changing the direction changes which unrelated month supplies the duration. Please convert the anchor to milliseconds for calendar arithmetic, preserve its sub-millisecond component, and cover both precisions in the parser tests.
There was a problem hiding this comment.
Confirmed, fixed in 6ed9afc. Calendar.setTimeInMillis was being handed a timestamp in timestampPrecision units, so at us and ns it anchored on an unrelated date; it now anchors at the millisecond the timestamp falls in, and since the offset is a whole number of milliseconds the sub-millisecond part of the anchor survives untouched.
dateExpressionNaturalMonthSubMillisecondPrecisionTest covers both precisions in both SQL models, including + 1mo, month-end clamping and a pre-1970 anchor. 2024-01-16T00:00:00+00:00 - 1mo gives December 16 now; against the previous head the same case fails with 1702771200000000, which is December 17.
One case there is ns only. At us precision getInstantWithPrecision adds 999999 in its epochSecond < 0 && nano > 0 branch, so 1969-12-31T23:59:59.999+00:00 parses as 998999 rather than -1000. That is independent of this change, so I left it alone here.
| @@ -188,7 +204,7 @@ public static long convertDurationStrToLong( | |||
| total, | |||
| convertDurationStrToLong( | |||
| currentTime == -1 ? -1 : Math.addExact(currentTime, total), | |||
There was a problem hiding this comment.
[P3] Treat -1 as a valid intermediate timestamp. In the tree model at millisecond precision, 1970-01-01T00:00:00+00:00 - 1ms1mo first subtracts 1ms, yielding -1. The mo conversion interprets that value as a missing anchor and uses 30 days. The PR returns 1969-12-01T23:59:59.999Z, while the base revision returns the expected 1969-11-30T23:59:59.999Z. Please represent the presence of an anchor separately from its value so this case uses calendar arithmetic.
There was a problem hiding this comment.
Confirmed, fixed in 6ed9afc. The presence of the anchor is now tracked separately from its value, and the date-expression parsers go through convertDateExpressionOffset, where the anchor is always real, so 1970-01-01T00:00:00+00:00 - 1ms1mo uses calendar arithmetic and returns 1969-11-30T23:59:59.999Z.
The -1 convention of the other overloads is unchanged, so GROUP BY, CQ, IoTDBDescriptor and constructTimeDuration keep their current behaviour. dateExpressionSubtractNaturalMonthTest now covers - 1ms - 1mo, - 1ms1mo and - 1ms + 1mo; the first fails against the previous head with -2592000001, the flat 30-day fallback.
Two problems in the date-expression duration path, both reachable from the tree and the table parser. Calendar.setTimeInMillis was fed a timestamp in timestampPrecision units, so with timestamp_precision=us or ns the calendar landed on an unrelated date and the month step measured that date's month instead. Anchor the calendar at the millisecond the timestamp falls in. The offset is a whole number of milliseconds, so the sub-millisecond part of the timestamp is carried over to the result untouched. A currentTime of -1 meant both "no anchor" and the timestamp 1969-12-31T23:59:59.999, so an intermediate value of -1 (for example 1970-01-01 - 1ms1mo) fell back to a flat 30 days. Represent the presence of an anchor separately from its value, and give the date expression parsers an entry point where the anchor is always present. The -1 convention of the other overloads is unchanged, so GROUP BY, CQ and constructTimeDuration behave exactly as before.
|
Both review comments are addressed in 6ed9afc, with the details in the two threads. The calendar anchor is now converted to milliseconds before the month step and the sub-millisecond part of the timestamp is preserved, and the presence of an anchor is tracked separately from its value so an intermediate Locally on JDK 25 with The workflows are sitting at |
Fixes #19210
Summary
Subtracting natural months in a date expression lands on the wrong day.
<time> - Nmocomputed the length of the N months after<time>and subtracted it, so the result depended on how long the following month is:2024-03-15 - 1mogave2024-02-13.2024-03-31 - 1mogave2024-03-01.2024-02-15 + 1mo - 1mogave2024-02-13.With this change,
- Nmogoes back N calendar months from the original time, the same way+ Nmoalready goes forward: it keeps the time of day in the session time zone and clamps to the end of the month.Root cause
All three date-expression parsers do the same thing:
ASTVisitor.parseDateExpression(ctx, String precision)andASTVisitor.parseDateExpression(ctx, long currentTime)(tree)AstBuilder.parseDateExpression(table)They call
DataNodeDateTimeUtils.convertDurationStrToLong(time, duration, false)and then applyMath.subtractExact(time, duration)for-. Formo, that conversion runscalendar.add(Calendar.MONTH, +N)fromtime. It always measures forward, even when the operator is-.Changes
DataNodeDateTimeUtils: addedconvertDateExpressionOffset(time, duration, [precision,] negative), the entry point the date-expression parsers use. Whennegativeis true, each unit in the duration string is converted with a negated value, so the month step becomesCalendar.MONTH, -Nfrom the original time, and the returned offset is negative.Math.floorDiv(currentTime, ticksPerMillisecond)), becausecurrentTimeis intimestampPrecisionunits whileCalendarworks in milliseconds. The offset it returns is a whole number of milliseconds, so the sub-millisecond part of the timestamp is carried over to the result untouched.currentTime == -1meant both "no anchor" and the timestamp1969-12-31T23:59:59.999, so an intermediate value of -1 fell back to a flat 30 days.convertDateExpressionOffsetalways has a real anchor; the-1convention of the other overloads is unchanged, so GROUP BY, CQ,IoTDBDescriptorandconstructTimeDurationbehave exactly as before."-".equals(op)and then doMath.addExact(time, offset). Addition produces the same values as before atmsprecision, where the new anchor arithmetic is an identity.2024-03-31 - 1mo1d=2024-02-29 - 1d=2024-02-28). Fixed-length units (d,w,h, ...) give the same result as before, and the precision conversion forms/us/nsunits is symmetric for negative values.Before and after, standalone server,
timestamp_precision=ms, server and CLI in UTC. The tree and table models returned the same rows in both builds:time <= 2024-03-15T00:00:00+00:00 - 1motime >= 2024-03-31T00:00:00+00:00 - 1motime >= 2024-02-15T00:00:00+00:00 + 1mo - 1mo AND time <= 2024-02-20T00:00:00+00:00Output after the fix (tree):
Tests
New tests in
DateTimeUtilsTest. Each case is parsed by the real tree parser (DataNodeDateTimeUtils.parseDateTimeExpressionToLong) and the real table parser (SqlParser.createExpression) with a registered session, and the resulting timestamp is checked:dateExpressionSubtractNaturalMonthTest(UTC) covers:-1mo/-2moacross different month lengths2024-03-31,2025-03-31,2024-05-311mo1dand1w2d(tree only, because the table grammar has no compound durations), plus- 1mo - 1d+ 1mo - 1moround trip+ 1mocases- 1ms - 1mo,- 1ms1moand- 1ms + 1mofrom1970-01-01dateExpressionSubtractNaturalMonthWithTimeZoneTest:America/New_Yorkacross the DST start, andAsia/Kolkata.dateExpressionNaturalMonthSubMillisecondPrecisionTestruns the same kind of cases attimestamp_precision=usandns:- 1moand+ 1mo, month-end clamping, the+ 1mo - 1moround trip, a sub-millisecond anchor, and a pre-1970 anchor.The new tests fail on current master (for example,
tree: 2024-03-15T00:00:00+00:00 - 1mo expected:<1707955200000> but was:<1707782400000>) and pass with this change.Local runs (JDK 25, macOS):
The skipped test is the existing
@IgnoredgetConvertDurationIncludingMonthUnit. Note thatconvertDatetimeStrToLongTest1in the same class fails in a+05:30local time zone on master as well, independent of this change, so the runs above useTZ=UTC.Microsecond and nanosecond precision
An earlier revision of this PR left
moarithmetic wrong attimestamp_precision=usandns, where the precision-scaled timestamp was passed toCalendar.setTimeInMillis. That is fixed here as well, in both directions, after review:2024-01-16T00:00:00+00:00 - 1monow gives December 16 at every precision, and+ 1mois corrected along with it.One unrelated problem remains, outside this change.
DateTimeUtils.getInstantWithPrecisionadds 999999 in itsepochSecond < 0 && nano > 0branch atusprecision, so1969-12-31T23:59:59.999+00:00parses as998999rather than-1000. It affects any pre-1970 literal with a sub-second part atusprecision, independent of durations, so the one test case that would hit it isnsonly.This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
DataNodeDateTimeUtilsASTVisitorAstBuilderDateTimeUtilsTest