Repository navigation
Conversation
Timestamp literals were rendered as CAST(<n> AS TIMESTAMP). IoTDB 2.0.11 fails a predicate on the time column written that way with error 301 (GenericLiteral cannot be cast to LongLiteral), so every scan with a pushed-down time filter and every lookup keyed on time failed. Render the plain number instead, which IoTDB accepts for the time column and for TIMESTAMP fields. The sink converted TimestampData with toTimestamp() and the source with fromTimestamp(), which both go through LocalDateTime in the JVM default time zone. On a non-UTC JVM the connector wrote times shifted by the zone offset and read them back shifted, while the filter and lookup literals use the epoch value, so they no longer matched the stored data. Convert through Instant on both sides. Signed-off-by: Zihan Dai <1436286758@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two timestamp problems I hit while running this branch (f453277) against IoTDB 2.0.11.
1. Time filters and time-keyed lookups fail. Timestamp literals are rendered as
CAST(<n> AS TIMESTAMP). IoTDB 2.0.11 fails a predicate on the time column written that way:The same happens with
=,INandBETWEENontime. The plain number (time >= 1704067201000) works, and so does comparing a TIMESTAMP field with a number, sorenderTimestampLiteralnow returns the number. Through Flink,SELECT * FROM t WHERE time >= TIMESTAMP '2024-01-01 00:00:01'and a lookup join keyed ontimeboth failed the job with that 301 error before the change.2. Times depend on the JVM time zone. The sink converted
TimestampDatawithtoTimestamp()and the source withTimestampData.fromTimestamp(). Both go throughLocalDateTimein the JVM default zone, while the literals above use the epoch value.RowDataSinkDataConverterTest.testGettersalready fails on a non-UTC machine (expected:<1000> but was:<-28799000>under Asia/Shanghai). Both sides now convert throughInstant.Writing
2024-01-01 00:00:00/01/02through the connector and reading back:time >= '…00:00:01'rows (after)Flink's own result for the same filter over a VALUES table is the same 2 rows. In a lookup join where
timeis only a join condition (lookup=[device_id=…]), the Asia/Shanghai run before the change returned NULL for a row that exists, because the looked-uptimecame back shifted; after the change it returns the value, in both zones and withiotdb.lookup.asyncon and off.Tests:
RowDataSinkDataConverterTestand the newRowDataDeserializationSchemaTestset the default zone to Asia/Shanghai and fail without the respective change;IoTDBLookupReaderTestnow expects the plain number. All 15 tests in the base module pass on a machine in Australia/Sydney.Note:
spotless:checkon this branch already reportsIoTDBUtils,TimestampPrecisionandIoTDBOptions; this change adds nothing to that list, and I ran the builds with-Dspotless.check.skip=true.