Skip to content

Fix timestamp filters and time zone handling in the Flink table connector - #142

Open
PDGGK wants to merge 1 commit into
apache:flink-iotdb-table-connectorfrom
PDGGK:fix/flink-table-timestamps
Open

PDGGK wants to merge 1 commit into
apache:flink-iotdb-table-connectorfrom
PDGGK:fix/flink-table-timestamps

Conversation

@PDGGK

@PDGGK PDGGK commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Two timestamp problems I hit while running this branch (f453277) against IoTDB 2.0.11.

1. Time filters and time-keyed lookups fail. Timestamp literals are rendered as CAST(<n> AS TIMESTAMP). IoTDB 2.0.11 fails a predicate on the time column written that way:

SELECT time, temperature FROM sensor WHERE time >= CAST(1704067201000 AS TIMESTAMP)
-> 301: class ...GenericLiteral cannot be cast to class ...LongLiteral

The same happens with =, IN and BETWEEN on time. The plain number (time >= 1704067201000) works, and so does comparing a TIMESTAMP field with a number, so renderTimestampLiteral now returns the number. Through Flink, SELECT * FROM t WHERE time >= TIMESTAMP '2024-01-01 00:00:01' and a lookup join keyed on time both failed the job with that 301 error before the change.

2. Times depend on the JVM time zone. The sink converted TimestampData with toTimestamp() and the source with TimestampData.fromTimestamp(). Both go through LocalDateTime in the JVM default zone, while the literals above use the epoch value. RowDataSinkDataConverterTest.testGetters already fails on a non-UTC machine (expected:<1000> but was:<-28799000> under Asia/Shanghai). Both sides now convert through Instant.

Writing 2024-01-01 00:00:00/01/02 through the connector and reading back:

JVM zone stored epoch millis (before) stored (after) time >= '…00:00:01' rows (after)
UTC 1704067200000… 1704067200000… 2
Asia/Shanghai 1704038400000… (−8 h) 1704067200000… 2

Flink's own result for the same filter over a VALUES table is the same 2 rows. In a lookup join where time is only a join condition (lookup=[device_id=…]), the Asia/Shanghai run before the change returned NULL for a row that exists, because the looked-up time came back shifted; after the change it returns the value, in both zones and with iotdb.lookup.async on and off.

Tests: RowDataSinkDataConverterTest and the new RowDataDeserializationSchemaTest set the default zone to Asia/Shanghai and fail without the respective change; IoTDBLookupReaderTest now expects the plain number. All 15 tests in the base module pass on a machine in Australia/Sydney.

Note: spotless:check on this branch already reports IoTDBUtils, TimestampPrecision and IoTDBOptions; this change adds nothing to that list, and I ran the builds with -Dspotless.check.skip=true.

Timestamp literals were rendered as CAST(<n> AS TIMESTAMP). IoTDB 2.0.11
fails a predicate on the time column written that way with error 301
(GenericLiteral cannot be cast to LongLiteral), so every scan with a
pushed-down time filter and every lookup keyed on time failed. Render the
plain number instead, which IoTDB accepts for the time column and for
TIMESTAMP fields.

The sink converted TimestampData with toTimestamp() and the source with
fromTimestamp(), which both go through LocalDateTime in the JVM default
time zone. On a non-UTC JVM the connector wrote times shifted by the zone
offset and read them back shifted, while the filter and lookup literals
use the epoch value, so they no longer matched the stored data. Convert
through Instant on both sides.

Signed-off-by: Zihan Dai <1436286758@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant