[platinumhamburg]

Purpose

Linked issue: close #4813

Implement global secondary indexes for primary-key tables as described in FIP-38, allowing Java clients and Flink lookup joins to find current rows by non-primary-key equality predicates.

Brief change log

• Add schema index definitions and Flink table options for single-column/composite indexes, independent index bucket counts, and sync/async visibility.
• Create system-managed index tables and maintain DATA/PROGRESS records from committed main-table WAL through the private PutIndex RPC.
• Persist index replay progress with KV snapshots, retain required WAL, and recover source/target state with idempotent retries and fail-stop behavior on unrecoverable gaps.
• Add candidate lookup followed by primary-key lookup and indexed-value validation; integrate Flink synchronous and asynchronous lookup joins.
• Handle index table lifecycle, authorization, index-safe schema evolution, and partition deletion through tombstones, read filtering, and compaction cleanup.
• Add replication budgets and aggregate metrics.

Tests

Passed local checks:

• Clean build of all 48 selected Maven modules, including test compilation, Checkstyle, and Spotless: mvn clean install -DskipTests -pl '!fluss-lake/fluss-lake-lance,!fluss-dist'.

• 47 common metadata/schema/encoding tests and 5 RPC API-key tests.

• 205 server tests covering index replication, ordering, source/target failover, remote WAL recovery, WAL gaps, snapshot serialization, KV lookups, metadata, and validation.

• 92 client tests covering secondary-index table lifecycle, partition deletion/recreation, lookup, authorization, and writer accumulation.

• 11 Flink conversion tests and 15 local secondary-index lookup-join cases across Flink 1.18, 1.19, 1.20, 2.2, and 2.3, including synchronous/asynchronous lookup.

The full unit-test run encountered five DiskWriteLockedException errors in ReplicaFetcherThreadTest: this host reports 95.55% disk usage against the fixture's default 85% write limit. A diagnostic rerun with the fixture's write-limit ratio set to 1.0 passed all 8 tests. The submitted fixture retains its original configuration.

The feature and E2E results listed above were obtained locally. No remote E2E pipeline was triggered.

API and Format

• Add index metadata to Schema, secondary-index lookup to the Java table API, and secondary-index.<name>.* Flink table options.
• Add private PutIndex v0 (API key 1075), optional index replay offsets in KV snapshot metadata, and partition tombstone metadata.
• Index tables use ordinary KV batches, WAL, and snapshots. Ordinary table write protocols retain their existing semantics.
• All participating TabletServers must support the index protocol before indexed tables are created. Preview binaries using the earlier PutIndex API key 1068 are not wire-compatible with this implementation.
• Online index creation/backfill, online removal, and range queries are outside this change.

Documentation

Add a secondary-index user guide covering table creation, Java lookups, Flink lookup joins, visibility guarantees, lifecycle, recovery, and limitations. Link it from the primary-key table and Java API guides, add a Flink join example, and document all index replication options and the client continuation thread setting. The design follows FIP-38.