Route keys with XXH3, matching the server and other clients

[merlimat]

Problem

ServiceDiscovery.get_shard hashed keys with xxhash.xxh64, which is the classic XXH64 algorithm. The server and every other client use XXH3-64 truncated to 32 bits:

Implementation	Hash
Go (server and client)	common/hash/hash.go: uint32(xxh3.HashString(key))
Java	LongHashFunction.xx3()
Node	xxh3.xxh64 from @node-rs/xxhash
Rust	xxhash_rust::xxh3::xxh3_64

The shard assignments even announce ShardKeyRouter.XXHASH3, and this client checks for it. Results on the Go test vectors (common/hash/hash_test.go):

key	Go (XXH3)	Python before this PR (XXH64)
foo	125730186	1506064959
bar	2687685474	2905056857
baz	862947621	1780773892

get_shard is the only place keys are hashed, and every routed operation goes through it:

• put, delete, and get by key
• anything with a partition_key
• sequential keys and get_sequence_updates

With N evenly sized shards, a key lands on the correct shard only about 1/N of the time. So in a multi-shard namespace, most records written by the Python client are on a different shard than the one the other clients and the server's split logic expect. Every release so far has this bug (0.1.0 through 0.2.1).

Why it went unnoticed

• The server doesn't validate routing. It never checks that a key's hash falls in the target shard's range. common/hash.Xxh332 is only used by the Go client's routing (oxia/internal/shard_strategy_impl.go) and by the shard-split filter (oxiad/dataserver/database/split_filter.go), so a misrouted write just succeeds. Against oxia/oxia:latest (0.16.10) with 10 shards, the unfixed client's writes to the wrong shard returned OK.
• The tests only round-trip. The existing multi-shard tests (e.g. test_floor_ceiling_get on 10 shards) only read back what the Python client wrote itself, which works with any hash.

Fix

Hash with xxhash.xxh3_64_intdigest(key) masked to 32 bits, in a small xxh332() helper named after the Go function. Nothing else changes, and the shard-assignment thread is untouched.

Tests

• tests/service_discovery_test.py::test_key_hash_matches_go pins the hash to the Go vectors.
• tests/client_test.py::test_shard_routing_matches_go_client runs on a 10-shard standalone container:
  • records written by the Python client are read with the Go CLI in the container (oxia client get)
  • records written with the Go CLI are read by the Python client
  • it covers plain, hierarchical, and non-ASCII keys, and routing by partition key

Without the fix (the helper still computing XXH64), both tests fail:

E       AssertionError: assert 1506064959 == 125730186
E       AssertionError: assert 2905056857 == 2687685474
E       AssertionError: assert 1780773892 == 862947621
E   AssertionError: 0 != 1 : oxia client get foo: b'Error: key not found\n'
4 failed, 1 passed

Each section of the integration test catches the bug on its own: 10 of 11 cross-client reads miss. The one that passes, bar, happens to land on the same shard under both hashes. With the fix, the full suite passes (68 tests).

⚠️ Compatibility: release note

Routing change for multi-shard namespaces. Versions 0.1.0 through 0.2.1 of the Python client used the wrong hash function to route keys (XXH64 instead of XXH3). In namespaces with more than one shard, most records they wrote are stored on a different shard than the one the server and the other Oxia clients use. This release fixes the routing. As a result, records written by earlier versions to a multi-shard namespace can appear to be missing:

• Single-shard operations miss them. get(key), delete(key), and every operation with a partition_key now look on the correct shard, so they don't find misplaced records: get raises KeyNotFound and delete returns False. This includes sequential keys and get_sequence_updates.
• Operations across all shards still see them. This covers list, range_scan, and delete_range without a partition_key, and get with a non-EQUAL comparison or a secondary index. So a key can show up in list() while get() on it raises KeyNotFound.
• Writing the key again creates a second copy on the correct shard, and list()/range_scan() then return the key twice. put(..., expected_version_id=EXPECTED_RECORD_DOES_NOT_EXIST) succeeds even though the key already exists on the old shard.
• Sequential keys restart from 1 on the correct shard unless the existing records are moved there first.
• Shard splits delete misplaced records. This applies to servers that support splits (first shipped in server v0.16.2). A split keeps each record only in the child whose range contains the record's XXH3 hash. The two children cover the parent's range, and the parent is deleted afterwards. So a record stored on a shard whose range doesn't contain its XXH3 hash survives in neither child. This comes from reading the server code (split_filter.go, shard_split_controller.go); it hasn't been reproduced on a cluster. Move misplaced records before splitting any shard that holds data written by the Python client, and consider disabling auto-split until then.
• Don't mix old and new Python clients against the same multi-shard namespace. They route keys differently.

Namespaces with a single shard are not affected. Records written by the Go, Java, Node, and Rust clients were always placed correctly, and they now become visible to the Python client.

Proposed migration path (not part of this PR)

A one-off script, built on the fixed client, that scans every shard and moves misplaced records:

1. Scan. For each shard S, range_scan it directly (the per-shard stubs from service discovery) and flag every record whose routing key hashes (XXH3) outside S's range.
2. Report. Dry run is the default: it prints per-shard counts and sample keys, which also tells users whether they're affected at all.
3. Move. With --apply, for each flagged record:
   • put it on the correct shard with EXPECTED_RECORD_DOES_NOT_EXIST
   • delete it from S, conditional on the version that was read
   • if the target already holds the key (for example, it was rewritten after the upgrade), report a conflict and leave both copies
   • re-running the script is safe
4. Stop Python writers during the migration. Run it after they stop and before they restart on the fixed client, so that sequences and CAS versions don't diverge.

Known limits, because the client read API doesn't return everything the server stores:

• Partition keys. Records written with a partition_key are routed by it, but reads don't return it (the server keeps it in StorageEntry.partition_key). The script needs a routing-key callback from the application (key -> partition_key). It defaults to the record key.
• Secondary indexes aren't returned by reads either, so a moved record loses them unless the application provides them.
• Ephemeral records are skipped. They belong to the old clients' sessions and go away when those sessions end.
• Versions reset. A moved record gets a new version_id, modification count, and timestamps, so a cached version fails CAS once.
• Notifications. Each move emits a put notification and a delete notification.

If partition keys or secondary indexes are widely used, a server-side relocation tool would be the full-fidelity option, since the server stores both. That would be a server feature, though.