[jonathan343]

Note

Stacked on #868, targeting feat/smithy-checksums-crc64nvme so this PR contains only the XXHash algorithms.

Summary

S3 also supports XXHash64, XXHash3 (64-bit), and XXHash128 checksums. This PR adds them to smithy-checksums as XXHash64, XXHash3, and XXHash128, also available through new(). The native side uses the twox-hash crate (MIT, added to LICENSE-THIRD-PARTY). The pure-Python side is a port of the xxHash reference implementation, limited to the seed of zero and default secret that checksums use. It runs at about 35 MB/s, faster than the pure-Python CRC32C.

Example

import base64

import smithy_checksums

for name in ("XXHASH64", "XXHASH3", "XXHASH128"):
    checksum = smithy_checksums.new(name)
    checksum.update(b"Hello world")
    print(checksum.name, checksum.digest_size, base64.b64encode(checksum.digest()))

The native and pure-Python builds print the same thing. These are also the x-amz-checksum-xxhash* header values botocore's tests expect for the same input:

xxhash64 8 b'xQCwyRKzdtg='
xxhash3 8 b'tqy52Eo4/3Q='
xxhash128 16 b'c1H4mBL5c4K5HQWzHgTdfw=='

Testing

The known answers include 15 of xxHash's own sanity-test vectors per algorithm, from tests/sanity_test_vectors.h. They cover every XXH3 size class and block boundary. Both implementations are also compared against the CRT and against each other:

uv run pytest packages/smithy-checksums -k "xxhash and (known_answers or matches_crt or native_matches_pure)" --no-cov -q

18 passed, 183 deselected in 0.42s

I also ran all 12,483 seed-zero vectors in that file once, covering every input length from 0 to 4,160 bytes. Both implementations matched every one.

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[kuhe]

please also review aws/aws-sdk-js-v3#8383 while this is fresh 😄