[tae898]

Port of humemai/arcadedb-embedded-python#521 (arcadedb_n_next_bin) together with #520 (bulk copy of the reply text) to this repository, plus a fix of one test that failed under a cgroup memory limit. Native build only; the JVM build is not touched.

Problem

arcadedb_n_exec with flag 4 sends float[] values of 8 or more elements in a binary float32 tail, but only for the first reply (256 rows). arcadedb_n_next, which returns every later batch (2048 rows), has no sink, so an embedding column came back as JSON number text from row 257 on. json.loads with the object hook was about half of the wall time of a wide SELECT (a 32-float vector column, 10k to 100k rows). Separately, the image wrote every reply byte by byte (p.write per byte, fp.write per float).

Change

• New entry point arcadedb_n_next_bin(rs, max): the same batch as arcadedb_n_next, encoded with the float sink. The tail code of ArcadeNative.replyOf becomes one helper, withFloatTail, used by replyOf and by the new entry point. arcadedb_n_next is unchanged, so an older Python on a newer image keeps working. The lazy command path (Prefetched) reads through the same entry point.
• src/arcadedb/_native/lib.py: NEXT_BIN is on when the library has the entry point. ARCADEDB_NATIVE_NEXT_BIN=0 turns it off. I kept the switch as a diagnostic off switch, like the other phase 8 switches in lib.py (it is how the A/B below toggles the path inside one image). No document lists the sibling switches, so none needed an update.
• NativeResultSet._fill reads batches with lib.call_bin("arcadedb_n_next_bin", ...) and loads(text, floats) when NEXT_BIN and the binary reply (ARCADEDB_NATIVE_BIN_REPLY) are on, and with arcadedb_n_next as before otherwise.
• Bulk copies (Native: bulk-copy reply text out of the image (draft, measurement pending) humemai/arcadedb-embedded-python#520): out() and the tail block of withFloatTail copy the reply with CTypeConversion.asByteBuffer(...).put(...) (Unsafe.copyMemory): the text, the zero padding up to the 4-byte boundary, then the float32 tail in native order. Before, they wrote one byte or one float per loop iteration.
• tests/test_native_result_transport.py: round trips of replies of every size (several batches, a small reply with a vector, rows of about a megabyte of multi-byte text, empty and one-row replies), and vectors of 0, 3, 8 and 32 floats, float32 extremes (1e-45, 3.4028235e38, +/-inf, -0.0) and rows with no vector, read in one list and through the row-by-row cursor, compared as the 8 bytes of the double. The tests use the public API only and pass on the number-text path (ARCADEDB_NATIVE_NEXT_BIN=0) as well as on the new one.
• tests/test_native_heap.py (from Native: start the isolate with -XX:MaximumHeapSizePercent=50 by default #74): test_default_max_heap_is_half_of_the_memory compared the default heap with the smaller of the host memory and /sys/fs/cgroup/memory.max (the root of the mount), so it failed whenever the process's own cgroup had a limit (a container, or a systemd scope such as the laptop job gate: the heap is 50% of the limit). _cgroup_memory_limit now follows /proc/self/cgroup to <root>/<path>/memory.max and walks up to the first value that is not max, with the host memory as fallback. Unit tests of the reader use fake files under tmp_path.

Measured upstream (not re-run on this tree)

A pre-registered laptop A/B on one image built from the upstream branch (which contains #520 and #521); the two arms differ only by ARCADEDB_NATIVE_NEXT_BIN. It passed its rule:

query	new time / old time
SELECT id, grp, val, cat, txt, vec FROM D LIMIT 10000 (32-float vector)	0.431 [0.430, 0.433]
SELECT id, grp, val, cat, txt, vec FROM D LIMIT 100000 (32-float vector)	0.456 [0.454, 0.458]

Guards stayed within [0.99, 1.01]: indexed point lookup in SQL and in Cypher, 2-hop, GROUP BY, and a 200-row read with vectors. The A/A control was clean and the decoded-row digests were equal in every run.

The state shipped here is the code of that B arm (#520 plus #521). Not measured: the effect of #520 on its own against today's main. It is a byte-copy equivalence, covered by the round-trip tests above and by the parity run. The A/B itself ran on the upstream tree, not on this one, so the absolute times on this tree are not measured either.

Hunk-by-hunk check

Every hunk of the upstream #521 diff (which includes #520's, as it is stacked on it) and of the #520 diff, compared line by line with this PR (added and removed lines counted per file):

file	upstream hunk	here
database.py	_fill reads through arcadedb_n_next_bin	taken, identical
lib.py	NEXT_BIN switch, arcadedb_n_next_bin signature [P, L, I], global ... NEXT_BIN, hasattr fallback	taken, identical
ArcadeNative.java	nextBin entry point after next	taken, identical
ArcadeNative.java	replyOf calls withFloatTail; helper javadoc and signature	taken; Supplier<String> with an import instead of the fully qualified name, because CLAUDE.md asks for imports
ArcadeNative.java	bulk copies in the body of withFloatTail (#520)	taken; ByteBuffer and ByteOrder imported instead of written out
ArcadeNative.java	bulk copy in out() (#520)	taken, identical
test_native_result_transport.py	all of the upstream file: #520 round-trip tests and #521 vector tests	taken, identical except the module docstring and two comments (DIM, the 5000-row test), which are reworded because the later batches now carry the tail

The comparison found no other upstream line missing, in either diff. No upstream hunk was left out. The only differences are the three import-for-qualified-name lines and the reworded docstring and comments.

Verification

• Native image built from this tree (arcadedb_n_next_bin is exported next to arcadedb_n_next), wheel built from it, and the full native/wheel/parity_wheel.py suite run on that wheel, in the memory-limited job scope (7 GB) the laptop uses for such runs.
• Full parity suite: 1421 passed, 0 failed, 0 error, 0 NOTYET, 0 timeout, 124 NA-JVM. test_native_heap.py (17 tests, 8 of them the new reader tests) passes inside that 7 GB scope.
• The four files that send SIGINT or use it (test_python_function_interrupt, test_sigint, test_interrupt_hygiene, test_transaction_context_interrupt) were run again in a foreground shell, because signal tests see SIGINT ignored in background jobs: 28 passed, 0 failed.
• Probe on the new wheel: a 5000-row read of a 32-float vector column makes 3 arcadedb_n_next_bin calls, each with a float tail, and the decoded rows have the same digest with ARCADEDB_NATIVE_NEXT_BIN=1 and =0.
• tests/test_native_result_transport.py passes with ARCADEDB_NATIVE_NEXT_BIN=1 and =0 (6 tests each). The result-reading tests (test_resultset, test_resultset_arrow, test_results_after_close, test_resultset_read_errors, test_select_native_matrix, test_float_vector_bulk, test_columnar_readers) with the new files: 183 passed, 23 skipped (all NA-JVM tests that need the Java bridge).
• Bandit 1.9.4 (--severity-level low --confidence-level low) clean, isort --profile black and black clean on the touched Python files, scripts/check_no_embedded_name.py passes.

🤖 Generated with Claude Code