Repository navigation
K2GO-507 feat(cx-core): export as a prepare/poll/fetch long operation - #691
Merged
Merged
Conversation
…fetch Replace open_export (which staged, preflighted and streamed in one call) with three primitives: prepare_export reports an ExportPlan (stages?, total bytes, name), stage_export stages into a caller-given dir, open_fetch opens the stream. The temp and staging-thread lifecycle moves to the op layer; the adapter keeps only the per-app staging / streaming logic. Read-only-on-source and traversal guards preserved.
…version A large Kolibri channel stages for minutes before the first byte, so a single blocking export request times out the client. Export becomes a long operation, uniform for all apps: prepare starts staging in a background thread (or marks a direct app ready) and returns an opId; status reports a staged-bytes heartbeat and flips to ready; fetch streams the bytes; delete aborts. The old GET /v1/export/item is removed (no alias: unused). export_ops.ExportOps owns the in-flight set and the one cleanup owner per temp: fetch on-close (normal), delete (abort), a lazy TTL sweep (abandoned), and a startup sweep of the op temps on disk (process death). State is in-memory only; the front recovers by re-prepare. Also add cx_core.VERSION, GET /v1/version (and version in GET /v1/apps), and have the cx_core role write the version into the [cx_core] section of iiab.ini, mirroring dash-node.
…ir is cleaned The 409-before-ready test released the staging thread then let the TemporaryDirectory be removed under it, a teardown race on Windows. Wait for the terminal state first.
…eep legacy temps Review fixes on the export long operation: - The TTL sweep could rmtree a live fetch's temp: a multi-gigabyte stream outlasts the TTL and bumps nothing, so a concurrent op request swept it mid-stream. Mark the op in_use when fetch opens the stream, skip in_use ops in the sweep, drop it in on_close. A process death loses the flag and the op, so the startup sweep still reclaims the orphan temp. - An abort (DELETE / sweep) removed the temp but left the Kolibri export CLI running into a deleted dir. stage_export now takes a should_cancel callback; the op layer passes "op is gone", and the CLI runs under Popen polled once a second and is terminated on cancel. - The startup sweep now also reclaims the pre-change "cxkolibri-" temp prefix for one release.
The in_use pin was set in fetch() before streaming started, so a failure in that window (open_fetch raises, or the client disconnects before the stream body) left the op in_use forever, since the sweep now skips in_use ops. Move the pin into the stream lifecycle: fetch() returns the stream unpinned (op merely fresh), an on_open hook pins it at the top of the server's streaming try, and the existing finally (on_close) clears it. A pre-stream failure now leaves the op reclaimable by the normal sweep; open_fetch raising drops the op and its temp and returns 500.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Large content export timed out: the box stages a Kolibri channel to a temp before
the first byte streams, so a single blocking GET exceeded the client read timeout.
Export is now a long operation on the core (cx_core 0.2.0):
Calibre) are ready at once; a Kolibri channel stages in a worker thread.
poll is the liveness signal, so there is no wall-clock stage timeout.
Op lifecycle (in-memory registry, one cleanup primitive):
so the TTL sweep never reclaims a temp that is being streamed.
release.
Python stdlib only. 150 unit tests.