Skip to content

fix(export): stream and paginate database dumps for large databases (fixes #59) - #355

Open
Semihzx wants to merge 1 commit into
outerbase:mainfrom
Semihzx:fix/stream-database-dump-59
Open

Semihzx wants to merge 1 commit into
outerbase:mainfrom
Semihzx:fix/stream-database-dump-59

Conversation

@Semihzx

@Semihzx Semihzx commented Nov 18, 2026

Copy link
Copy Markdown

/claim #59

Summary

Fixes #59 ("Database dumps do not work on large databases").

Previously, dumpDatabaseRoute in src/export/dump.ts executed an unbounded SELECT * FROM <table>; for every table and concatenated the entire SQL dump into an in-memory string/Blob. On large databases, this exceeded Cloudflare Workers' memory and RPC payload limits.

Changes

  • Paginated table reads (LIMIT ? OFFSET ?): generateDumpChunks now fetches rows in configurable batches (default 500 rows via DumpDatabaseOptions.batchSize) instead of loading entire tables in a single query.
  • Backpressure-aware streaming (ReadableStream<Uint8Array>): Replaced in-memory string concatenation and Blob buffering with a ReadableStream using pull() and TextEncoder, keeping memory usage bounded to a single batch.
  • Stream cancellation & error propagation: Cancelling the response stream stops subsequent batch queries via iterator.return(), and mid-stream query errors propagate via controller.error(error) while pre-stream failures continue returning HTTP 500 JSON.
  • SQLite dump correctness & safety:
    • Excludes internal sqlite_% system tables from sqlite_master queries.
    • Uses parameterized sqlite_master schema lookup (name=?) and safely quotes non-standard table identifiers.
    • Properly formats NULL/undefined, booleans (1/0), and Uint8Array/ArrayBuffer BLOB values (X'...').
  • Unit tests (src/export/dump.test.ts): Added tests covering multi-page batching, exact batch multiples, 1,250-row streaming with default batch size, NULL/boolean/BLOB/special-identifier formatting, mid-stream error propagation, and early stream cancellation.

Testing

  • npx vitest run src/export/ — 4 test files passed, 29 tests passed (11/11 in src/export/dump.test.ts).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database dumps do not work on large databases

1 participant