Repository navigation
Description
Describe the bug
If you try to use any of the database dump endpoints such as SQL, CSV or JSON the data is loaded into memory and then created as a dump file. To support any size database we should investigate enhancements to allow any sized database to be exported. Currently the size limitations are 1GB for Durable Objects with 10GB in the future. Operate under the assumption that we might be attempting to dump a 10GB database into a .sql file.
Another consideration to make is because Durable Objects execute synchronous operations we may need to allow for "breathing intervals". An example might be we allow our export operation to run for 5 seconds, and take 5 seconds off if other requests are in a queue, then it can pick up again. The goal here would be to prevent locking the database for long periods of time.
But then poses the questions:
- How do we continue operations that need more than 30 seconds to work?
- Where is the data stored as it's being created? (R2, S3, something else)?
- How do we deliver that dump information to the user after its completed?
To Reproduce
Steps to reproduce the behavior:
- Hit the
/export/dumpendpoint on a large database - Will eventually fail when the 30 second request response time window closes
Run the following command in Terminal (replace the URL with yours) and if your operation exceeds 30 seconds you should see a failed network response instead of a dump file.
curl --location 'https://starbasedb.YOUR-ID-HERE.workers.dev/export/dump' \
--header 'Authorization: Bearer ABC123' \
--output database_dump.sql
If you can't create a large enough test database feel free to add code in to sleep for 29 seconds before proceeding with the /export/dump functional code and should also see the failure.
Expected behavior
As a user I would expect any and all of the specified data to be dumped out without an error and without partial results. Where it ends up for the user to access if the operation takes more than 30 seconds is up for discussion. Ideally if shorter than 30 seconds it could be returned as our cURL above works today (downloads the file from the response of the origin request), but perhaps after the timeout it continues on uploads it to a destination source to access afterwards?
Proposed Solution:
- For backups require an R2 binding
- Have a
.sqlfile that gets created in R2 with the filename likedump_20240101-170000.sqlwhere it represents2024-01-01 17:00:00 - Create the file and continuously append new chunks to it until reaching the end
- May need to utilize a DO alarm to continue the work after X time if a timeout occurs & mark where it currently is in the process in internal memory so it can pick up and continue.
- Provide a callback URL when the operation is finally completed so users can create custom logic to notify them (e.g. Email, Slack, etc)
Activity
Brayden commentedon Jan 24, 2025
/bounty $250
b4s36t4 commentedon Jan 24, 2025
Hi, @Brayden. instead of using DO alarm API we can use queues if the database is size greater than certain size (say 200 or 500)?
My suggestion is to attach the cloudflare queues to the new private endpoints where the consumer and producer are a single worker which will export the data into R2 for certain time say (20 to 30sec) depends on usage limits, this way we can have the queues with the capability of the last record of the export queue and there's blockage of any request/response.
https://developers.cloudflare.com/durable-objects/api/sql-storage/#databasesize we can depend on this property of DO to trigger the queue or something like.
Brayden commentedon Jan 24, 2025
Hey @b4s36t4!
Instead of using the DO alarm API, I think queues is also an acceptable approach. Just to talk through it a bit more so I fully understand how you're thinking of approaching it. A couple of questions:
Thanks for looking into this!
onyedikachi-david commentedon Jan 25, 2025
/attempt #59
JavaScript & more
algora-pbc commentedon Jan 25, 2025
💡 @onyedikachi-david submitted a pull request that claims the bounty. You can visit your bounty board to reward.
BenraouaneSoufiane commentedon Jan 27, 2025
/attempt #59 may I ask you for cloudflare paid account because I don't have one (it may resolved without cloudflare account following your guidances)
BenraouaneSoufiane commentedon Feb 2, 2025
/cancel #59
Kunal-Darekar commentedon Mar 6, 2025
/attempt #93
Hey @Brayden
Implementation Plan for Database Dump Feature:
Core Implementation:
Key Functions to Implement:
Testing Strategy:
Features to Cover:
Timeline:
Status: Implementation complete and ready for review. All tests passing.
Options
algora-pbc commentedon Mar 7, 2025
💡 @Kunal-Darekar submitted a pull request that claims the bounty. You can visit your bounty board to reward.
Kunal-Darekar commentedon Mar 16, 2025
@Brayden Apologies, I mistakenly deleted my original PR while making some update .I’ve created a new PR #93 and included the necessary documentation, including the wrangler.toml variables and the cURL command/endpoint for triggering. Everything should be clear and ready for testing. Could you please check? Thanks for your patience!
ezzeddinp commentedon Jan 30, 2026
hey @Brayden nice to know you!, are you still having these kind of issues? or it has already solved? let me know!, thanks @Brayden
whynice724-cell commentedon Feb 25, 2026
/attempt #59
Implementation plan (primary):
/export/dump) with a small sync fast-path for tiny DBs.I will start with SQL dump first (end-to-end), then extend the same job pipeline to CSV/JSON.
feat: implement streaming export for large databases (outerbase#59)
94 remaining items
Yuvi1192p commentedon Oct 24, 2026
I have resolved this issue in PR #308.
LIMIT 1000 OFFSET ${offset}generator (generateDumpChunks).escapeSqlValuesupporting SQLite NULL, booleans, bigints, andX'...'hex BLOB literals.ReadableStreamwithContent-Type: application/x-sqlite3.?location=r2(DATABASE_DUMP_BUCKETbinding) withctx.waitUntilreturning202 Acceptedimmediately and notifying an optional callback URL on completion./claim #59
isearchxsolve commentedon Oct 24, 2026
🛠️ Production Implementation & Technical Verification
dumpDatabaseRoute()currently callsexecuteOperation()for every full table, which reachescursor.toArray()inside the Durable Object. It then concatenates the entire database intodumpContentand creates oneBlob. A 10GB database therefore requires multiple 10GB-scale allocations before the first byte is sent.The fix is a pull-driven
ReadableStreamwith bounded page reads. It:pageSizerows per RPC._rowid_ > ?) where available; this preventsOFFSETbecoming O(n²).LIMIT/OFFSETforWITHOUT ROWIDtables.Blob.This directly replaces the failure mode in the current implementation. The existing source material confirms that the old route builds one full string after
cursor.toArray()materializes query output. Current dump implementationProduction Source (
src/export/dump.ts)Test Suite (
src/export/dump.test.ts)Run:
pnpm test --run src/export/dump.test.tsThis implementation removes the full-database allocation path, maintains bounded export memory, protects SQL correctness, and includes a runnable regression suite covering pagination,
WITHOUT ROWID, serialization, pre-stream failures, and invalid configuration. It is ready for immediate branch PR / merge after the stated test command is run in the repository checkout.Settlement coordinates:
doom14ac0e785bffabe0357cb522608cf88c80a33cd7f3f9257572qi849TBq72XMpsp8PjR4YfegDEzc7BRFZaarwQ2tKgLisearch.xsolve@gmail.comMilindChatrabhuji commentedon Oct 25, 2026
/attempt #59
Submitted PR #317 with incremental streaming via ReadableStream and constant-memory chunked pagination (CHUNK_SIZE = 1000). All 9/9 Vitest unit tests passing.
soundcake commentedon Oct 28, 2026
Is the $250 bounty still available for the resumable R2-backed export described here? Current main still buffers the dump, while #249 and #317 add response streaming; #76 and #93 cover broader background export designs.
I can work on a focused resumable export with persisted checkpoints, bounded chunks, concurrent-request yielding, and restart/round-trip tests. Please confirm whether to extend an existing approach or submit a separate implementation, and the required consistency semantics when writes occur during a backup.
Yuvi1192p commentedon Oct 29, 2026
/attempt #59
Submitted in PR #308 (opened Oct 24):
LIMIT 1000 OFFSET ${offset}) to support up to 10GB database exports without worker OOM.application/x-sqlite3) + background Cloudflare R2 multipart offload withwaitUntil.All acceptance criteria complete in PR #308, ready for maintainer review!
duongynhi000005-oss commentedon Oct 30, 2026
I investigated #59 (including the review feedback on PR #76) and traced the root causes in
src/export/dump.ts:Root Cause
src/export/dump.ts:13,SELECT name FROM sqlite_master WHERE type='table'includes internalsqlite_%tables, andSELECT * FROM ${table}(line 39) does not quote table identifiers ("${table.replace(/"/g, '""')}"), which throws a SQLite syntax error whenever a table name contains a period (e.g.users.table) or reserved keyword.SELECT *into an in-memory string hits the Cloudflare Worker 128 MB memory limit and 30-second request wall-clock limit on large databases.Proposed Fix
/export/dumpas the unified endpoint, escape table identifiers with"${table.replace(/"/g, '""')}", parameterize thesqlite_masterschema lookup, and filtername NOT LIKE 'sqlite_%'.WHERE rowid > ? ORDER BY rowid ASC LIMIT 1000) while incrementingprocessedTablesonly after each table's chunks complete.I have a branch ready (
duongynhi000005-oss/starbasedb:fix-issue-59-streaming-chunked-database-dump) and would be happy to submit a clean PR if you'd like to assign this to me!fix(export): stream chunked database dump and escape table identifiers (
test(export): add unit tests for paginated chunked dumps and breathin…
KrazyArcher commentedon Nov 11, 2026
/claim #59
Submitted streaming paginated database dump implementation and Vitest test coverage in PR: #343
Implementation Summary
dumpDatabaseRouteinsrc/export/dump.tsto stream SQL chunks viaReadableStream<Uint8Array>instead of buffering the entire dump string in memory before creating aBlob.batchSize) and cooperative event-loop breathing intervals (maxWorkSliceMs) to prevent Durable Object lockups and OOM crashes on large databases.src/export/dump.test.tscases and added multi-batch streaming unit test coverage.Payout Wallets
4Ch1arer4BF2ndVDq2pSRQdRfDQ8BNp7fg9tc3c5zc3Y0xde5e802f702446806a5f809e8f1f55a60214c991ItsMeSoumo commentedon Nov 13, 2026
/attempt #59
Implementation Plan: Streaming Chunked Dumps for Large Databases in StarbaseDB
I would like to claim this bounty to resolve the database dump failure on large databases.
Technical Architecture & Deliverables:
sqlite3/ DO SQLite step iteration).ReadableStream.BEGIN DEFERRED / IMMEDIATE) ensuring schema and data consistency.fix: stream NDJSON/SQL database dumps and support chunked import (out…
fix: stream NDJSON/SQL database dumps and support chunked import (out…
fix: stream NDJSON/SQL database dumps and support chunked import (out…
fix: stream NDJSON/SQL database dumps and support chunked import (out…
fix(export): stream and paginate database dumps for large databases (f…