You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Replicate data from external source to internal source with a Plugin #72
Is your feature request related to a problem? Please describe.
StarbaseDB instances support by default an internal database (SQLite offered by the Durable Object) as well as an optional external data source. External data sources can be powered in one of two ways, both by providing values in the wrangler.toml file of the project.
Outerbase API Key
Connection details of the database
Describe the solution you'd like
What would be beneficial for some use cases is the ability to bring in an external data source (e.g. a Postgres on Supabase) and have a pull mechanism where data can be brought into the internal DO SQLite so that the instance serves as a close-to-edge replica that can be queried alternatively to querying the Supabase Postgres instance.
Describe alternatives you've considered
Considering the pull vs push mechanism. A pull mechanism seems to be a better global solution where a push mechanism would be required to live elsewhere on a per provider basis.
Additional context
Might be beneficial for users to be able to define in the plugin what intervals data should be pulled at
Might be beneficial to allow users to define which tables should have data pulled into it (perhaps not all tables need replicated)
Likely need a way to know for each table what the last queried items were so you can do append-only type polling for new data. Does a user need to define a column to base this on (e.g. id or created_at columns perhaps)?
I will inspect the current plugin/replication architecture, implement the narrowest external-source-to-internal-source replication path, and add tests around the new plugin behavior.
Implementing pull-based external->internal replication as a plugin: per-table config with an optional incremental cursor column (persisted sync state), idempotent INSERT OR REPLACE upserts into the internal DO SQLite, POST /replicate[/:table] endpoints, and cron-triggerable intervals. PR incoming.
Implemented the pull-based ReplicationPlugin with incremental cursor tracking, idempotent upserts, schema-aware auto-provisioning, concurrency locks, REST endpoints, and 100% test coverage (17 new tests, 172/172 passing).
Submitted ReplicationPlugin implementation and Vitest test suite in PR: #344
Implementation Summary
Created ReplicationPlugin (plugins/replication/index.ts) extending StarbasePlugin with a pull-based replication engine to sync external databases (such as Supabase/Postgres or external HTTP/adapter sources) into the Durable Object's internal SQLite storage.
Supports automatic schema creation (CREATE TABLE IF NOT EXISTS), paginated batch pulls, idempotent INSERT OR REPLACE upserts, and incremental cursor tracking (_starbase_replication_state).
Exposes POST /replication/sync and GET /replication/status routes and includes unit test coverage in plugins/replication/index.test.ts.
Activity
Brayden commentedon Jan 24, 2025
/bounty $250
onyedikachi-david commentedon Jan 24, 2025
/attempt #72
JavaScript & more
algora-pbc commentedon Jan 24, 2025
💡 @onyedikachi-david submitted a pull request that claims the bounty. You can visit your bounty board to reward.
itsxdamdam commentedon Feb 7, 2025
@Brayden is this still open?
Kunal-Darekar commentedon Mar 19, 2025
/attempt #95
Options
algora-pbc commentedon Mar 23, 2025
💡 @Kunal-Darekar submitted a pull request that claims the bounty. You can visit your bounty board to reward.
MAVRICK-1 commentedon Jul 23, 2025
/attempt #95
feat: implement data replication plugin for external database sync
whynice724-cell commentedon Feb 25, 2026
/attempt #72
Secondary/fallback track:
whynice724-cell commentedon Feb 26, 2026
Implemented in PR #98.
This adds a Data Sync plugin for external -> internal replication with:
Please review when convenient. I will attach a short demo video in the PR.
justinleeyang commentedon Mar 6, 2026
I can take bounty #72. I will share a scoped implementation plan and first PR shortly.
justinleeyang commentedon Mar 6, 2026
Quick follow-up on #72: I am still actively working on this. Please let me know if you want me to narrow scope to a smaller first PR.
justinleeyang commentedon Mar 6, 2026
Quick follow-up on #72: I am still actively working on this. Please let me know if you want me to narrow scope to a smaller first PR.
44 remaining items
NAEL592 commentedon May 22, 2026
/attempt #72
I will inspect the current plugin/replication architecture, implement the narrowest external-source-to-internal-source replication path, and add tests around the new plugin behavior.
ThaiTrevor commentedon May 24, 2026
/attempt #72
FishEnjoyer2025 commentedon Jun 2, 2026
/attempt #72
Implementing pull-based external->internal replication as a plugin: per-table config with an optional incremental cursor column (persisted sync state), idempotent INSERT OR REPLACE upserts into the internal DO SQLite, POST /replicate[/:table] endpoints, and cron-triggerable intervals. PR incoming.
mikhaeelatefrizk commentedon Jun 7, 2026
/attempt #72
ThaiTrevor commentedon Jun 14, 2026
C:/Program Files/Git/attempt #72
PR #250 implements the ReplicatorPlugin. Already submitted — adding attempt registration now. Happy to address any review feedback.
Shaidyk commentedon Jun 15, 2026
/attempt #72
Shaidyk commentedon Jun 16, 2026
/claim #72
PR: #280
feat(plugins): implement external data replication plugin (outerbase#72)
test(plugins): add unit tests for ReplicationPlugin (outerbase#72)
feat(plugins): export ReplicationPlugin from dist/plugins.ts (outerba…
almagas commentedon Nov 10, 2026
/claim #72
Pull Request: #339
Implemented the pull-based
ReplicationPluginwith incremental cursor tracking, idempotent upserts, schema-aware auto-provisioning, concurrency locks, REST endpoints, and 100% test coverage (17 new tests, 172/172 passing).KrazyArcher commentedon Nov 11, 2026
/claim #72
Submitted
ReplicationPluginimplementation and Vitest test suite in PR: #344Implementation Summary
ReplicationPlugin(plugins/replication/index.ts) extendingStarbasePluginwith a pull-based replication engine to sync external databases (such as Supabase/Postgres or external HTTP/adapter sources) into the Durable Object's internal SQLite storage.CREATE TABLE IF NOT EXISTS), paginated batch pulls, idempotentINSERT OR REPLACEupserts, and incremental cursor tracking (_starbase_replication_state).POST /replication/syncandGET /replication/statusroutes and includes unit test coverage inplugins/replication/index.test.ts.Payout Wallets
4Ch1arer4BF2ndVDq2pSRQdRfDQ8BNp7fg9tc3c5zc3Y0xde5e802f702446806a5f809e8f1f55a60214c991khuxaiphi-stack commentedon Nov 12, 2026
/attempt #72
Implementation Plan
src/replication/index.tsimplementingReplicationPluginon top of StarbaseDB'sPluginlifecycle interface (src/plugin.ts).src/do.tsand add comprehensive Vitest unit tests insrc/replication/replication.test.ts.ItsMeSoumo commentedon Nov 13, 2026
/attempt #72
Implementation Plan: Streaming Data Replication Plugin for StarbaseDB
I would like to claim this bounty to implement the streaming data replication plugin for StarbaseDB.
Technical Architecture & Deliverables: