Skip to content

Replicate data from external source to internal source with a Plugin #72

Open

Description

@Brayden

Is your feature request related to a problem? Please describe.
StarbaseDB instances support by default an internal database (SQLite offered by the Durable Object) as well as an optional external data source. External data sources can be powered in one of two ways, both by providing values in the wrangler.toml file of the project.

  • Outerbase API Key
  • Connection details of the database
Image

Describe the solution you'd like
What would be beneficial for some use cases is the ability to bring in an external data source (e.g. a Postgres on Supabase) and have a pull mechanism where data can be brought into the internal DO SQLite so that the instance serves as a close-to-edge replica that can be queried alternatively to querying the Supabase Postgres instance.

Describe alternatives you've considered

  • Considering the pull vs push mechanism. A pull mechanism seems to be a better global solution where a push mechanism would be required to live elsewhere on a per provider basis.

Additional context

  • Might be beneficial for users to be able to define in the plugin what intervals data should be pulled at
  • Might be beneficial to allow users to define which tables should have data pulled into it (perhaps not all tables need replicated)
  • Likely need a way to know for each table what the last queried items were so you can do append-only type polling for new data. Does a user need to define a column to base this on (e.g. id or created_at columns perhaps)?

Activity

  1. Brayden commented on Jan 24, 2025

    @Brayden
    MemberAuthor

    /bounty $250

  2. onyedikachi-david commented on Jan 24, 2025

    @onyedikachi-david

    /attempt #72

    Algora profile Completed bounties Tech Active attempts Options
    @onyedikachi-david 14 bounties from 7 projects
    TypeScript, Python,
    JavaScript & more
    Cancel attempt
  3. algora-pbc commented on Jan 24, 2025

    @algora-pbc

    💡 @onyedikachi-david submitted a pull request that claims the bounty. You can visit your bounty board to reward.

  4. itsxdamdam commented on Feb 7, 2025

    @itsxdamdam

    @Brayden is this still open?

  5. Kunal-Darekar commented on Mar 19, 2025

    @Kunal-Darekar

    /attempt #95

  6. algora-pbc commented on Mar 23, 2025

    @algora-pbc

    💡 @Kunal-Darekar submitted a pull request that claims the bounty. You can visit your bounty board to reward.

  7. MAVRICK-1 commented on Jul 23, 2025

    @MAVRICK-1

    /attempt #95

  8. added a commit that references this issue on Jul 23, 2025
    df846c9
  9. whynice724-cell commented on Feb 25, 2026

    @whynice724-cell

    /attempt #72

    Secondary/fallback track:

    • I plan a plugin-based pull replication flow from external source -> internal DO SQLite.
    • Scope: configurable schedule, table allowlist, checkpoint-based incremental sync, and retry-safe upserts.
    • If Database dumps do not work on large databases #59 gets blocked, I can switch to this and deliver with demo video + docs.
  10. whynice724-cell commented on Feb 26, 2026

    @whynice724-cell

    Implemented in PR #98.

    This adds a Data Sync plugin for external -> internal replication with:

    • scheduled + manual runs
    • incremental checkpoints
    • conflict-safe upserts
    • target table auto-create + schema expansion

    Please review when convenient. I will attach a short demo video in the PR.

  11. justinleeyang commented on Mar 6, 2026

    @justinleeyang

    I can take bounty #72. I will share a scoped implementation plan and first PR shortly.

  12. justinleeyang commented on Mar 6, 2026

    @justinleeyang

    Quick follow-up on #72: I am still actively working on this. Please let me know if you want me to narrow scope to a smaller first PR.

  13. justinleeyang commented on Mar 6, 2026

    @justinleeyang

    Quick follow-up on #72: I am still actively working on this. Please let me know if you want me to narrow scope to a smaller first PR.

  14. 44 remaining items

  15. NAEL592 commented on May 22, 2026

    @NAEL592

    /attempt #72

    I will inspect the current plugin/replication architecture, implement the narrowest external-source-to-internal-source replication path, and add tests around the new plugin behavior.

  16. ThaiTrevor commented on May 24, 2026

    @ThaiTrevor

    /attempt #72

  17. FishEnjoyer2025 commented on Jun 2, 2026

    @FishEnjoyer2025

    /attempt #72

    Implementing pull-based external->internal replication as a plugin: per-table config with an optional incremental cursor column (persisted sync state), idempotent INSERT OR REPLACE upserts into the internal DO SQLite, POST /replicate[/:table] endpoints, and cron-triggerable intervals. PR incoming.

  18. mikhaeelatefrizk commented on Jun 7, 2026

    @mikhaeelatefrizk

    /attempt #72

  19. ThaiTrevor commented on Jun 14, 2026

    @ThaiTrevor

    C:/Program Files/Git/attempt #72

    PR #250 implements the ReplicatorPlugin. Already submitted — adding attempt registration now. Happy to address any review feedback.

  20. Shaidyk commented on Jun 15, 2026

    @Shaidyk

    /attempt #72

  21. Shaidyk commented on Jun 16, 2026

    @Shaidyk

    /claim #72

    PR: #280

  22. almagas commented on Nov 10, 2026

    @almagas

    /claim #72

    Pull Request: #339

    Implemented the pull-based ReplicationPlugin with incremental cursor tracking, idempotent upserts, schema-aware auto-provisioning, concurrency locks, REST endpoints, and 100% test coverage (17 new tests, 172/172 passing).

  23. KrazyArcher commented on Nov 11, 2026

    @KrazyArcher

    /claim #72

    Submitted ReplicationPlugin implementation and Vitest test suite in PR: #344

    Implementation Summary

    • Created ReplicationPlugin (plugins/replication/index.ts) extending StarbasePlugin with a pull-based replication engine to sync external databases (such as Supabase/Postgres or external HTTP/adapter sources) into the Durable Object's internal SQLite storage.
    • Supports automatic schema creation (CREATE TABLE IF NOT EXISTS), paginated batch pulls, idempotent INSERT OR REPLACE upserts, and incremental cursor tracking (_starbase_replication_state).
    • Exposes POST /replication/sync and GET /replication/status routes and includes unit test coverage in plugins/replication/index.test.ts.

    Payout Wallets

    • Solana (USDC / SOL): 4Ch1arer4BF2ndVDq2pSRQdRfDQ8BNp7fg9tc3c5zc3Y
    • EVM (Base / Polygon / Arbitrum / ERC-20): 0xde5e802f702446806a5f809e8f1f55a60214c991
  24. khuxaiphi-stack commented on Nov 12, 2026

    @khuxaiphi-stack

    /attempt #72

    Implementation Plan

    1. Create src/replication/index.ts implementing ReplicationPlugin on top of StarbaseDB's Plugin lifecycle interface (src/plugin.ts).
    2. Support configurable pull intervals, target table allowlists, primary-key upsert synchronization, and external source adapter queries (Outerbase API / external Postgres connection).
    3. Integrate Durable Object alarm/interval scheduling in src/do.ts and add comprehensive Vitest unit tests in src/replication/replication.test.ts.
  25. ItsMeSoumo commented on Nov 13, 2026

    @ItsMeSoumo

    /attempt #72

    Implementation Plan: Streaming Data Replication Plugin for StarbaseDB

    I would like to claim this bounty to implement the streaming data replication plugin for StarbaseDB.

    Technical Architecture & Deliverables:

    1. Extensible Plugin Interface:
      • Define clean TypeScript plugin interfaces for source connectors (PostgreSQL WAL/logical replication, MySQL binlog, SQLite WAL).
      • Standardize change events (INSERT, UPDATE, DELETE, DDL schema mutations).
    2. Streaming CDC & Change Consumer:
      • Ingest change streams with cursor/offset tracking and checkpointing to ensure zero message loss and exactly-once delivery.
      • Batch incoming change vectors for high throughput into StarbaseDB SQLite instances.
    3. Conflict Resolution & Fault Tolerance:
      • Implement deterministic conflict resolution strategies (last-write-wins, source-priority, custom resolver hooks).
      • Exponential backoff and dead-letter queue for unparseable records.
    4. Integration Testing & Documentation:
      • Comprehensive test harness simulating high-concurrency source writes and verifying data consistency in StarbaseDB.
      • Setup guide with Docker Compose examples.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions