[kalyan-fireweave]

Closes #274.

Google Drive (#275) and OneDrive (#276) stay vote cards. This pull request is the S3 ingest path only. It is separate from #61, which asks to store the wiki itself on S3-compatible storage.

A maintainer needs to add the connector:s3 label so the Workbench vote search for unshipped providers can find this issue. This account cannot create labels on VectifyAI/OpenKB.

Summary

openkb source add|sync|remove
  credentials.py          # process env, then <kb>/.env, then ~/.config/openkb/.env
  store.py                # connectors list in .openkb/config.yaml
  sync.py
    S3Backend.list/download
    add_single_file(identity=s3://bucket/key)
  connector-state.json    # last sync only; no secrets

Workbench
  Settings (no KB)        # S3 says to open a knowledge base
  KbDetail Documents      # form, or list + Sync / Remove / Add another
  Drive, OneDrive         # vote links

Problem

The Workbench shows Amazon S3 as a GitHub vote card. There is no way to bring a bucket prefix into the local wiki through the existing compile path.

Fix

• openkb source add s3, list, sync, and remove. One binding is one region, bucket, and prefix. An empty prefix is the bucket root.
• Credentials are OPENKB_S3_ACCESS_KEY_ID and OPENKB_S3_SECRET_ACCESS_KEY. They are required before the binding is saved. They are never written to config.yaml, the wiki, or connector-state.json.
• Sync lists the prefix, downloads supported extensions, and compiles each file with add_single_file. The registry key is s3://{bucket}/{key}. The same key and same bytes are skipped. The same key and new bytes keep the wiki name and recompile. A later file failure keeps the files that already succeeded. Transient HTTP errors are retried 3 times.
• openkb source remove deletes the binding and leaves compiled pages in place.
• boto3==1.43.124 is the optional s3 extra. The core install is unchanged. A missing extra prints pip install "openkb[s3]".
• The API calls the same functions as the CLI, on a worker thread. POST /api/v1/connectors/sync defaults to non-interactive.
• The S3 card has three states: vote only while the provider is unshipped; a connect form when a knowledge base is open and has no binding; a list once a binding exists. Settings, which has no knowledge base selected, tells the user to open one.

Evidence

• Before: the Documents pane and Settings showed three vote cards, including Amazon S3.
• After: on a knowledge base, Amazon S3 shows Name / Region / Bucket / Prefix and Save binding. After save, the card lists papers, s3://demo-bucket/papers/, "Not synced since this change", Sync, Remove, and Add another. Google Drive and OneDrive stay "Want it? Vote". Settings, with no knowledge base selected, says "Open a knowledge base to connect Amazon S3." Sync without the extra shows pip install "openkb[s3]". Remove clears the binding and leaves wiki/keep.md in place. config.yaml holds the location only.

Verification

uv run ruff check .
uv run ruff format --check .
uv run mypy openkb
uv run pytest -q

Results here: ruff check passed, ruff format check passed, mypy reported no issues in 66 files, pytest 1263 passed.

cd frontend && npm ci && npm run build

The web build passed (i18n:check, tsc -b, vite build). A later label tweak on the S3 card was rechecked with tsc -b.

Test plan

• uv run ruff check .
• uv run ruff format --check .
• uv run mypy openkb
• uv run pytest
• cd frontend && npm ci && npm run build
• pip install "openkb[s3]", set the two OPENKB_S3_* keys in <kb>/.env, openkb source add s3, openkb source sync
• Confirm openkb source remove leaves compiled pages in the wiki
• On a knowledge-base page, confirm the S3 form, the binding list, and that Drive and OneDrive are still vote links
• Add the connector:s3 label

Merge Danger

Door: two-way

The command, the router, and the card can be reverted. A binding already saved under connectors: is ignored by a build that does not read it. Pages already compiled stay in the wiki, which is the same result as openkb source remove.

Blast Radius: ingest

Local openkb add is unchanged: identity is omitted unless a remote object supplies one. The new SDK is an extra, so the core install and CI's uv sync --locked --extra dev --extra api do not install boto3. The lock timeout defaults to the previous blocking acquire; only connector writes use the 30 second bound.

Made with Cursor