Skip to content

feat(source): compile a Google Drive folder into the local wiki - #278

Open
kalyan-fireweave wants to merge 2 commits into
VectifyAI:mainfrom
kalyan-fireweave:feat/gdrive-remote-source
Open

kalyan-fireweave wants to merge 2 commits into
VectifyAI:mainfrom
kalyan-fireweave:feat/gdrive-remote-source

Conversation

@kalyan-fireweave

Copy link
Copy Markdown

Closes #275.

This branch includes the unmerged S3 work from #277 (commit 6f8d0be). Merge #277 first. The Drive-only commit is 2ebeb18. OneDrive (#276) stays a vote card. This is separate from #61, which asks to store the wiki itself on S3-compatible storage.

A maintainer needs to add the connector:gdrive label so the Workbench vote search for unshipped providers can find this issue. This account cannot create labels on VectifyAI/OpenKB.

Summary

openkb source add gdrive
  parse folder URL or id
  access token (refresh, or loopback sign-in)
    oauth.py listens on 127.0.0.1:7567 only
    refresh token -> <kb>/.env mode 0600
  store.py writes connectors[].folder_id
openkb source sync
  list the folder recursively
  add_single_file(identity=gdrive://{folderId}/{path})

Workbench
  Settings (no KB)   # Drive says to open a knowledge base
  KbDetail Documents # form, or list + Sync / Remove / Add another
  OneDrive           # still a vote link

Problem

The Workbench shows Google Drive as a GitHub vote card. There is no way to bring a folder from the user's own Drive into the local wiki through the existing compile path.

Fix

  • openkb source add gdrive --name NAME --folder FOLDER. The folder is a URL or a raw id from the user's own Drive. Shared Drives are out of scope.
  • Credentials are OPENKB_GDRIVE_CLIENT_ID, OPENKB_GDRIVE_CLIENT_SECRET, and OPENKB_GDRIVE_REFRESH_TOKEN. The refresh token is written by sign-in into <kb>/.env only, through the existing mode 0600 env merge (openkb/envfile.py). The global config env stays read-only. Secrets are not written to config.yaml, the wiki, or connector-state.json.
  • Sign-in is a loopback on 127.0.0.1:7567 with PKCE and a state parameter. The redirect sent to Google is http://127.0.0.1:7567/callback. Google's desktop-app guide documents http://127.0.0.1:<port> for installed apps, and a Desktop client has no redirect-URI field. Scope is drive.readonly. --non-interactive never opens a browser; add and sync then succeed only when a refresh token is already stored.
  • A rejected refresh token runs that same sign-in once, then lists the folder. Closing sign-in fails every Drive binding and leaves other providers running.
  • Sync downloads supported extensions and compiles them with add_single_file. The registry key is gdrive://{folderId}/{path from that folder}. Google Docs without an extension are skipped. Transient token and file failures retry 3 times.
  • google-api-python-client==2.201.0 and google-auth==2.62.1 are the optional gdrive extra. Tests inject a fake client and do not import the SDK. CI's uv sync --locked --extra dev --extra api does not install them.
  • The API calls the same functions as the CLI, on a worker thread, with non_interactive=True. A Workbench save without a refresh token returns 400 and tells the user to sign in with the CLI. It does not open a browser.
  • The Drive card flips from a vote link to the form or the list. S3 stays as shipped in feat(source): compile an Amazon S3 prefix into the local wiki #277. OneDrive stays a vote link.

Evidence

  • Before: Settings and the Documents pane showed Google Drive as "Want it? Vote".
  • After: Settings, with no knowledge base selected, says "Open a knowledge base to connect Google Drive." OneDrive stays "Want it? Vote". On a knowledge base, Drive shows Name, Folder, and Save binding, plus the hint to run openkb source add gdrive once. Saving without a refresh token shows "Sign-in is required" and writes no binding. With a binding present, the card lists notes, gdrive://1AbCdefghij/, "Not synced since this change", Sync, Remove, and Add another. Sync without a token records that sign-in error on the card. Remove clears the binding and leaves wiki/index.md in place. The same strings render in Chinese.

Verification

uv run ruff check .
uv run ruff format --check .
uv run mypy openkb
uv run pytest -q

Results here: ruff check passed, ruff format check passed, mypy reported no issues in 69 files, pytest 1281 passed.

cd frontend && npm run build

The web build passed (i18n:check, tsc -b, vite build).

Test plan

  • uv run ruff check .
  • uv run ruff format --check .
  • uv run mypy openkb
  • uv run pytest
  • cd frontend && npm ci && npm run build
  • Create a Google Desktop-app OAuth client, set OPENKB_GDRIVE_CLIENT_ID and OPENKB_GDRIVE_CLIENT_SECRET in <kb>/.env, run openkb source add gdrive, then openkb source sync
  • openkb source add gdrive --non-interactive with no refresh token fails before writing the binding
  • Confirm openkb source remove leaves compiled pages in the wiki
  • On a knowledge-base page, confirm the Drive form, the binding list, that S3 still connects, and that OneDrive is still a vote link
  • Add the connector:gdrive label
  • Merge only after feat(source): compile an Amazon S3 prefix into the local wiki #277

Merge Danger

Door: two-way

The command, the router, and the card can be reverted. A binding already saved under connectors: with provider: gdrive is ignored by a build that does not read it. Pages already compiled stay in the wiki, which is the same result as openkb source remove. The refresh token in <kb>/.env is left in place on revert.

Blast Radius: ingest

Local openkb add is unchanged. The new libraries are an extra, so the core install and CI's locked dev + api sync do not install them. oauth.py is the shared loopback OneDrive will reuse; this pull request does not add OneDrive. The env-file writer moved from api_config.py into openkb/envfile.py so sign-in does not import FastAPI. REST config patch still calls that same function.

Made with Cursor

Kalyan Chowdary and others added 2 commits November 5, 2026 18:56
Bindings stay in config and credentials stay in env files, so a remote object can recompile through the existing add path without a new storage backend.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign-in stays on loopback and writes the refresh token only into the knowledge base env file, so a binding can be saved without putting a secret in the wiki.

Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remote data source: ingest a Google Drive folder into the local knowledge base

1 participant