feat: add a guarded octolens backfill script and keep running sync units locked

[themarolt]

Summary

The Octolens mention poll starts at "now" and no longer backfills on its own. This adds the manual way to get older mentions in: a connectors_worker script that queues per-keyword backfills or moves the poll's start date back, writing to the mention poll's watermark only while no run holds the sync unit.

To make that lock check trustworthy, the dispatcher now keeps a sync unit locked for the whole time its run is going. Before, it cleared the lock whenever it found a unit still running. The regular poll is also capped at 20 pages per run, so a large rewind is read over several runs at the normal 5-minute cadence and doesn't eat the shared Octolens request budget.

Changes

• New script pnpm --filter @crowd/connectors-worker run script:backfill-octolens-mentions -- (--keywords <id,...> | --from <date>) [--dry-run].
  • --keywords appends keywords to the poll's backfill queue, skipping ones already queued (a repeated id counts once) and keeping existing entries and their cursors.
  • --from moves startDate back and resets cursor and pendingMaxTimestamp, keeping the backfill queue.
• Everything is checked before writing:
  • exactly one of the two flags;
  • keyword ids are positive integers, each with exactly one active subproject mapping (a keyword mapped twice is refused, since the poll treats it as unmapped);
  • --from is a real date or a timestamp with a zone, not in the future, at most a year back, and earlier than the current start date;
  • exactly one mentionPoll sync unit under the segment-less Octolens integration, and it is active.
• Exit codes: 0 done or nothing to do, 1 invalid input or state, 2 a poll run is in progress or the start date changed since the check (retry in a few minutes).
• The write is one conditional UPDATE in the DAL (services/libs/data-access-layer/src/octolens). It merges into the stored JSON and only applies when the unit is active and lockedAt is NULL, so it can't race a run that would overwrite the watermark at its end. A rewind also only applies over the start date it was validated against, so two concurrent rewinds can't undo each other, and a queued entry counts as queued only if the poll would use it (cursor null or a string).
• Sync unit lock: every run is now covered by lockedAt from start to end. The dispatcher's claim sets it, executeSync sets it again before reading the unit (a previous run's end write can clear it between the claim and the start), and the run's own end write clears it.
  • When a claimed unit's workflow is still running, the dispatcher renews the lease (lockedAt = now(), nextRunAt = end of the 5-minute lease) instead of clearing the lock. If the run has already recorded its end, the renewal does nothing.
  • A unit claimed over a lapsed lease and then deferred for lack of token headroom also keeps its claim until the lease ends (claimDueUnits returns leaseLapsed).
  • This applies to every connector. For GitHub, a run longer than the 5-minute lease now stays locked until it ends and is re-checked when each lease lapses, about every 5 minutes, as before. Freshly claimed units that get deferred are released as before.
• New optional SyncOutcome.resumeAtCadence: an unfinished run waits for the sync's normal cadence instead of the ~1-minute short defer, and records lastRunComplete = false. Existing { complete: true } / { complete: false } outcomes behave as before.
• Mention poll: the regular poll stops after 20 pages per run and commits its cursor like the out-of-time path. It skips the keyword backfill that run and resumes at the next cadence. Worst case is 20 + 8 pages per run.
• Tests:
  • real-DB tests for the watermark writes (append, dedup, no or malformed backfill, rewind, refusal when locked or not active), the lease renewal, and the claim query skipping a renewed unit;
  • unit tests for argument and plan validation, dispatcher back-off and deferral, the sync run's handling of each outcome, and the poll cap.

Deploy note

Runs started by the new code lock their unit for their whole duration, so the script's lock check is reliable for them. A mention poll run started by the old code before the deploy could still be running with its lock cleared. Runs time out after 60 minutes, so wait an hour after the connectors_worker deploy before the first script run. A normal poll run takes seconds, so in practice nothing old is left after a few minutes.

Type of change

• Bug fix
• New feature
• Refactor / cleanup
• Performance improvement
• Chore / dependency update
• Documentation

Related issue

Refs linuxfoundation/lfx-cdp-insights-ops#748

[cursor]

PR Summary

Medium Risk
Changes dispatcher locking and scheduling for every connector sync unit, plus conditional watermark writes; behavior is well-tested but affects core connector infrastructure.

Overview
Adds a connectors_worker script to manually backfill Octolens mentions by either queuing keyword IDs on the mention-poll watermark or rewinding startDate (with validation, dry-run, and conditional DAL writes only when the unit is idle and unlocked).

Sync-unit locking is tightened for all connectors: runs reassert lockedAt at start; the dispatcher renews the lease when a workflow is already running (replacing Temporal probe rescheduling); deferred units with a lapsed lease keep the claim instead of being released. SyncOutcome.resumeAtCadence schedules the next attempt on the normal cadence when a run is incomplete.

The Octolens mention poll caps the regular fetch at 20 pages per run, persists cursor state, and skips keyword backfill until the cycle completes—so large rewinds spread across cadence ticks without blowing the API budget.

^{Reviewed by Cursor Bugbot for commit cef0431. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

🟡 Changes recommended

Malformed queue entries can suppress backfills, and concurrent rewind commands can overwrite an earlier requested window.

2 open issues

• Malformed queued entries suppress keyword backfills · New
• Concurrent rewinds can overwrite earlier start dates · New

What changed in this PR

Adds manual Octolens backfills and strengthens connector locking to protect watermark updates.

Changes:

• Adds validated keyword-backfill and date-rewind commands.
• Retains sync-unit locks throughout running syncs.
• Caps regular mention polling at 20 pages, resuming at normal cadence.

File	Description
services/​libs/​data-access-layer/​src/​octolens/​mentionPollWatermark.test.ts	Tests guarded watermark writes.
services/​libs/​data-access-layer/​src/​octolens/​index.ts	Adds poll lookup and watermark updates.
services/​libs/​data-access-layer/​src/​connectors/​types.ts	Tracks lapsed leases on claims.
services/​libs/​data-access-layer/​src/​connectors/​syncUnits.ts	Adds lease renewal and claim reassertion.
services/​libs/​data-access-layer/​src/​connectors/​syncUnits.test.ts	Tests claim and lease behavior.
services/​libs/​connectors/​src/​types.ts	Adds cadence-based continuation option.
services/​libs/​connectors/​src/​connectors/​octolens/​syncs/​mentionPoll.ts	Caps regular polling pages.
services/​libs/​connectors/​src/​connectors/​octolens/​syncs/​mentionPoll.test.ts	Tests page caps and resumption.
services/​apps/​connectors_worker/​src/​workflows/​dispatcher.ts	Passes lease state to deferral.
services/​apps/​connectors_worker/​src/​scheduling.ts	Removes obsolete running-probe scheduling.
services/​apps/​connectors_worker/​src/​octolensBackfill.ts	Validates arguments and builds backfill plans.
services/​apps/​connectors_worker/​src/​octolensBackfill.test.ts	Tests argument and plan validation.
services/​apps/​connectors_worker/​src/​bin/​backfill-octolens-mentions.ts	Adds the guarded backfill command.
services/​apps/​connectors_worker/​src/​activities/​syncRunActivities.ts	Reasserts claims and handles cadence continuation.
services/​apps/​connectors_worker/​src/​activities/​syncRunActivities.test.ts	Tests claims and outcome scheduling.
services/​apps/​connectors_worker/​src/​activities/​dispatcherActivities.ts	Preserves claims for running units.
services/​apps/​connectors_worker/​src/​activities/​dispatcherActivities.test.ts	Tests lease renewal and deferral.
services/​apps/​connectors_worker/​package.json	Registers the backfill script.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Connector-wide lock ownership and timeout overlap remain unresolved and require human validation before deployment.

1 open issue

• Run-end update can unlock replacement after activity timeout · New

2 resolved since last review

• Concurrent rewinds can overwrite earlier start dates
• Malformed queued entries suppress keyword backfills

_{Review effort: Balanced}