feat: alert when octolens mapping discovery looks unhealthy

[themarolt]

Summary

Adds an hourly cron job that posts to the integrations alerts channel when Octolens keyword mapping discovery looks unhealthy. Discovery is judged once for the whole platform, across every per-subproject Octolens integration: it alerts when discovery hasn't written any mappings in over 3 hours, and when it deactivated mappings since the last report. It's quiet when healthy, and quiet before discovery has ever written a mapping (no token yet, or never run). This is visibility only; ingestion is unchanged.

Mentions are no longer re-fetched automatically, so both alerts give the exact recovery command for the manual backfill script (pnpm --filter @crowd/connectors-worker run script:backfill-octolens-mentions, added in #5013): --keywords for a keyword that gets mapped again, --from <last write> after a discovery outage, each with a hint to try --dry-run first. If --from is refused (not earlier than the current start date), the poll may already have read past skipped mentions, so the alert points to a --keywords backfill for keywords mapped since then.

The diff is about 1.1k lines, mostly tests (about 700). The backfill script PR must merge before this one, since both alerts name its command.

Changes

• DAL (protected path, additive only): new octolens/health.ts, plus export on the existing OWNS_NO_ACTIVE_SYNC_UNITS filter in octolens/index.ts so both share it.
  • getOctolensMappingDiscoveryHealth returns null when there are no active mappings, otherwise the active mapping count, the number of subproject integrations they belong to, and the newest lastSeenAt among them. Only live subproject integrations count. Soft-deleted ones, the segment-less anchor that owns the mention poll, and any integration owning an active sync unit are left out, using the same rule as discovery. Discovery stamps every mapping it keeps in one transaction, so the newest lastSeenAt is its last write.
  • listOctolensKeywordMappingsDeactivatedBetween returns mappings deactivated in a window with their project name and slug, plus the project the keyword is mapped to now if it moved (only a live subproject integration counts, by the same rule as above, so a stray mapping under a legacy sync-unit owner can't hide a real deactivation). It keeps mappings whose subproject integration is now soft-deleted, because discovery soft-deletes a subproject in the same run that deactivates its last keyword, and that's the deactivation most worth seeing.
• New octolens-mapping-health-check job: hourly at :30 (discovery runs on the hour), prod only, reads from the primary. It checks staleness first, then deactivations, each in its own try/catch, so a failure in one never skips the other.
• No active mappings: the job does nothing and writes no state. A discovery run that finds no mappings writes nothing, so a populated table can't end up empty. Zero active mappings therefore means discovery has never written any, for example while CROWD_OCTOLENS_API_TOKEN is unset. This keeps it from alerting every hour before the first run.
• Stale alert (warning) when the newest lastSeenAt is over 3 hours old, which with hourly discovery means about three runs in a row wrote nothing. A failed run, an incomplete Octolens snapshot, a run that finds no mappings and a missing CROWD_OCTOLENS_API_TOKEN all write nothing; the alert names them, the schedule and the logs to check, and the --from <last write> command that recovers mentions of keywords added meanwhile. One alert per episode, keyed on that timestamp, plus a reminder at most once a day (Redis key with a 24h expiry).
• Deactivation alert lists keyword (id), project (slug), deactivation time, whether the project has no Octolens keywords left, and the new project when the keyword moved (its mentions keep flowing there, so moves don't count toward a burst). For keywords that are simply gone, it gives the --keywords backfill command for when they are mapped again. If any discovery run in the window deactivated 3 or more mappings, or at least 25% of them, the alert goes out as a warning titled "in bulk" instead of info. Rows from one run share a timestamp, because discovery deactivates in one transaction. The list shows the newest deactivations first and is capped at 25 mappings and at 2,800 characters after escaping, with "… and N more" beyond that. Names are truncated when long and shown as code, so *, _ or ~ in a keyword or project name can't format the message. That keeps it inside one Slack section block, so @crowd/slack never prunes the message after the cursor has moved.
• A Redis cursor records how far deactivations have been reported. It only moves after a successful Slack send, so a failed send is retried next run, and the failure is logged with the window start so a growing backlog is visible. The job logs its result as a structured object. The window ends 5 minutes before the database's current time. That margin has to exceed the reconcile transaction's duration, because deactivatedAt is that transaction's start time. The first run looks back 24 hours. If the cursor is lost, the next run repeats up to a day of deactivations; none are dropped.
• Decision and formatting logic lives in a separate module with unit tests. The DAL queries have DB tests, including one that runs the real reconcile and checks that a soft-deleted subproject's deactivations are still reported and a moved keyword names its new project but never a soft-deleted, other-platform or legacy integration; the list also leaves out the anchor and other platforms.

Type of change

• Bug fix
• New feature
• Refactor / cleanup
• Performance improvement
• Chore / dependency update
• Documentation

Related issue

Closes linuxfoundation/lfx-cdp-insights-ops#806

[cursor]

PR Summary

Medium Risk
Additive read-only monitoring on the Octolens integration path; alert correctness depends on primary DB timing, Redis cursor state, and Slack delivery, but no mapping or ingestion writes.

Overview
Adds production monitoring for Octolens keyword-to-project mapping discovery via a new hourly cron (octolens-mapping-health-check) that posts to the integrations Slack channel when discovery looks unhealthy. Ingestion behavior is unchanged; this is visibility and runbook guidance only.

New DAL helpers report platform-wide discovery health (getOctolensMappingDiscoveryHealth) and keyword mappings deactivated in a time window (listOctolensKeywordMappingsDeactivatedBetween), reusing the existing subproject-integration filter (OWNS_NO_ACTIVE_SYNC_UNITS is now exported). The job stays silent when there are no active mappings (e.g. before first discovery). When mappings exist, it warns if the newest lastSeenAt is older than 3 hours (once per episode, Redis-deduped), and reports deactivations since the last successful alert using a Redis cursor that only advances after Slack succeeds (24h first lookback, 5-minute safety margin, primary DB reads).

Alerts distinguish bulk deactivations (warning) from smaller batches (info), cap long Slack lists, escape/truncate names, and include manual backfill-octolens-mentions recovery commands (--from after staleness, --keywords after deactivations). Staleness and deactivation checks run in isolation so one failure does not skip the other.

^{Reviewed by Cursor Bugbot for commit 0ed0c15. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

🟡 Changes recommended

Alerts can be lost during retries or payload pruning, and partial discovery writes can incorrectly indicate healthy discovery.

4 open findings

• Persist retry cursor start time before delivery · New
• Batch Slack alerts and require all batches to succeed · New
• Persist deactivation events independently of current mappings · New
• Advance discovery health timestamp only after successful completion · New

What changed in this PR

Adds production monitoring for Octolens mapping discovery, alerting when discovery appears stale or mappings are deactivated.

Changes:

• Adds health-state and deactivation queries.
• Adds an hourly Slack alert job with Redis retry and reminder tracking.
• Adds database and alert-logic tests.

File	Description
services/​libs/​data-access-layer/​src/​octolens/​index.ts	Adds discovery health queries.
services/​libs/​data-access-layer/​src/​octolens/​index.test.ts	Tests query results and time boundaries.
services/​apps/​cron_service/​src/​jobs/​octolensMappingHealthCheck.job.ts	Connects the hourly job to PostgreSQL, Redis, and Slack.
services/​apps/​cron_service/​src/​jobs/​octolensMappingHealth.ts	Implements alert formatting and delivery tracking.
services/​apps/​cron_service/​src/​jobs/​octolensMappingHealth.test.ts	Tests thresholds, reminders, and retries.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Integrations that never acquire their first mapping remain invisible to the health check.

1 open finding

• Detect stale Octolens integrations with no mappings · New

4 resolved since last review

• Persist deactivation events independently of current mappings
• Batch Slack alerts and require all batches to succeed
• Persist retry cursor start time before delivery
• Advance discovery health timestamp only after successful completion

🧠 Review effort: Balanced

[Copilot]

🟢 Approval recommended

No unresolved blocking findings remain within the accepted monitoring scope.

0 open findings

1 resolved since last review

• Detect stale Octolens integrations with no mappings

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

Historical burst percentages use current mapping counts, which can downgrade qualifying warnings after later additions.

1 open issue

• Use each run's mapping baseline for burst percentage thresholds · New

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Recovery guidance incorrectly treats a refused rewind as proof that skipped mentions need no backfill.

1 open issue

• Refused rewind can leave skipped mentions unrecovered · New

1 resolved since last review

• Use each run's mapping baseline for burst percentage thresholds

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

Static review found no unresolved blocking issues in the additive monitoring changes; automated tests were not executed.

No open issues 🎉

1 resolved since last review

• Refused rewind can leave skipped mentions unrecovered

_{Review effort: Balanced}