Skip to content

feat(anomaly-detection) add ADv2 - #131732

Draft
saponifi3d wants to merge 8 commits into
masterfrom
jcallender/anomaly-detection-poc-spike
Draft

saponifi3d wants to merge 8 commits into
masterfrom
jcallender/anomaly-detection-poc-spike

Conversation

@saponifi3d

@saponifi3d saponifi3d commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Description

Vibed a vibe.

This PR shall remain in draft, it's a way to visualize the diff / changes needed to turn the PoC into a Sentry Detector.

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Nov 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR has a migration; here is the generated SQL for src/sentry/migrations/1263_anomaly_detection_state.py

for 1263_anomaly_detection_state in sentry

--
-- Create model AnomalyDetectionState
--
CREATE TABLE "sentry_anomalydetectionstate" ("id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "date_updated" timestamp with time zone NOT NULL, "date_added" timestamp with time zone NOT NULL, "next_minute" timestamp with time zone NULL, "checkpoint" jsonb NULL, "delivery" jsonb NULL, "organization_id" bigint NOT NULL, CONSTRAINT "sentry_anomaly_state_org_uniq" UNIQUE ("organization_id"));
ALTER TABLE "sentry_anomalydetectionstate" ADD CONSTRAINT "sentry_anomalydetect_organization_id_291ac681_fk_sentry_or" FOREIGN KEY ("organization_id") REFERENCES "sentry_organization" ("id") DEFERRABLE INITIALLY DEFERRED NOT VALID;
ALTER TABLE "sentry_anomalydetectionstate" VALIDATE CONSTRAINT "sentry_anomalydetect_organization_id_291ac681_fk_sentry_or";

Comment on lines +97 to +103
if payload is not None:
return [
(datetime.fromisoformat(ts), signal, metric, project, env, dim, value)
for ts, signal, metric, project, env, dim, value in orjson.loads(
zlib.decompress(payload)
)
]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Malformed hourly snapshot cache entries abort detector tasks

If the cache returns a corrupt or malformed snapshot, _hourly_rows lets decompression, JSON parsing, row unpacking, or timestamp parsing errors escape instead of recomputing the snapshot. The task retries, and calls using the same hourly key can continue to fail until the entry expires or a different hourly key is used. Catch these decode errors, delete the entry, and fall through to collect(...). For example:

try:
    decoded = [
        (datetime.fromisoformat(ts), signal, metric, project, env, dim, value)
        for ts, signal, metric, project, env, dim, value in orjson.loads(zlib.decompress(payload))
    ]
except (zlib.error, orjson.JSONDecodeError, TypeError, ValueError):
    cache.delete(key)
else:
    return decoded
Evidence
  • _hourly_rows returns cached data whenever cache.get(key) is non-None; it decompresses, parses, unpacks each row, and parses timestamps without handling failures.
  • _run_organization calls _hourly_rows directly, and run_anomaly_detection has task retries but no handler that clears a bad cache entry or falls back to collection.
  • The cache key is hourly and the cache write uses a 3600-second TTL, so repeated calls using that key can keep failing until it expires or processing uses a different hourly key.

Identified by Warden · sentry-backend-bugs · DTN-U3U

no need to add all this to the execution.md file.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant