diff --git a/README.md b/README.md index 65f9ee0..6ffeb6f 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,14 @@ [![CI](https://github.com/MCCProgrammer/extpulse/actions/workflows/ci.yml/badge.svg)](https://github.com/MCCProgrammer/extpulse/actions/workflows/ci.yml) [![CodeQL](https://github.com/MCCProgrammer/extpulse/actions/workflows/codeql.yml/badge.svg)](https://github.com/MCCProgrammer/extpulse/actions/workflows/codeql.yml) +[![Quality Gate](https://sonarcloud.io/api/project_badges/measure?project=MCCProgrammer_extpulse&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=MCCProgrammer_extpulse) +[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=MCCProgrammer_extpulse&metric=coverage)](https://sonarcloud.io/summary/new_code?id=MCCProgrammer_extpulse) Daily stats, reviews and alerts for VS Code and Chrome extensions, across the VS Code Marketplace, Open VSX and the Chrome Web Store. -> **Status:** in development. Accounts, publishers, stats, a reviews inbox and alerts (email and Discord) work through the REST API and a server-rendered dashboard, with an opt-in public page and README badges. The daily collection runs on its own. The production setup and automatic deploys are ready; the server is being created. +**Live at **, in production since October 2026 on a +free Oracle Cloud VM, deployed from every merge to `main`. ## Why @@ -15,14 +18,66 @@ extension on the Marketplace and on Open VSX, nobody tells you the total, how ma installs you gained this week or that a 1★ review just arrived. ExtPulse saves one snapshot per extension, per store, per day, and builds stats and alerts on top. -## Planned (v1) +## What it does + +- **Finds a publisher's extensions** on the Marketplace and Open VSX; Chrome extensions + are added by their item ID (the store can't be searched by publisher). +- **Collects every day at 06:00** (Europe/Lisbon): one snapshot per listing, retries with + backoff, one request per second per store, and a failing store doesn't stop the others. +- **Stats:** combined installs across stores, daily and weekly gains, charts over 30, 90 + or 365 days, through the REST API and a server-rendered dashboard. +- **One inbox for the reviews of every store**, with unread and unanswered filters. +- **Alerts** by email or Discord: low ratings, install spikes, new versions, a failing + collection and a weekly summary, delivered through an outbox with retries. +- **A public portfolio page** (opt-in) and SVG installs badges for READMEs. + +## Architecture + +A modular monolith: one Spring Boot app, one package per feature, and the store +adapters as the only code that talks to the stores. + +```mermaid +flowchart LR + Users((Users)) --> Caddy + CI[GitHub Actions] -- "image on GHCR, deploy over SSH" --> App + subgraph VM[Oracle Cloud VM] + Caddy[Caddy: HTTPS] --> App + App["Spring Boot app
store adapters, daily collection, stats,
reviews, alerts outbox, REST API, dashboard"] + App --> DB[(PostgreSQL)] + Mon[Prometheus, Alertmanager, Grafana] -- scrapes --> App + DB --> Backup[pg_dump, rclone] + end + App -- alerts --> Notify[Email, Discord] + Mon -- alerts --> Notify + App -- "every day at 06:00" --> Stores[VS Code Marketplace, Open VSX, Chrome Web Store] + Backup --> Bucket[(Object Storage)] +``` -- Find all extensions of a publisher on the three stores -- Daily collection at 06:00 (Europe/Lisbon), one snapshot per listing per day -- Combined totals, daily and weekly gains, split by store -- One inbox for the reviews of every store -- Alerts: low ratings, install spikes, new versions, weekly summary -- Public portfolio page and a combined installs badge for READMEs +## Engineering highlights + +- **Tests against recorded store responses:** WireMock serves real, anonymized responses + of each store; a weekly workflow runs the same adapters against the live stores to + catch changes. Testcontainers runs the database tests on a real PostgreSQL. A JaCoCo + gate keeps coverage above 85% of lines. +- **SQL where it says more:** stats use window functions (`LAG`, `FILTER`, `LATERAL`) + with carry-forward for missing days; reviews are upserts that keep when each was + first seen. +- **Security:** two Spring Security chains (stateless JWT for `/api`, session with CSRF + and a strict Content Security Policy for the website), BCrypt, the same answer and + timing for an unknown email and a wrong password, and Discord webhooks checked + against an allow-list (SSRF). +- **Infrastructure as code at 0 €:** Terraform creates the network, firewall, VM, + Object Storage bucket, IAM policies and quotas that refuse anything outside Always + Free. Backups go off the server to a bucket the server can write to but not delete + from. +- **CI/CD:** build, tests and coverage gate, an Arm image on GHCR, a secret scan, + CodeQL, SonarQube Cloud, Terraform and monitoring checks; a merge to `main` deploys + over SSH, waits for the health check and rolls back if it fails. +- **Observability:** JSON logs with the collection run ID, Micrometer metrics with + Prometheus, a Grafana dashboard kept as code, and alert rules with unit tests, + delivered to Discord. +- **Decisions written down:** [19 architecture decision records](docs/decisions), from + the Java version to the trusted certificate authorities. ## Tech stack diff --git a/deploy/README.md b/deploy/README.md index 929e484..849a4ab 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -46,6 +46,25 @@ On the server only, never in the repository: docker compose restart alertmanager ``` + Alerts set up in the app (low ratings, spikes, the weekly summary) can also go by + email, through any SMTP service. With Resend: verify the sending domain, create an + API key with sending access, then on the server write the settings and pipe the key + from the clipboard on your machine: + ```sh + cat >> app.env <<'SETTINGS' + SPRING_MAIL_HOST=smtp.resend.com + SPRING_MAIL_PORT=587 + SPRING_MAIL_USERNAME=resend + SPRING_MAIL_PROPERTIES_MAIL_SMTP_STARTTLS_ENABLE=true + SPRING_MAIL_PROPERTIES_MAIL_SMTP_STARTTLS_REQUIRED=true + EXTPULSE_ALERTS_MAIL_FROM=ExtPulse + SETTINGS + chmod 600 app.env + # on your machine, with the API key copied: + pbpaste | ssh deploy@ 'cd /opt/extpulse && { printf "SPRING_MAIL_PASSWORD="; cat; echo; } >> app.env && ./deploy.sh "$(cat .deployed-tag)"' + ``` + Then send a test from an email alert in the dashboard (Alerts, "Send test"). + 6. Turn on the copies to Object Storage: the namespace is `terraform output -raw backup_namespace` in `infra/oracle`. Then on the server: ```sh