From 7c4def9b98882e76e770142af57ddb842092f190 Mon Sep 17 00:00:00 2001 From: MCCarmo Date: Wed, 28 Oct 2026 18:13:31 +0000 Subject: [PATCH] README: live link, features, architecture and engineering highlights The README still said in development and listed the features as planned. It now links the live site, says what works, draws the architecture and lists the engineering choices worth a look. The runbook explains alert emails through an SMTP service. --- README.md | 71 ++++++++++++++++++++++++++++++++++++++++++------ deploy/README.md | 19 +++++++++++++ 2 files changed, 82 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 65f9ee0..6ffeb6f 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,14 @@ [![CI](https://github.com/MCCProgrammer/extpulse/actions/workflows/ci.yml/badge.svg)](https://github.com/MCCProgrammer/extpulse/actions/workflows/ci.yml) [![CodeQL](https://github.com/MCCProgrammer/extpulse/actions/workflows/codeql.yml/badge.svg)](https://github.com/MCCProgrammer/extpulse/actions/workflows/codeql.yml) +[![Quality Gate](https://sonarcloud.io/api/project_badges/measure?project=MCCProgrammer_extpulse&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=MCCProgrammer_extpulse) +[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=MCCProgrammer_extpulse&metric=coverage)](https://sonarcloud.io/summary/new_code?id=MCCProgrammer_extpulse) Daily stats, reviews and alerts for VS Code and Chrome extensions, across the VS Code Marketplace, Open VSX and the Chrome Web Store. -> **Status:** in development. Accounts, publishers, stats, a reviews inbox and alerts (email and Discord) work through the REST API and a server-rendered dashboard, with an opt-in public page and README badges. The daily collection runs on its own. The production setup and automatic deploys are ready; the server is being created. +**Live at **, in production since October 2026 on a +free Oracle Cloud VM, deployed from every merge to `main`. ## Why @@ -15,14 +18,66 @@ extension on the Marketplace and on Open VSX, nobody tells you the total, how ma installs you gained this week or that a 1★ review just arrived. ExtPulse saves one snapshot per extension, per store, per day, and builds stats and alerts on top. -## Planned (v1) +## What it does + +- **Finds a publisher's extensions** on the Marketplace and Open VSX; Chrome extensions + are added by their item ID (the store can't be searched by publisher). +- **Collects every day at 06:00** (Europe/Lisbon): one snapshot per listing, retries with + backoff, one request per second per store, and a failing store doesn't stop the others. +- **Stats:** combined installs across stores, daily and weekly gains, charts over 30, 90 + or 365 days, through the REST API and a server-rendered dashboard. +- **One inbox for the reviews of every store**, with unread and unanswered filters. +- **Alerts** by email or Discord: low ratings, install spikes, new versions, a failing + collection and a weekly summary, delivered through an outbox with retries. +- **A public portfolio page** (opt-in) and SVG installs badges for READMEs. + +## Architecture + +A modular monolith: one Spring Boot app, one package per feature, and the store +adapters as the only code that talks to the stores. + +```mermaid +flowchart LR + Users((Users)) --> Caddy + CI[GitHub Actions] -- "image on GHCR, deploy over SSH" --> App + subgraph VM[Oracle Cloud VM] + Caddy[Caddy: HTTPS] --> App + App["Spring Boot app
store adapters, daily collection, stats,
reviews, alerts outbox, REST API, dashboard"] + App --> DB[(PostgreSQL)] + Mon[Prometheus, Alertmanager, Grafana] -- scrapes --> App + DB --> Backup[pg_dump, rclone] + end + App -- alerts --> Notify[Email, Discord] + Mon -- alerts --> Notify + App -- "every day at 06:00" --> Stores[VS Code Marketplace, Open VSX, Chrome Web Store] + Backup --> Bucket[(Object Storage)] +``` -- Find all extensions of a publisher on the three stores -- Daily collection at 06:00 (Europe/Lisbon), one snapshot per listing per day -- Combined totals, daily and weekly gains, split by store -- One inbox for the reviews of every store -- Alerts: low ratings, install spikes, new versions, weekly summary -- Public portfolio page and a combined installs badge for READMEs +## Engineering highlights + +- **Tests against recorded store responses:** WireMock serves real, anonymized responses + of each store; a weekly workflow runs the same adapters against the live stores to + catch changes. Testcontainers runs the database tests on a real PostgreSQL. A JaCoCo + gate keeps coverage above 85% of lines. +- **SQL where it says more:** stats use window functions (`LAG`, `FILTER`, `LATERAL`) + with carry-forward for missing days; reviews are upserts that keep when each was + first seen. +- **Security:** two Spring Security chains (stateless JWT for `/api`, session with CSRF + and a strict Content Security Policy for the website), BCrypt, the same answer and + timing for an unknown email and a wrong password, and Discord webhooks checked + against an allow-list (SSRF). +- **Infrastructure as code at 0 €:** Terraform creates the network, firewall, VM, + Object Storage bucket, IAM policies and quotas that refuse anything outside Always + Free. Backups go off the server to a bucket the server can write to but not delete + from. +- **CI/CD:** build, tests and coverage gate, an Arm image on GHCR, a secret scan, + CodeQL, SonarQube Cloud, Terraform and monitoring checks; a merge to `main` deploys + over SSH, waits for the health check and rolls back if it fails. +- **Observability:** JSON logs with the collection run ID, Micrometer metrics with + Prometheus, a Grafana dashboard kept as code, and alert rules with unit tests, + delivered to Discord. +- **Decisions written down:** [19 architecture decision records](docs/decisions), from + the Java version to the trusted certificate authorities. ## Tech stack diff --git a/deploy/README.md b/deploy/README.md index 929e484..849a4ab 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -46,6 +46,25 @@ On the server only, never in the repository: docker compose restart alertmanager ``` + Alerts set up in the app (low ratings, spikes, the weekly summary) can also go by + email, through any SMTP service. With Resend: verify the sending domain, create an + API key with sending access, then on the server write the settings and pipe the key + from the clipboard on your machine: + ```sh + cat >> app.env <<'SETTINGS' + SPRING_MAIL_HOST=smtp.resend.com + SPRING_MAIL_PORT=587 + SPRING_MAIL_USERNAME=resend + SPRING_MAIL_PROPERTIES_MAIL_SMTP_STARTTLS_ENABLE=true + SPRING_MAIL_PROPERTIES_MAIL_SMTP_STARTTLS_REQUIRED=true + EXTPULSE_ALERTS_MAIL_FROM=ExtPulse + SETTINGS + chmod 600 app.env + # on your machine, with the API key copied: + pbpaste | ssh deploy@ 'cd /opt/extpulse && { printf "SPRING_MAIL_PASSWORD="; cat; echo; } >> app.env && ./deploy.sh "$(cat .deployed-tag)"' + ``` + Then send a test from an email alert in the dashboard (Alerts, "Send test"). + 6. Turn on the copies to Object Storage: the namespace is `terraform output -raw backup_namespace` in `infra/oracle`. Then on the server: ```sh