From ad2708d774594767f37a285501d3a392e148d68a Mon Sep 17 00:00:00 2001 From: MCCarmo Date: Wed, 28 Oct 2026 21:00:23 +0000 Subject: [PATCH] Move the architecture decision records out of the repository They are kept in the maintainer's private notes from now on. The README and the runbook no longer link to them. --- .zap/rules.tsv | 2 +- README.md | 5 -- deploy/README.md | 5 +- .../0001-java-25-and-spring-boot-4.md | 33 ---------- docs/decisions/0002-modular-monolith.md | 37 ------------ docs/decisions/0003-postgresql-and-flyway.md | 30 ---------- .../0004-docker-compose-and-testcontainers.md | 31 ---------- docs/decisions/0005-store-adapters.md | 43 ------------- docs/decisions/0006-daily-collection.md | 48 --------------- docs/decisions/0007-accounts-and-jwt.md | 42 ------------- docs/decisions/0008-catalog-and-stats.md | 49 --------------- docs/decisions/0009-reviews.md | 57 ------------------ docs/decisions/0010-alerts.md | 50 ---------------- docs/decisions/0011-container-image.md | 40 ------------- docs/decisions/0012-quality-gates.md | 43 ------------- docs/decisions/0013-oracle-cloud-terraform.md | 42 ------------- docs/decisions/0014-deploy-and-operations.md | 52 ---------------- docs/decisions/0015-website.md | 55 ----------------- docs/decisions/0016-observability.md | 50 ---------------- docs/decisions/0017-offsite-backups.md | 60 ------------------- docs/decisions/0018-always-free-quotas.md | 41 ------------- .../0019-trusted-certificate-authorities.md | 45 -------------- docs/decisions/0020-security-scans.md | 49 --------------- docs/decisions/README.md | 28 --------- 24 files changed, 3 insertions(+), 934 deletions(-) delete mode 100644 docs/decisions/0001-java-25-and-spring-boot-4.md delete mode 100644 docs/decisions/0002-modular-monolith.md delete mode 100644 docs/decisions/0003-postgresql-and-flyway.md delete mode 100644 docs/decisions/0004-docker-compose-and-testcontainers.md delete mode 100644 docs/decisions/0005-store-adapters.md delete mode 100644 docs/decisions/0006-daily-collection.md delete mode 100644 docs/decisions/0007-accounts-and-jwt.md delete mode 100644 docs/decisions/0008-catalog-and-stats.md delete mode 100644 docs/decisions/0009-reviews.md delete mode 100644 docs/decisions/0010-alerts.md delete mode 100644 docs/decisions/0011-container-image.md delete mode 100644 docs/decisions/0012-quality-gates.md delete mode 100644 docs/decisions/0013-oracle-cloud-terraform.md delete mode 100644 docs/decisions/0014-deploy-and-operations.md delete mode 100644 docs/decisions/0015-website.md delete mode 100644 docs/decisions/0016-observability.md delete mode 100644 docs/decisions/0017-offsite-backups.md delete mode 100644 docs/decisions/0018-always-free-quotas.md delete mode 100644 docs/decisions/0019-trusted-certificate-authorities.md delete mode 100644 docs/decisions/0020-security-scans.md delete mode 100644 docs/decisions/README.md diff --git a/.zap/rules.tsv b/.zap/rules.tsv index 393db5d..e5c3ca0 100644 --- a/.zap/rules.tsv +++ b/.zap/rules.tsv @@ -1,6 +1,6 @@ 10003 FAIL (Vulnerable JS Library: a known-vulnerable script served by the site, such as Swagger UI's) 10063 FAIL (Permissions Policy Header Not Set: Caddy sets it on every response) -10038 WARN (Content Security Policy Header Not Set: only Swagger UI, left out of the policy on purpose, ADR 0015) +10038 WARN (Content Security Policy Header Not Set: only Swagger UI, left out of the policy on purpose because it builds its page with inline code) 10096 IGNORE (Timestamp Disclosure: numbers inside Swagger UI's bundled scripts, not timestamps) 10109 IGNORE (Modern Web Application: informational) 10111 IGNORE (Authentication Request Identified: informational, the sign-in form) diff --git a/README.md b/README.md index 8e38d47..ada0e89 100644 --- a/README.md +++ b/README.md @@ -100,8 +100,6 @@ flowchart BT - **Observability:** JSON logs with the collection run ID, Micrometer metrics with Prometheus, a Grafana dashboard kept as code, and alert rules with unit tests, delivered to Discord. -- **Decisions written down:** [20 architecture decision records](docs/decisions), from - the Java version to the trusted certificate authorities. ## Tech stack @@ -126,8 +124,6 @@ flowchart BT | CI/CD | GitHub Actions: build and tests, coverage gate, Arm image, secret scan (gitleaks), image scan (Trivy) with an SBOM and provenance attestation, deploy over SSH with health check and rollback | | Code quality | Spring Java Format (checked in the build), CodeQL, SonarQube Cloud, Dependabot, a weekly OWASP ZAP baseline scan of the live site | -Why each choice was made: [docs/decisions](docs/decisions). - ## Getting started **You need:** Java 25 and a running Docker. Maven is not needed: `./mvnw` downloads it. @@ -240,7 +236,6 @@ compose.yaml local PostgreSQL, and Mailpit for alert emails .github/workflows/live-stores.yml weekly test against the real stores infra/oracle/ Terraform for the server (Oracle Cloud Always Free) deploy/ what runs on the server, and how to operate it -docs/decisions/ architecture decision records ``` Each package has a `package-info.java` saying what belongs there. diff --git a/deploy/README.md b/deploy/README.md index 8a37d05..22bddda 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -1,7 +1,6 @@ # Production -What runs on the server, in `/opt/extpulse`, and how to operate it. Why it is built -this way: [ADR 0014](../docs/decisions/0014-deploy-and-operations.md). +What runs on the server, in `/opt/extpulse`, and how to operate it. | File | What it does | | --- | --- | @@ -154,7 +153,7 @@ To look inside a dump without touching production, restore it into another datab ### Copies off the server Every dump also goes to the bucket `extpulse-backups` in Object Storage, kept 30 days -([ADR 0017](../docs/decisions/0017-offsite-backups.md)). The server can add and read them, +(`infra/oracle/backups.tf`). The server can add and read them, not replace or delete them. A copy now, and the list in the bucket: ```sh diff --git a/docs/decisions/0001-java-25-and-spring-boot-4.md b/docs/decisions/0001-java-25-and-spring-boot-4.md deleted file mode 100644 index d406a9e..0000000 --- a/docs/decisions/0001-java-25-and-spring-boot-4.md +++ /dev/null @@ -1,33 +0,0 @@ -# 0001. Java 25 and Spring Boot 4.1 - -**Date:** 2026-10-21 · **Status:** accepted - -## Context - -The project is a back end meant to show the stack most Java jobs in Portugal ask -for. In October 2026 the Spring Initializr offers Spring Boot 4.0 and 4.1 only; the -3.x line is no longer offered for new projects. Java 25 is the latest LTS release -and is already installed on the development machine. - -## Decision - -Java 25 and Spring Boot 4.1, built with Maven through the Maven Wrapper. - -## Consequences - -- Current versions: long support ahead, and virtual threads are available for the - calls to the stores. -- Many companies still run Java 17 or 21 and Spring Boot 3. The code written here - (Spring MVC, JPA, Security) is almost the same in those versions, so the skills carry over. -- Spring Boot 4 split some starters (`spring-boot-starter-webmvc` instead of - `spring-boot-starter-web`) and moved some test classes to new packages - (`org.springframework.boot.webmvc.test.autoconfigure`). Older tutorials may show - the old names. -- The Maven Wrapper (`mvnw`) pins the Maven version in the repository, so everyone - and the CI build with the same one, without installing Maven. - -## Alternatives considered - -- **Java 21 and Spring Boot 3.5:** closer to what many companies run today, but not - offered for new projects and with less support left. -- **Gradle:** faster builds, but Maven is more common in Portuguese companies. diff --git a/docs/decisions/0002-modular-monolith.md b/docs/decisions/0002-modular-monolith.md deleted file mode 100644 index 1cb68ff..0000000 --- a/docs/decisions/0002-modular-monolith.md +++ /dev/null @@ -1,37 +0,0 @@ -# 0002. One application, split into modules by feature - -**Date:** 2026-10-21 · **Status:** accepted - -## Context - -ExtPulse has a few clear parts: talking to the stores, the daily collection, -stats, reviews, alerts, the public page and accounts. It runs for one person at -first, then for a small number of developers. - -## Decision - -One Spring Boot application (a monolith), with one package per feature: -`store`, `collection`, `stats`, `reviews`, `alerts`, `publicpage`, `auth`. -Each package owns its classes; others use it only through its public services. - -Since 2026-10-28 this is checked, not only intended: Spring Modulith treats each -top-level package as a module, and `ModularityTests` fails on a cycle between modules -or on a dependency a module's `package-info.java` doesn't allow -(`@ApplicationModule(allowedDependencies = ...)`). `store` and `auth` depend on no other -module; the others list theirs. - -## Consequences - -- One thing to build, test, deploy and debug. -- The packages can become separate services later if one part needs to scale on its own. -- Packaging by feature (not by layer: `controllers/`, `services/`, `repositories/`) - keeps everything about reviews in one place. -- A new dependency between features is a one-line change to a `package-info.java`, - visible in review, instead of an import nobody notices. - -## Alternatives considered - -- **Microservices:** separate deploys, network calls and data consistency problems, - with no benefit at this size. -- **Packages by layer:** common in tutorials, but a change to one feature touches - every folder. diff --git a/docs/decisions/0003-postgresql-and-flyway.md b/docs/decisions/0003-postgresql-and-flyway.md deleted file mode 100644 index 82fcc88..0000000 --- a/docs/decisions/0003-postgresql-and-flyway.md +++ /dev/null @@ -1,30 +0,0 @@ -# 0003. PostgreSQL, with the schema owned by Flyway - -**Date:** 2026-10-21 · **Status:** accepted - -## Context - -The data is relational (users, publishers, extensions, listings, daily snapshots, -reviews) and the most important rule is "one snapshot per listing per day", which -the database should enforce. Daily gains need window functions (`LAG`). - -## Decision - -PostgreSQL 17. The schema is created and changed only by Flyway migrations in -`src/main/resources/db/migration` (`V1__...sql`, `V2__...sql`). Hibernate runs with -`ddl-auto: validate`: it checks that the entities match the tables and never changes them. - -## Consequences - -- Every schema change is a reviewed SQL file in Git, applied the same way in - development, tests, CI and production. -- A migration that has run is never edited; a fix is a new migration. -- Unique constraints and checks live in the database, so a bug in the code can't - break them. - -## Alternatives considered - -- **`ddl-auto: update`:** quick at the start, but it guesses changes, can't drop or - rename safely and leaves no history. -- **Liquibase:** same idea as Flyway with XML/YAML changelogs; plain SQL is simpler to read. -- **MongoDB:** the data has relations and needs constraints across tables. diff --git a/docs/decisions/0004-docker-compose-and-testcontainers.md b/docs/decisions/0004-docker-compose-and-testcontainers.md deleted file mode 100644 index fbc4e91..0000000 --- a/docs/decisions/0004-docker-compose-and-testcontainers.md +++ /dev/null @@ -1,31 +0,0 @@ -# 0004. Docker Compose for development, Testcontainers for tests - -**Date:** 2026-10-21 · **Status:** accepted - -## Context - -The application needs PostgreSQL to start. Installing PostgreSQL by hand differs on -every machine, and tests against an in-memory database (H2) behave differently from -PostgreSQL (window functions, `ON CONFLICT`, types). - -## Decision - -- **Development:** `compose.yaml` describes PostgreSQL 17. With - `spring-boot-docker-compose`, `./mvnw spring-boot:run` starts it, connects to it - and stops it on exit; no datasource settings are needed. -- **Tests:** Testcontainers starts a throwaway PostgreSQL 17 for the test run - (`TestcontainersConfiguration`, `@ServiceConnection`). -- Both use the same image version. - -## Consequences - -- Docker must be running to start the app or run the tests. -- Tests run against the real database engine, so what passes in tests works in production. -- CI needs nothing extra: GitHub's Ubuntu runners have Docker. - -## Alternatives considered - -- **H2 in memory:** faster, but not PostgreSQL; the queries this project needs would - behave differently or not work. -- **A PostgreSQL installed on the machine:** works, but every developer and the CI - must set it up by hand. diff --git a/docs/decisions/0005-store-adapters.md b/docs/decisions/0005-store-adapters.md deleted file mode 100644 index befa8e3..0000000 --- a/docs/decisions/0005-store-adapters.md +++ /dev/null @@ -1,43 +0,0 @@ -# 0005. Store adapters, and reading the Chrome Web Store page - -**Date:** 2026-10-26 · **Status:** accepted - -## Context - -ExtPulse reads three stores. The VS Code Marketplace has a public but undocumented -query API. Open VSX has a documented REST API. The Chrome Web Store has no public API -for install counts or ratings; its publishing API only covers the developer's own -uploads and needs OAuth. - -## Decision - -- One `StoreSource` implementation per store, in the `store` package. They are the - only code that talks to the stores, and they return plain records (`ListingStats`, - `DiscoveredListing`), so nothing else depends on how a store answers. -- The Chrome Web Store is read from its public item page with jsoup, matching the - visible English text ("37 users", "5 out of 5", "2 ratings"), never the generated - CSS class names. If the text isn't found, the adapter throws instead of saving a zero. -- Tests use WireMock with responses recorded from the real stores. A separate test - class tagged `live` calls the real stores and runs weekly in GitHub Actions. -- Retries and request pacing belong to the daily collection (step 4), not to the - adapters: an adapter makes one attempt with a timeout and reports what happened. - -## Consequences - -- Adding a store (for example, JetBrains Marketplace) is one new class. -- The Chrome numbers depend on Google's page. A change breaks the parsing loudly: the - weekly live run fails, and the collection records the error instead of bad data. -- The Chrome Web Store can't be searched by publisher, so Chrome listings are added by - their item ID. -- The live tests found what recorded responses couldn't: the store redirects to a URL - with raw UTF-8 in it, which the JDK HTTP client can't follow, so the Chrome adapter - follows redirects itself. - -## Alternatives considered - -- **A headless browser (Playwright, Selenium):** sees the page as a person does, but - adds a browser to the deployment for three numbers that are already in the HTML. -- **Third-party stats sites:** another dependency that scrapes the same page, with its - own delays and terms. -- **Matching CSS selectors:** shorter code, but the class names are generated and change - without notice. diff --git a/docs/decisions/0006-daily-collection.md b/docs/decisions/0006-daily-collection.md deleted file mode 100644 index 0d8e647..0000000 --- a/docs/decisions/0006-daily-collection.md +++ /dev/null @@ -1,48 +0,0 @@ -# 0006. The daily collection - -**Date:** 2026-10-26 · **Status:** accepted - -## Context - -Once a day, ExtPulse asks the stores for every listing's numbers and saves a -snapshot. Stores fail now and then, the job can run twice (a retry, a manual run, a -restart), and a day is a day in Lisbon, not in UTC. - -## Decision - -- **Idempotent writes.** Snapshots are saved with `INSERT ... ON CONFLICT (listing_id, - taken_on) DO UPDATE`. Running the collection again on the same day updates that - day's numbers; it never adds a row or fails on the unique constraint. -- **Isolated failures.** Each listing is collected on its own. A failure is recorded in - `collection_run_errors` and the run moves on. A run ends as `SUCCEEDED`, `PARTIAL` or - `FAILED`. -- **Retries and pacing** with Resilience4j, used directly (the `retry` and - `ratelimiter` modules, no annotations): up to 4 attempts with waits of 1 s, 4 s and - 16 s, and at most one request per second to each store. A listing the store doesn't - have is not retried. -- **No transaction around the run.** Store calls can take seconds; each snapshot and - the run record are saved in their own short transactions, so no database - connection is held during network calls. -- **JPA for entities, SQL where SQL is the point.** `Listing` and `CollectionRun` are - JPA entities; the upsert is plain SQL through `JdbcClient`. -- **One `Clock` bean** in `extpulse.zone` (Europe/Lisbon) decides what "today" is, and - the schedule runs in the same zone. -- A new `catalog` package holds publishers, extensions and listings, which several - packages read. - -## Consequences - -- Retrying a whole day is safe: run it again. -- A run cut short (the app stopped) stays `RUNNING`; that is visible in the table. -- With more than one instance, each would run the job. ShedLock (a lock in the - database) comes with the deploy that needs it. - -## Alternatives considered - -- **Resilience4j's Spring Boot starter with `@Retry` annotations:** less code, but it - works through proxies (a call from inside the same class silently skips it) and - ties the version to Spring Boot's; two small objects are easier to read and to test. -- **Spring Framework 7's own `@Retryable`:** new and built in, but it has no rate - limiter, and Resilience4j is what most teams already use. -- **Errors as a JSON column:** one table less, but "which listings failed this week" - becomes JSON querying instead of a join. diff --git a/docs/decisions/0007-accounts-and-jwt.md b/docs/decisions/0007-accounts-and-jwt.md deleted file mode 100644 index 550a6f7..0000000 --- a/docs/decisions/0007-accounts-and-jwt.md +++ /dev/null @@ -1,42 +0,0 @@ -# 0007. Accounts and stateless login with self-issued JWTs - -**Date:** 2026-10-26 · **Status:** accepted - -## Context - -Every resource under `/api` belongs to an account (publishers have a `user_id`), and -nothing should be deployed without login. The clients are scripts, a future web page -and READMEs (the badge, which stays public). There is one application and no -existing identity provider. - -## Decision - -- Accounts live in `users`, with the password stored as a BCrypt hash through - Spring Security's `DelegatingPasswordEncoder` (`{bcrypt}...`). -- `POST /api/auth/login` returns a JWT signed by the app with HS256. Every other - `/api` request sends it as `Authorization: Bearer ` and is checked by Spring - Security's OAuth2 resource server support (signature, expiry, issuer). No session. -- The token's `sub` is the account ID and `scope` is `user` or `user admin`; - `/api/admin/**` needs `SCOPE_admin`. Tokens last one hour. -- The signing key comes from `EXTPULSE_AUTH_JWT_SECRET` (at least 32 bytes). When it - isn't set, a random key is made at startup and a warning is logged. -- Errors are Problem Details (RFC 9457). Login answers the same 401 for a wrong - password and an unknown email, and takes the same time for both. - -## Consequences - -- Any instance can check any token without shared state, which suits scaling out. -- A token can't be revoked before it expires; the one-hour validity limits that. - A role change applies at the next login. -- No refresh tokens yet: the client logs in again after an hour. -- Changing the key logs everyone out. - -## Alternatives considered - -- **Server sessions with a cookie:** simple for a browser, but needs CSRF protection - and shared session storage across instances, and fits scripts less well. -- **An identity provider (Keycloak, Auth0, AWS Cognito):** the right call for many - apps and teams, but a second system to run for a single application. The resource - server setup used here is the same one that would validate their tokens. -- **RS256 (key pair):** needed when other services must verify tokens without being - able to issue them. Here the issuer and the verifier are the same app. diff --git a/docs/decisions/0008-catalog-and-stats.md b/docs/decisions/0008-catalog-and-stats.md deleted file mode 100644 index 950154e..0000000 --- a/docs/decisions/0008-catalog-and-stats.md +++ /dev/null @@ -1,49 +0,0 @@ -# 0008. The catalog API and how stats are computed - -**Date:** 2026-10-26 · **Status:** accepted - -## Context - -A user adds a publisher once and expects every extension it has on every store to -be tracked. The same extension appears on the Marketplace as -`MCCProgrammer.diff-checker` and on Open VSX as `MCCProgrammer/diff-checker`. Stats -must hold up against real data: a store missing on some days, a listing added weeks -after the others, and install counts that go down. - -## Decision - -- **Adding a publisher discovers everything at once.** `POST /api/publishers` asks - every store, groups the results by a store-independent key (the VS Code extension - ID in lowercase, set by each store adapter) and saves one extension with one listing - per store. The store calls run before the transaction; if any store fails, nothing - is saved (502). Chrome extensions, which can't be found by publisher, are added with - `POST /api/extensions` and `POST /api/extensions/{id}/listings` by item ID, after - checking that the store has it. -- **Stats are computed on read, in SQL,** from `snapshots`. There are no - pre-aggregated tables to keep in sync. Each lookup is "the latest snapshot of a - listing on or before a day", which the `(listing_id, taken_on)` index answers directly. -- **A missing day carries the last known number** (`JOIN LATERAL ... taken_on <= day - ORDER BY taken_on DESC LIMIT 1`), so a failed collection doesn't show as a drop. -- **Gains are per listing, then summed** (`LAG(...) OVER (PARTITION BY listing_id)`). - A listing's first day has no gain, so adding Open VSX later doesn't count all its - installs as one day's growth. Gains can be negative. -- **Weeks are ISO weeks** (Monday first), grouped from the daily series in Java. -- **Another user's extension answers 404**, the same as a missing one. - -## Consequences - -- One account per listing: `listings` is unique by store and ID, so two accounts can't - track the same extension (409). Sharing would need listings separate from their - owners, which isn't needed while each developer tracks their own extensions. -- Stats only exist from the first collection on; there is no backfill of history. -- A query per request: if the overview gets slow with many users, a daily summary - table filled by the collection is the next step. - -## Alternatives considered - -- **Pre-aggregated daily totals per extension:** faster reads, but one more thing to - keep correct when a listing is added or a day is re-collected. -- **Computing gains in Java:** possible, but window functions do it in one pass where - the data is, and it's a good SQL showcase. -- **Treating a missing day as zero or skipping it:** zero shows false drops; skipping - makes the next day's gain cover two days. diff --git a/docs/decisions/0009-reviews.md b/docs/decisions/0009-reviews.md deleted file mode 100644 index 69970cb..0000000 --- a/docs/decisions/0009-reviews.md +++ /dev/null @@ -1,57 +0,0 @@ -# 0009. Reviews: one inbox for every store - -**Date:** 2026-10-26 · **Status:** accepted - -## Context - -A 1-star review on one store is easy to miss when each store shows only its own. -The stores differ: the Marketplace has review IDs and publisher replies; Open VSX has -neither, and allows one review per user per extension; the Chrome Web Store shows -reviews on a separate page, without a stable ID per review. - -## Decision - -- Reviews are collected with the daily snapshots, by a new `fetchReviews` on every - store adapter (the latest 100 per listing), and saved in a `reviews` table. -- A review is identified by `(listing_id, external_id)`: the review ID on the - Marketplace, the author (`github:login`) on Open VSX. Saving is an upsert: a review - seen again is updated (text, stars, reply), never duplicated, and keeps - `first_seen_at`, the day ExtPulse first saw it, which alerts will use. -- The Marketplace query uses the default filter. `filterOptions=1`, used in the first - plan, leaves out ratings without text, and a 1-star without a comment is exactly the - review that matters. -- `read_at` is the user's: `PATCH /api/reviews/{id}` sets it. A review whose stars or - text change becomes unread again. -- `GET /api/reviews` filters by unread, unanswered (reviews without a reply, on the - stores that have replies: not Open VSX), stars, store and extension, newest first, - with `limit`/`offset` and a total. -- Chrome Web Store reviews (added 2026-10-28) come from the item's reviews page, which - carries them as JSON for its scripts (`AF_initDataCallback`), each with a review ID, - stars, text, version, last-edit time and the developer's reply. The page lists the 10 - latest reviews in its language, so `extpulse.stores.chrome-web-store-review-languages` - (English and Portuguese) is one request each. Ratings without a comment aren't listed - by the store. -- If reading reviews fails, the day's snapshot is still saved; the listing is counted as - failed with a `reviews:` message. - -## Consequences - -- One more request per listing per day, paced like the others; a run takes about twice - as long (12 listings: about 12 s). -- A deleted review stays in the table as it was last seen; the store's deletions are - not mirrored yet. -- Only the latest 100 reviews per listing are read each day (10 per language on the - Chrome Web Store), which is enough unless a listing gets more than that in a day. - Reviews in a language that isn't configured are not read. -- The Chrome Web Store adds one request per configured language per listing. - -## Alternatives considered - -- **Reading Chrome reviews from the visible text:** there were no stars and no review ID - in it, so the key would have been author + date, which breaks when a review is - edited. The JSON the page carries has both. -- **The store's internal data API (`batchexecute`):** returns 100 reviews per call, but - it is an undocumented endpoint for the site's own scripts; the page is what any - visitor gets. -- **A JPA entity for reviews:** the writes are upserts and the reads are filtered joins - across four tables; plain SQL says both more clearly. diff --git a/docs/decisions/0010-alerts.md b/docs/decisions/0010-alerts.md deleted file mode 100644 index 842cb48..0000000 --- a/docs/decisions/0010-alerts.md +++ /dev/null @@ -1,50 +0,0 @@ -# 0010. Alerts through an outbox, by email and Discord - -**Date:** 2026-10-27 · **Status:** accepted - -## Context - -Users want to know about a bad review, a spike or a new version without opening the -app, by email or in a Discord channel. Checks run after every collection, and a -collection can run several times a day. Sending can fail (a mail server down, Discord -rate limits), and the same thing must never be announced twice. - -## Decision - -- **Rules** (`alert_rules`): a type (`LOW_RATING`, `INSTALL_SPIKE`, `NEW_VERSION`, - `COLLECTION_FAILING`, `WEEKLY_SUMMARY`), an optional threshold, a channel (`EMAIL`, - `DISCORD`) and a target. -- **Detection in SQL, words in Java.** Each check returns rows with a key naming what - they are about (`review:42`, `version:7:0.3.1`, `spike:7:2026-10-26`, - `summary:2026-10-19`). Checks look back further than one run, so a check that didn't - run is made up for. -- **An outbox** (`alert_events`, unique per rule and key): an alert is written first - and sent after. A repeated detection hits the unique key and adds nothing; a failed - send stays unsent and is retried every 5 minutes, up to 5 attempts. -- **Decoupled from the collection:** the collection publishes a `CollectionFinished` - application event; the alerts package listens. The collection doesn't know alerts exist. -- **Email** through Spring Mail (Mailpit in development, any SMTP in production). - **Discord** through webhooks: the URL must start with `https://discord.com/api/webhooks/` - (or `discordapp.com`), checked when saving and again when sending, because the server - calls a URL a user typed; messages go with `allowed_mentions: {parse: []}`, so a - review saying `@everyone` pings no one; the webhook token is hidden in API answers. -- **A mail server that is down doesn't make the app unhealthy** (`management.health.mail.enabled: false`). - -## Consequences - -- Alerts arrive right after the collection that found them, and at most once per rule - and key, even if collections or checks run again. -- Delivery is at least once: if the app stops between sending and recording it, that - alert can go out twice. Rare, and acceptable for notifications. -- With more than one instance, two could send the same pending alert; claiming rows - with `SELECT ... FOR UPDATE SKIP LOCKED` is the next step then. - -## Alternatives considered - -- **Sending inside the check:** simpler, but a failed send is lost and a repeated check - sends again. -- **A message broker (Kafka, RabbitMQ):** the right tool between services; inside one - application the outbox table does the job with the database already there. The v3 - plan moves these events to Kafka. -- **Slack:** same idea as Discord (incoming webhooks); Discord came first because it's - what the developers this is for use. diff --git a/docs/decisions/0011-container-image.md b/docs/decisions/0011-container-image.md deleted file mode 100644 index af923a3..0000000 --- a/docs/decisions/0011-container-image.md +++ /dev/null @@ -1,40 +0,0 @@ -# 0011. The container image: multi-stage, layered, Arm, on GHCR - -**Date:** 2026-10-27 · **Status:** accepted - -## Context - -ExtPulse will run on an Oracle Cloud Always Free virtual machine (Ampere A1, Arm) with -Docker Compose, at no cost. The image must be small to update, safe to run on a -server on the internet, and configured without rebuilding it. - -## Decision - -- **A multi-stage Dockerfile:** the first stage compiles with the JDK and the Maven - Wrapper; the second has only the JRE (`eclipse-temurin:25-jre-noble`) and the app. - `docker build .` from a clean checkout is enough; tests run in CI before. -- **Layers by how often they change:** the jar is extracted with Spring Boot's - `jarmode=tools` into dependencies, loader, snapshot dependencies and application. A - code change replaces the application layer (under 1 MB), not the libraries (68 MB). -- **A user without privileges** (uid 10001), and the heap sized from the container's - memory limit (`-XX:MaxRAMPercentage=75`); at 512 MB it uses about 300 MB. -- **Arm only for now**, built natively on GitHub's Arm runner (free for public - repositories), without emulation. -- **GitHub Container Registry**, next to the code: every build of `main` is pushed as - `sha-` and `latest`; pull requests build without pushing. -- **All configuration from environment variables**; no secret is baked into the image. - -## Consequences - -- A deploy names an exact image (`sha-…`), so rolling back is running the previous tag. -- An x86 server would need a second platform in the build (`linux/amd64`). -- The base image tags follow Java 25's latest update, so rebuilding picks up security fixes. - -## Alternatives considered - -- **Cloud Native Buildpacks (`spring-boot:build-image`):** no Dockerfile to maintain, - but less visible and harder to adjust; a Dockerfile is what most teams read. -- **Building the jar on the runner and copying it in:** faster in CI, but the image - would then depend on what the runner built, instead of building anywhere the same way. -- **Docker Hub:** works the same, but needs another account and token; GHCR uses the - repository's own `GITHUB_TOKEN`. diff --git a/docs/decisions/0012-quality-gates.md b/docs/decisions/0012-quality-gates.md deleted file mode 100644 index d3b8d77..0000000 --- a/docs/decisions/0012-quality-gates.md +++ /dev/null @@ -1,43 +0,0 @@ -# 0012. Quality gates: formatting, static analysis, secrets and dependencies - -**Date:** 2026-10-27 · **Status:** accepted - -## Context - -The repository is public and changes land through pull requests checked by CI. Tests -and the coverage gate (JaCoCo) say the code works; they don't say it -is consistently formatted, free of known security patterns, free of committed secrets, -or on dependencies without known vulnerabilities. Every tool here is free for public -repositories. - -## Decision - -- **Spring Java Format**, checked in the `validate` phase: unformatted code fails the - build; `./mvnw spring-javaformat:apply` fixes it. The formatting commit is listed in - `.git-blame-ignore-revs`. -- **CodeQL** on every pull request, on `main` and weekly, for the Java code and the - GitHub Actions workflows (`security-and-quality` queries, no build needed). -- **gitleaks** in CI over the whole history, as a pinned binary checked against its - published SHA-256. -- **SonarQube Cloud** (sonarcloud.io) analysis with JaCoCo coverage, run by CI once the - `SONAR_TOKEN` secret exists. -- **Dependabot** weekly for Maven, GitHub Actions, the Dockerfile's base images and - compose.yaml, with minor and patch updates grouped. -- **Branch protection on `main`:** changes through pull requests, with CI passing; no - force pushes or deletion. - -## Consequences - -- A pull request shows formatting, tests, coverage, image build, secrets and CodeQL - before it can merge. -- One reformat touched 59 files once; after that, diffs only show real changes. -- Dependabot opens a few pull requests a week that need merging. - -## Alternatives considered - -- **Spotless with Palantir or Google Java Format:** widely used too, but they use - spaces; switching from tabs would rewrite every line. Spring Java Format matches the - code's Spring style. -- **Checkstyle:** rules on top of formatting (naming, imports); can come later. -- **The gitleaks GitHub Action:** convenient, but needs a licence for organizations; the - binary does the same with nothing to trust beyond the checksum. diff --git a/docs/decisions/0013-oracle-cloud-terraform.md b/docs/decisions/0013-oracle-cloud-terraform.md deleted file mode 100644 index ac5f1ef..0000000 --- a/docs/decisions/0013-oracle-cloud-terraform.md +++ /dev/null @@ -1,42 +0,0 @@ -# 0013. Infrastructure on Oracle Cloud Always Free, with Terraform - -**Date:** 2026-10-27 · **Status:** accepted - -## Context - -ExtPulse must run all the time (the collection runs at 06:00) and should cost nothing. -Free tiers that sleep (Render) would miss the daily run, and their free databases -expire; AWS gives credits for six months, then the account closes. Oracle Cloud's -Always Free tier includes Arm (Ampere A1) virtual machines for the life of the account. - -## Decision - -- **Oracle Cloud Always Free, home region Madrid** (`eu-madrid-1`, the closest to Portugal). -- **Everything in Terraform** (`infra/oracle`): a VCN with one public subnet, an internet - gateway, a security list that lets in only TCP 22, 80 and 443, one - `VM.Standard.A1.Flex` instance (1 OCPU, 6 GB, 50 GB boot volume, Ubuntu 24.04 Arm), - and a budget with an alert on the first cent charged. -- **cloud-init** for the first boot: Docker, fail2ban, unattended upgrades, a `deploy` - user for CI, SSH without passwords or root, and the host firewall opened for the web - (Oracle's Ubuntu images reject all but SSH in iptables). -- **1 OCPU and 6 GB, not the whole free allowance:** Oracle may reclaim free instances - that look idle (CPU, network and memory under 20% for 7 days); the app, PostgreSQL and - monitoring keep 6 GB above that line. -- **Credentials stay out of the repository:** the provider reads `~/.oci/config` (an API - signing key); real values are in an ignored `terraform.tfvars`; the state is local. -- **CI checks formatting and validity**; plan and apply run from the maintainer's machine. - -## Consequences - -- The server can be rebuilt from code; the data needs its own backups. -- Free A1 capacity is limited: creating the instance can fail with "Out of host - capacity" and has to be retried. Upgrading the account to Pay As You Go usually ends - that, while Always Free resources stay free. -- Local state means one maintainer; remote state (Object Storage) comes with a second one. - -## Alternatives considered - -- **Creating the server by hand in the console:** quicker once, but undocumented and - not repeatable. -- **AWS (EC2 or ECS + RDS):** the most asked-for cloud, planned for v2 on the six-month credits. -- **The AMD micro shape (1 GB):** also Always Free, too small for the JVM and PostgreSQL. diff --git a/docs/decisions/0014-deploy-and-operations.md b/docs/decisions/0014-deploy-and-operations.md deleted file mode 100644 index 05c54c6..0000000 --- a/docs/decisions/0014-deploy-and-operations.md +++ /dev/null @@ -1,52 +0,0 @@ -# 0014. Deploy and operations: Compose on the server, deploys from CI - -**Date:** 2026-10-27 · **Status:** accepted - -## Context - -The server (ADR 0013) is one Arm VM. It has to run the app, PostgreSQL and HTTPS, -take a new version from every merge to `main` without anyone logging in, recover -from a bad version on its own, and keep copies of the data. The API is public, but -the accounts on this server are only the maintainer's. - -## Decision - -- **Docker Compose in production** (`deploy/compose.yaml`, project `extpulse-production`): - the app, PostgreSQL 17 (no published port), Caddy and a backup container. Each one has - a memory limit and bounded logs (5 × 10 MB). -- **Caddy for HTTPS:** a Let's Encrypt certificate it obtains and renews itself, HTTP - redirected to HTTPS, HSTS and a few security headers, and the app behind it. The app - trusts its `X-Forwarded-*` headers. -- **Deploys from CI over SSH:** after the image job publishes `sha-`, the deploy - job copies `deploy/` to `/opt/extpulse` and runs `deploy.sh sha-` as the `deploy` - user, with its key and the server's host key in the `production` environment's secrets. - It runs one at a time and is skipped until the `DEPLOY_HOST` variable exists. A final - check calls `/actuator/health` through the domain. -- **Rollback by health:** `deploy.sh` waits up to about 3 minutes for the container's - health check; if it fails, it prints the logs and starts the previous tag again. -- **Secrets created on the server:** the first deploy writes `.env` (mode 600) with a - random database password and JWT key. They are not in GitHub or anywhere else. -- **Registration closed:** `extpulse.auth.registration-open=false` in production, with an - allow-list for the maintainer's email; anyone else gets 403. -- **Backups:** `pg_dump` in custom format every day at 03:00 UTC, before the collection, - into `/opt/extpulse/backups`, kept 14 days. Restoring is in `deploy/README.md`. - -## Consequences - -- A merge to `main` is in production a few minutes later, and a broken one doesn't stay. -- Deploys need a short restart of the app (seconds); with one server there is no - rolling update. -- Backups are on the same disk as the database: they help with mistakes and bad - migrations, not with losing the VM. Copies to Object Storage came next (ADR 0017). -- A database migration that the previous version can't run on is not undone by the - rollback; migrations stay additive. - -## Alternatives considered - -- **Kubernetes (k3s):** what large teams run, but for one VM and four containers it - adds more to operate than it solves. -- **Pull-based deploys (Watchtower):** no SSH key in CI, but no health check, rollback - or record of what was deployed in the pipeline. -- **Nginx and Certbot:** two pieces and a renewal cron where Caddy does both. -- **Secrets in GitHub, copied at each deploy:** more places to leak from, for values - that never need to leave the server. diff --git a/docs/decisions/0015-website.md b/docs/decisions/0015-website.md deleted file mode 100644 index 7fd9b83..0000000 --- a/docs/decisions/0015-website.md +++ /dev/null @@ -1,55 +0,0 @@ -# 0015. The website: Thymeleaf pages in the same app, signed in with a session - -**Date:** 2026-10-27 · **Status:** accepted - -## Context - -Everything worked through the REST API, which needs curl or Swagger. The plan has a -public portfolio page and README badges, and the owner needs a dashboard to read -stats and reviews and to manage alerts without crafting requests. - -## Decision - -- **Thymeleaf, rendered by the app itself:** the pages are in the same jar and image, - served by Caddy under the same domain. No separate frontend build or hosting. -- **Each feature owns its pages,** next to its API: `CatalogPages`, `DashboardPages`, - `ReviewPages`, `AlertPages`, `AdminPages`, `AccountPages` and `publicpage`. They call - the same services, which stay package-private; `StatsService.overview` and - `PublicProfiles` became public for the public pages. -- **Two security filter chains:** `/api/**` keeps stateless JWT (no session, no CSRF); - everything else signs in with a form and a session cookie (HttpOnly, SameSite=Lax, - Secure in production), with CSRF tokens in every form and a strict Content Security - Policy (only this site's files, no inline scripts or styles, no framing). Swagger UI - is left out of the policy, since it builds its page with inline code. -- **Charts are SVG computed in Java** (`StatsCharts`): a line for installs, columns - from a zero baseline for daily gains, tooltips with SVG ``, the numbers also as - a table. No JavaScript, so nothing conflicts with the policy, and the shapes are - unit-tested. -- **Public by choice:** an account has a public page and badges only after choosing an - address (`users.public_slug`, V6). Pages show installs only; reviews, alerts and the - email stay private. Badges are SVG, cached for an hour. -- **Errors as pages:** the Problem Details handler now applies only to REST - controllers; pages get `error.html`. -- **"Keep me signed in"** (added 2026-10-28): a box on the sign-in form keeps the - browser signed in for 30 days with Spring Security's persistent remember-me tokens - (`persistent_logins`, V7). Each use replaces the token; an old token coming back - means a copy exists, so every remembered login of the account ends and the visit - carries on signed out (`RememberedLogins`). Signing out deletes them. - -## Consequences - -- One deployable for API and website; a change to both ships in one PR. -- Sessions live in memory: a deploy signs everyone out, except browsers that ticked - "Keep me signed in", which sign back in on their next visit. Spring Session (JDBC) - would keep every session across restarts. -- Pages are tested with MockMvc like the API, including CSRF and redirects. - -## Alternatives considered - -- **A single-page app (React or Angular) on the API:** what many teams build, but a - second project, build and deploy, and the token would live in the browser. Kept as - an option for later; the API is ready for it. -- **JWT in a cookie for the website:** one mechanism instead of two, but then the API - would need CSRF protection too, and signing out can't revoke a token. -- **A chart library (Chart.js):** more interactive, but a script to load and allow in - the policy, for two simple charts. diff --git a/docs/decisions/0016-observability.md b/docs/decisions/0016-observability.md deleted file mode 100644 index 793872f..0000000 --- a/docs/decisions/0016-observability.md +++ /dev/null @@ -1,50 +0,0 @@ -# 0016. Observability: JSON logs, Prometheus, Grafana and Alertmanager on the server - -**Date:** 2026-10-27 · **Status:** accepted - -## Context - -ExtPulse runs unattended: the collection at 06:00, alert delivery every 5 minutes. A -failure nobody looks at (a store changing its page, the collection not running, the -app restarting in a loop) would go unnoticed for days. It should cost nothing and run -on the same VM. - -## Decision - -- **Structured logs:** in production the console logs are JSON in the Elastic Common - Schema (Spring Boot's built-in structured logging), one object per line, kept by - Docker and read with `jq`. Every line of a collection run carries its run ID (MDC). -- **Metrics with Micrometer, read by Prometheus:** the standard ones (JVM, HTTP - requests with latency histograms, the connection pool) and ExtPulse's own: - `extpulse.collection.duration` (by result), `extpulse.collection.listings` (by store - and outcome), `extpulse.collection.last.success` (read from the database, so a - restart doesn't reset it) and `extpulse.alerts.delivered` (by channel and outcome). -- **The actuator on its own port** in production (8081), not published and not - proxied. Only the containers next to the app reach the metrics; the public site has - no `/actuator` at all, and the deploy's outside check uses the home page. -- **Prometheus** (15 days of data) with six alert rules about ExtPulse itself; - **Alertmanager** sends them to Discord, with the webhook read from a file that exists - only on the server; **Grafana** has its data source and dashboard provisioned from - files in the repository. -- **No public monitoring:** Prometheus, Alertmanager and Grafana listen on the server's - localhost and are opened through an SSH tunnel. - -## Consequences - -- A collection that didn't run, a store that started failing or the app being down - reaches Discord within minutes, without anyone looking. -- About 350 MB more memory on the VM; the 6 GB keep plenty of room. -- Logs stay on the server, bounded by Docker's rotation (5 × 10 MB per service): enough - to investigate a recent problem, not to search months back. -- The dashboard is code: reviewed in PRs, the same after a rebuild. - -## Alternatives considered - -- **Grafana Cloud's free tier:** nothing to run, but metrics and logs leave the server, - and an account and token to manage. -- **Loki (or the ELK stack) for logs:** searchable history across restarts, at the cost - of another service and its storage; worth it with more than one server. -- **Grafana on a public subdomain behind Caddy:** easier to open, but a login page on - the internet to protect, for one person who already has SSH. -- **Grafana alerting instead of Alertmanager:** one fewer container, but Prometheus - rules with Alertmanager are the common pattern, and the rules live in plain files. diff --git a/docs/decisions/0017-offsite-backups.md b/docs/decisions/0017-offsite-backups.md deleted file mode 100644 index c2d748f..0000000 --- a/docs/decisions/0017-offsite-backups.md +++ /dev/null @@ -1,60 +0,0 @@ -# 0017. Backups off the server: Object Storage, write-only for the server - -**Date:** 2026-10-27 · **Status:** accepted - -## Context - -The daily `pg_dump` (ADR 0014) is on the same disk as the database. It helps with a bad -migration or a mistake, not with losing the VM, its disk or the account on it. A copy -has to live somewhere else, cost nothing, and survive whoever might take over the -server: an attacker who can delete the database could otherwise delete its copies too. - -## Decision - -- **A bucket in Oracle Object Storage** (`extpulse-backups`, private, Standard tier), - created by Terraform with the rest of the infrastructure. Always Free includes 20 GB; - a month of dumps is a few megabytes. -- **The server signs in as itself:** a dynamic group that matches the instance's OCID - and an IAM policy for it (instance principal). There is no access key on the server, - in `.env` or in GitHub, and nothing to rotate. A rebuilt server gets a new OCID and - Terraform moves the rule to it. -- **Write-only for the server:** the policy allows `OBJECT_CREATE`, `OBJECT_INSPECT` and - `OBJECT_READ` on that bucket only; not overwriting or deleting. The bucket's lifecycle - rule deletes objects after 30 days, run by the Object Storage service with its own - narrow policy. -- **rclone in its own container** (`offsite`), with its remote defined in environment - variables. It copies what the bucket doesn't have yet: when it starts (so every - deploy checks the access works) and every day at 03:30 UTC, half an hour after the - backup. A day that fails is caught up the next; an existing copy is never replaced - (`--immutable`). -- **Off until configured:** the bucket's namespace goes in the server's `.env` by hand - once (`OCI_NAMESPACE`); until then the container only logs that copies are off. - -## Consequences - -- Losing the VM loses at most the day since the last backup; the dumps of the last 30 - days are in the bucket and can be restored on a new server (`deploy/README.md`). -- A compromised server can add junk to the bucket but cannot remove or alter the - copies already there. -- The copies are encrypted at rest by Oracle, not by ExtPulse: anyone with access to the - Oracle account can read them, so its sign-in (with MFA) protects them. -- Both copies are in the same region and account. Losing the Oracle account itself - still loses everything; a second provider would cover it, at the cost of a key to - store. -- A failed copy used to show only in the `offsite` container's log. Since 2026-10-28 - `offsite.sh` pushes the time of the newest dump and of the last copy that worked to a - Pushgateway, and Prometheus alerts when either is more than 26 hours old - (`BackupLate`, `OffsiteBackupLate`). - -## Alternatives considered - -- **A customer secret key (S3-compatible API) in `.env`:** works with any S3 tool, but - it is a long-lived secret on the server, tied to a user, with that user's - permissions. -- **OCI CLI instead of rclone:** Oracle's own tool, but a far larger image, and a - loop to write by hand to copy only what is missing. -- **Block volume backups (snapshots of the disk):** Always Free allows five, but they - copy the whole disk, not a consistent database dump, and are restored as a disk. -- **Encrypting the dumps before upload (rclone crypt):** protects them from the - account too, but the passphrase must then live outside the server, or losing the - server loses the means to read its backups. diff --git a/docs/decisions/0018-always-free-quotas.md b/docs/decisions/0018-always-free-quotas.md deleted file mode 100644 index cd27796..0000000 --- a/docs/decisions/0018-always-free-quotas.md +++ /dev/null @@ -1,41 +0,0 @@ -# 0018. Quotas: the account only creates what Always Free covers - -**Date:** 2026-10-28 · **Status:** accepted - -## Context - -To get past "Out of host capacity" for the Ampere A1 instance, the account may be -upgraded to Pay As You Go. A paid account charges for anything beyond the Always Free -allowances, and Oracle Cloud has no spending cap: the budget (ADR 0013) emails on the -first cent but stops nothing. ExtPulse must cost nothing, so a mistake should fail -instead of being charged. - -## Decision - -- **A tenancy-wide quota policy** (`extpulse-always-free`, `infra/oracle/quotas.tf`), - managed by Terraform with the rest: - - compute: every core and memory quota zeroed, then Ampere A1 given back up to the free - 4 OCPUs and 24 GB, per availability domain and per region; - - block and boot volumes: the free 200 GB and 5 backups; - - Object Storage: 10 GB, half the free 20 GB; - - databases and load balancers: zero. -- **Order matters:** within a quota, a later statement overrides an earlier one, so each - family is zeroed first and the free allowance set after. -- **The budget alert stays:** quotas cover the services where a charge is likely, not - every one of Oracle's services. - -## Consequences - -- Creating a paid shape, a bigger A1, more disk or a database fails with a quota error, - whether from Terraform, the console or a stolen key. -- Growing past Always Free is a deliberate change to this file, reviewed in a PR. -- The Always Free AMD micro instances and Autonomous Databases are blocked too; ExtPulse - doesn't use them. - -## Alternatives considered - -- **Only the budget alert:** tells about a charge after it happens. -- **A function that stops resources when the budget fires:** complex, and stopping an - instance doesn't stop the charge for its disk, or for whatever else caused it. -- **A prepaid or virtual card with a low limit:** a refused payment doesn't cancel the - debt, and the upgrade's card check needs about 100 USD available. diff --git a/docs/decisions/0019-trusted-certificate-authorities.md b/docs/decisions/0019-trusted-certificate-authorities.md deleted file mode 100644 index 0a28b47..0000000 --- a/docs/decisions/0019-trusted-certificate-authorities.md +++ /dev/null @@ -1,45 +0,0 @@ -# 0019. Trusted certificate authorities follow Ubuntu's list - -**Date:** 2026-10-28 · **Status:** accepted - -## Context - -On 19 October 2026 the private keys of eight intermediate certificates issued by -eMudhra were disclosed as compromised. Anyone holding them could impersonate an HTTPS -site to a client that still trusts eMudhra's roots. Ubuntu's list had already dropped -the root concerned (emSign ECC Root CA - G3), but the newest Temurin JDK still shipped -its own list with it, and so did the images of Alertmanager, Grafana and rclone. -ExtPulse calls out over HTTPS to the stores, Discord and Oracle Cloud. - -## Decision - -- **The app image** replaces the JDK's `cacerts` with Ubuntu's trusted authorities - (`trust extract`), at build time. Each rebuild picks up the base image's security - updates. -- **Alertmanager, Grafana and the `offsite` container** read the server's own - `/etc/ssl/certs/ca-certificates.crt`, mounted read-only; unattended-upgrades keeps - it current. -- Prometheus and Caddy keep their images' lists: Prometheus 3.15.1 dropped the root - itself, and Caddy only talks to Let's Encrypt. - -## Consequences - -- A root that Ubuntu distrusts stops being trusted by ExtPulse with the next build or - restart, without waiting for each project's release. -- Compared with the JDK's list: emSign ECC Root CA - G3, Atos TrustedRoot 2011 and - Chunghwa's ePKI root are gone, five newer roots are in. Every site ExtPulse calls - still connects (checked from the image), and a collection of all 12 listings - succeeded in it. -- A change to the server's list reaches a running container at its next restart - (every deploy restarts them). -- The production Compose file now expects an Ubuntu (Debian-style) host: running it on - a Mac needs that mount removed. - -## Alternatives considered - -- **Deleting only the eMudhra entries with `keytool`:** fixes this case, and the next - one needs another edit. -- **Waiting for new JDK and image releases:** the JDK's list changes with its releases, - days or months after the distributions. -- **`USE_SYSTEM_CA_CERTS` in the Temurin image:** adds Ubuntu's authorities to the - JDK's, but removes none. diff --git a/docs/decisions/0020-security-scans.md b/docs/decisions/0020-security-scans.md deleted file mode 100644 index 2ff634d..0000000 --- a/docs/decisions/0020-security-scans.md +++ /dev/null @@ -1,49 +0,0 @@ -# 0020. Security scans: Trivy on the image, an SBOM, and ZAP on the live site - -**Date:** 2026-10-28 · **Status:** accepted - -## Context - -CodeQL reads the code and Dependabot watches the declared dependencies (ADR 0012), but -nothing looked at what actually ships: the image's operating system packages, the jars -inside it, or what the live site answers. A first ZAP scan found two real things: the -Swagger UI that springdoc brings had a DOMPurify with known vulnerabilities, and no -response set a `Permissions-Policy`. - -## Decision - -- **Trivy in the `image` job**, on the image before it is published: - - HIGH and CRITICAL findings, OS and libraries, are reported; from `main` they go to - Security, Code scanning (SARIF). - - Only a CRITICAL vulnerability with a fix, in a library the app brings, fails the - build. That one is ours to fix with a version bump; a base image waiting for its - own update doesn't block deploys. -- **An SBOM:** Trivy writes a CycloneDX list of everything in the image (the run's - artifact), and the image pushed from `main` carries BuildKit's SBOM and a provenance - attestation (which commit, which workflow). -- **OWASP ZAP baseline** every Thursday against <https://extpulse.mccprogrammer.com>: - passive, it only reads pages any visitor gets. `.zap/rules.tsv` says which findings - fail the run (known-vulnerable scripts, a missing `Permissions-Policy`), which are - warnings (Swagger UI without the CSP, ADR 0015) and which are noise. -- **What the scan found, fixed:** Swagger UI pinned to 5.33.2 (DOMPurify 3.4.16) until - springdoc brings it, and Caddy sets `Permissions-Policy` for features the site never - uses. - -## Consequences - -- A vulnerable library shipped in the image or a header lost in a Caddy change is - noticed within a week at most, and a fixable critical one never reaches production. -- The image job takes longer: Trivy's database download and three scans, about a - minute. -- The Trivy and ZAP images are pinned in the workflows, where Dependabot doesn't look; - they are updated by hand. - -## Alternatives considered - -- **Failing on every HIGH or CRITICAL:** most come from the base image and are fixed - upstream; deploys would stop for problems the project can't fix. -- **ZAP's full (active) scan:** it attacks forms and parameters, which on production - means junk accounts, alert rules and load; the baseline's passive checks are the - part that belongs on a live site. -- **Grype or Docker Scout instead of Trivy:** similar results; Trivy also writes the - SBOM and is the one most often met in CI pipelines. diff --git a/docs/decisions/README.md b/docs/decisions/README.md deleted file mode 100644 index a8bf66b..0000000 --- a/docs/decisions/README.md +++ /dev/null @@ -1,28 +0,0 @@ -# Architecture decision records - -Each file records one decision: the context, what was decided, what it costs and -what else was considered. They are numbered and never rewritten; a decision that -changes gets a new record that replaces the old one. - -| # | Decision | -| --- | --- | -| [0001](0001-java-25-and-spring-boot-4.md) | Java 25 and Spring Boot 4.1 | -| [0002](0002-modular-monolith.md) | One application, split into modules by feature | -| [0003](0003-postgresql-and-flyway.md) | PostgreSQL, with the schema owned by Flyway | -| [0004](0004-docker-compose-and-testcontainers.md) | Docker Compose for development, Testcontainers for tests | -| [0005](0005-store-adapters.md) | Store adapters, and reading the Chrome Web Store page | -| [0006](0006-daily-collection.md) | The daily collection: idempotent, isolated failures, retries and pacing | -| [0007](0007-accounts-and-jwt.md) | Accounts and stateless login with self-issued JWTs | -| [0008](0008-catalog-and-stats.md) | The catalog API and how stats are computed | -| [0009](0009-reviews.md) | Reviews: one inbox for every store | -| [0010](0010-alerts.md) | Alerts through an outbox, by email and Discord | -| [0011](0011-container-image.md) | The container image: multi-stage, layered, Arm, on GHCR | -| [0012](0012-quality-gates.md) | Quality gates: formatting, static analysis, secrets and dependencies | -| [0013](0013-oracle-cloud-terraform.md) | Infrastructure on Oracle Cloud Always Free, with Terraform | -| [0014](0014-deploy-and-operations.md) | Deploy and operations: Compose on the server, deploys from CI | -| [0015](0015-website.md) | The website: Thymeleaf pages in the same app, signed in with a session | -| [0016](0016-observability.md) | Observability: JSON logs, Prometheus, Grafana and Alertmanager on the server | -| [0017](0017-offsite-backups.md) | Backups off the server: Object Storage, write-only for the server | -| [0018](0018-always-free-quotas.md) | Quotas: the account only creates what Always Free covers | -| [0019](0019-trusted-certificate-authorities.md) | Trusted certificate authorities follow Ubuntu's list | -| [0020](0020-security-scans.md) | Security scans: Trivy on the image, an SBOM, and ZAP on the live site |