From 50ae7c127bb57ee2047192b81097f3529cafd6e5 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Tue, 13 Oct 2026 18:12:14 +0000
Subject: [PATCH 01/32] Back up the ClickHouse cluster to Cloudflare R2

Nothing backed up the production cluster: the weekly ooni-db-backup export
reads a legacy local ClickHouse (its fastpath files have been empty since
mid-2025), and scripts/dump-tables-ch.sh listed tables that no longer
exist while missing obs_*, analysis_web_measurement and others.

Add a clickhouse_backup role that runs ClickHouse's native BACKUP daily
from a systemd timer. Each table has its own chain: a full backup every
30 days and incrementals in between, so a failure in obs_web (3.9 TB)
doesn't cost the other tables their backup. The two newest complete
chains are kept; chains pinned with --keep (e.g. before an upgrade step)
are kept until released.

A host runs the jobs set in its host_vars, one per bucket:
- data3, "measurements": the replicated tables, minus the private ones
  and data1's. data3 isn't behind clickhouseproxy and doesn't run Airflow.
- data3, "private": msmt_feedback, faulty_measurements, url_priorities,
  in their own bucket, since R2 tokens grant access per bucket.
- data1, "data1": the plain MergeTree tables whose data only data1 holds
  (labeling_frames, event_detector_cusums, event_detector_v2_changepoints,
  changepoint_label). Remove a table from clickhouse_backup_data1_tables
  once it's replicated and data3 picks it up.

The R2 endpoint and token come from SSM. R2 rejects multipart uploads
whose parts (all but the last) differ in size, so the uploads use fixed
128 MiB parts (s3_upload_part_size_multiply_factor = 1). With R2's
10,000-part limit a file can reach 1.25 TiB; the largest file of a backup
is one column of one part, and parts stop growing at 150 GB.

After each table the script writes its result to a node_exporter textfile
(ooni_clickhouse_backup_*), and prometheus_node_exporter gets a
node_exporter_textfile_dir option for it. New alerts: a table's backup
failed, no backup of a table for 3 days, no results from data1 or data3
for a day, and data3's load over half its CPUs for 15 minutes. TextfileJam
skips the backup file, which changes daily rather than hourly.

Things found while testing against 24.8.6.70 and MinIO:
- BACKUP saves only the schema of EmbeddedRocksDB and Join tables, so
  their rows are exported in Native format alongside the backup.
- An incremental backup stores its base_backup argument, keys included,
  in its .backup file. ClickHouse gets the bucket keys from <s3>
  endpoints in config.d instead, which it reloads without a restart.
- max_backup_bandwidth is a query setting (BACKUP ... SETTINGS rejects
  it), and actual throughput is about a third of the value.
- A 24.8 server segfaults restoring a backup taken on 26.8, so the
  runbook pins a backup before each upgrade step.

Remove the outdated dump and restore scripts.
---
 ansible/deploy-clickhouse.yml                 |  13 +
 ansible/group_vars/clickhouse/vars.yml        |  21 +
 ansible/host_vars/data1.htz-fsn.prod.ooni.nu  |   7 +
 ansible/host_vars/data3.htz-fsn.prod.ooni.nu  |  13 +
 .../roles/clickhouse_backup/defaults/main.yml |  67 ++
 .../files/ooni_clickhouse_backup.py           | 644 ++++++++++++++++++
 .../roles/clickhouse_backup/tasks/main.yml    |  75 ++
 .../templates/clickhouse-backup.json.j2       |  19 +
 .../templates/ooni-clickhouse-backup.service  |  14 +
 .../templates/ooni-clickhouse-backup.timer    |  11 +
 .../templates/ooni_backup_s3.xml.j2           |  19 +
 .../roles/prometheus/files/alert_rules.yml    |  39 +-
 .../defaults/main.yml                         |   3 +
 .../tasks/install.yml                         |  12 +
 .../templates/node_exporter.service.j2        |   2 +-
 docs/Runbooks.md                              | 155 +++++
 scripts/dump-tables-ch.sh                     |  46 --
 scripts/restore-dumps.sh                      |  35 -
 18 files changed, 1111 insertions(+), 84 deletions(-)
 create mode 100644 ansible/roles/clickhouse_backup/defaults/main.yml
 create mode 100644 ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
 create mode 100644 ansible/roles/clickhouse_backup/tasks/main.yml
 create mode 100644 ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2
 create mode 100644 ansible/roles/clickhouse_backup/templates/ooni-clickhouse-backup.service
 create mode 100644 ansible/roles/clickhouse_backup/templates/ooni-clickhouse-backup.timer
 create mode 100644 ansible/roles/clickhouse_backup/templates/ooni_backup_s3.xml.j2
 delete mode 100755 scripts/dump-tables-ch.sh
 delete mode 100755 scripts/restore-dumps.sh

diff --git a/ansible/deploy-clickhouse.yml b/ansible/deploy-clickhouse.yml
index 630d1736..b40c3ec7 100644
--- a/ansible/deploy-clickhouse.yml
+++ b/ansible/deploy-clickhouse.yml
@@ -11,3 +11,16 @@
       tags: [node_exporter]
     - role: oonidata_clickhouse
       tags: [clickhouse]
+
+# data3 backs up the replicated tables: they hold the same data on every
+# node, and data3 isn't behind clickhouseproxy and doesn't run Airflow.
+# data1 backs up the plain MergeTree tables whose data only it holds (see
+# clickhouse_backup_data1_tables). The jobs are set in host_vars.
+- name: Deploy ClickHouse backups
+  hosts:
+    - data1.htz-fsn.prod.ooni.nu
+    - data3.htz-fsn.prod.ooni.nu
+  become: true
+  roles:
+    - role: clickhouse_backup
+      tags: [clickhouse_backup]
diff --git a/ansible/group_vars/clickhouse/vars.yml b/ansible/group_vars/clickhouse/vars.yml
index 73275001..9b5ad668 100644
--- a/ansible/group_vars/clickhouse/vars.yml
+++ b/ansible/group_vars/clickhouse/vars.yml
@@ -411,6 +411,27 @@ clickhouse_prometheus:
   asynchronous_metrics: true
   status_info: true
 
+# For textfile metrics, e.g. from clickhouse_backup
+node_exporter_textfile_dir: /var/lib/node_exporter/textfile_collector
+
+# clickhouse_backup: Cloudflare R2. The endpoint names the account, so it's
+# kept in SSM with the token's keys.
+clickhouse_backup_s3_endpoint: "{{ lookup('amazon.aws.aws_ssm', '/oonidevops/secrets/clickhouse_backup_r2_endpoint', profile='oonidevops_user_prod') }}"
+clickhouse_backup_s3_access_key_id: "{{ lookup('amazon.aws.aws_ssm', '/oonidevops/secrets/clickhouse_backup_r2_access_key_id', profile='oonidevops_user_prod') }}"
+clickhouse_backup_s3_secret_access_key: "{{ lookup('amazon.aws.aws_ssm', '/oonidevops/secrets/clickhouse_backup_r2_secret_access_key', profile='oonidevops_user_prod') }}"
+# Tables with personal or unpublished data, backed up to the private bucket
+clickhouse_backup_private_tables:
+  - msmt_feedback        # account ids and free-text comments
+  - faulty_measurements  # until its contents are reviewed
+  - url_priorities       # until it's decided whether it's published
+# Plain MergeTree tables whose data only data1 holds (2026-10-20). Remove a
+# table from here once it's replicated, and data3 picks it up.
+clickhouse_backup_data1_tables:
+  - labeling_frames
+  - event_detector_cusums
+  - event_detector_v2_changepoints
+  - changepoint_label
+
 prometheus_nginx_proxy_config:
   - location: /metrics/node_exporter
     proxy_pass: http://127.0.0.1:8100/metrics
diff --git a/ansible/host_vars/data1.htz-fsn.prod.ooni.nu b/ansible/host_vars/data1.htz-fsn.prod.ooni.nu
index 74f6d22f..a0b90b64 100644
--- a/ansible/host_vars/data1.htz-fsn.prod.ooni.nu
+++ b/ansible/host_vars/data1.htz-fsn.prod.ooni.nu
@@ -2,3 +2,10 @@ tls_cert_dir: /var/lib/dehydrated/certs
 ssl_domains:
    - "data1.htz-fsn.prod.ooni.nu"
    - "airflow.prod.ooni.io"
+
+# Only the tables whose data only data1 holds; data3 backs up the rest
+clickhouse_backup_jobs:
+  - name: data1
+    s3_bucket: ooni-clickhouse-backup
+    include_tables: "{{ clickhouse_backup_data1_tables }}"
+    exclude_tables: []
diff --git a/ansible/host_vars/data3.htz-fsn.prod.ooni.nu b/ansible/host_vars/data3.htz-fsn.prod.ooni.nu
index bf1b24a0..39c522cd 100644
--- a/ansible/host_vars/data3.htz-fsn.prod.ooni.nu
+++ b/ansible/host_vars/data3.htz-fsn.prod.ooni.nu
@@ -1,2 +1,15 @@
 non_admin_usernames: []
 clickhouse_base_path: /data/clickhouse
+
+# The replicated tables, minus the ones only data1 holds data for. Private
+# tables go to their own bucket: R2 tokens grant access per bucket, and
+# researchers may be given read access to the measurement tables.
+clickhouse_backup_jobs:
+  - name: measurements
+    s3_bucket: ooni-clickhouse-backup
+    include_tables: []
+    exclude_tables: "{{ clickhouse_backup_private_tables + clickhouse_backup_data1_tables }}"
+  - name: private
+    s3_bucket: ooni-clickhouse-backup-private
+    include_tables: "{{ clickhouse_backup_private_tables }}"
+    exclude_tables: []
diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml
new file mode 100644
index 00000000..ed9b6a9b
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/defaults/main.yml
@@ -0,0 +1,67 @@
+---
+# Back up the ooni ClickHouse database to S3-compatible object storage
+# (Cloudflare R2) with ClickHouse's native BACKUP. Deploy where the data is:
+# data3 for the replicated tables, data1 for the tables only data1 holds
+# (see deploy-clickhouse.yml and host_vars).
+# Restore and upgrade procedure: docs/Runbooks.md, "ClickHouse backup".
+
+# false stops and disables the timer, e.g. while upgrading the cluster
+clickhouse_backup_enabled: true
+# Daily, at a quiet time (UTC). A run backs up every job's tables, one job
+# after the other; a run still going (a new full backup of obs_web takes
+# more than a day) delays the next one instead of overlapping it.
+clickhouse_backup_on_calendar: "*-*-* 02:30:00"
+
+clickhouse_backup_database: ooni
+clickhouse_backup_clickhouse_url: "http://127.0.0.1:8123/"
+clickhouse_backup_clickhouse_user: "{{ clickhouse_admin_user }}"
+clickhouse_backup_clickhouse_password: "{{ clickhouse_admin_password }}"
+
+# Cloudflare R2: the endpoint names the account,
+# https://<account id>.r2.cloudflarestorage.com. One API token with Object
+# Read & Write on every bucket the jobs below use.
+clickhouse_backup_s3_endpoint: ""
+clickhouse_backup_s3_region: auto
+clickhouse_backup_s3_prefix: prod
+clickhouse_backup_s3_access_key_id: ""
+clickhouse_backup_s3_secret_access_key: ""
+
+# Query settings for ClickHouse's uploads. R2 rejects a multipart upload
+# whose parts (all but the last) differ in size, and ClickHouse grows the
+# part size as an upload goes on unless the factor is 1. R2 also allows at
+# most 10,000 parts, so with 128 MiB parts one file can reach 1.25 TiB.
+clickhouse_backup_s3_settings:
+  s3_upload_part_size_multiply_factor: 1
+  s3_min_upload_part_size: 134217728
+  s3_max_upload_part_size: 134217728
+
+# One job per bucket. A job backs up the tables matching include_tables
+# (every table if empty) minus the excludes below and its own.
+clickhouse_backup_jobs:
+  - name: measurements
+    s3_bucket: ""
+    include_tables: []
+    exclude_tables: []
+
+# A new full backup per table every this many days; incrementals in between
+clickhouse_backup_full_interval_days: 30
+# Complete chains kept per table, besides chains pinned with --keep: the
+# current one and the one before, so a month or two of restore points
+clickhouse_backup_keep_chains: 2
+# Bytes/s passed to ClickHouse as max_backup_bandwidth. On 24.8 actual
+# throughput measured at about a third of this, so 120 MB/s gives ~40 MB/s,
+# which leaves room on the 1 Gbit/s link for replication.
+clickhouse_backup_max_bandwidth: 120000000
+
+# fnmatch patterns, for every job. Materialized views' .inner tables are
+# always skipped: they're backed up with their view.
+clickhouse_backup_exclude_tables:
+  - "*_tmp"               # updater staging tables
+  - msmt_feedback_old
+  - counters_test_list    # unused materialized view, to be dropped
+  - fastpath_tuning       # experiment on data2
+
+# Results for Prometheus, read by node_exporter's textfile collector
+# (node_exporter_textfile_dir in the prometheus_node_exporter role)
+clickhouse_backup_metrics_file: "{{ node_exporter_textfile_dir | default('/var/lib/node_exporter/textfile_collector') }}/ooni_clickhouse_backup.prom"
+clickhouse_backup_state_file: /var/lib/ooni-clickhouse-backup/state.json
diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
new file mode 100644
index 00000000..a96cda7c
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
@@ -0,0 +1,644 @@
+#!/usr/bin/env python3
+"""
+Back up a ClickHouse database to S3-compatible object storage using
+ClickHouse's native BACKUP, with one incremental chain per table.
+
+Configured by /etc/ooni/clickhouse-backup.json (see the clickhouse_backup
+Ansible role). Runs from a systemd timer. The config lists one or more jobs,
+run one after the other; each job backs up a set of tables to its own bucket
+(R2 tokens grant access per bucket, so private tables get a bucket of their
+own).
+
+Layout in each bucket:
+
+    <prefix>/<database>/<table>/<chain>/<backup>/          a native backup
+    <prefix>/<database>/<table>/<chain>/<backup>/data.native.zst
+                                                           data export, for
+                                                           EmbeddedRocksDB and
+                                                           Join tables only
+    <prefix>/<database>/<table>/<chain>/KEEP-<label>       pin, never pruned
+
+<chain> and <backup> are UTC timestamps (20261013T023000Z). The first backup
+of a chain is a full backup; every later one is incremental on top of the
+previous one. A backup is complete once ClickHouse has written its .backup
+file, which it does last.
+
+Why per table: a new full backup of obs_web takes more than a day at our
+bandwidth cap, so one failure shouldn't throw away the other tables' work,
+and chains can be restarted one table at a time.
+
+Why the data export: BACKUP writes only the CREATE statement for
+EmbeddedRocksDB and Join tables (checked on 24.8.6.70), so their rows are
+exported separately in Native format.
+
+ClickHouse reads the bucket credentials from an <s3> endpoint in its config
+(config.d/ooni_backup_s3.xml); this script only uses them itself, through
+boto3, to list and delete backups.
+
+Commands:
+    run [--job J ...] [--full] [--table T ...] [--keep LABEL]
+    list [--job J ...] [--table T ...]
+    release [--job J ...] LABEL
+
+After each table, the result goes to a Prometheus textfile (metrics_file in
+the config) for node_exporter: last success and attempt per table, whether
+the last attempt failed, its duration and size, and whether a job is running.
+
+Monitor with:  sudo journalctl -f --identifier ooni-clickhouse-backup
+"""
+
+import argparse
+import fcntl
+import fnmatch
+import json
+import logging
+import os
+import re
+import signal
+import sys
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+from datetime import datetime, timezone
+
+import boto3  # debdeps: python3-boto3
+from botocore.config import Config as BotoConfig
+
+log = logging.getLogger("ooni-clickhouse-backup")
+
+DEFAULT_CONFIG = "/etc/ooni/clickhouse-backup.json"
+TS_FMT = "%Y%m%dT%H%M%SZ"
+SAFE_NAME = re.compile(r"^[A-Za-z0-9_]+$")
+# BACKUP stores only the schema of these engines, so their rows are exported
+EXPORT_ENGINES = {"EmbeddedRocksDB", "Join"}
+
+
+def utcnow() -> datetime:
+    return datetime.now(timezone.utc)
+
+
+def ts(dt: datetime) -> str:
+    return dt.strftime(TS_FMT)
+
+
+def parse_ts(s: str) -> datetime:
+    return datetime.strptime(s, TS_FMT).replace(tzinfo=timezone.utc)
+
+
+def size(n: int) -> str:
+    for unit in ("B", "KiB", "MiB", "GiB"):
+        if n < 1024:
+            return f"{n:.0f} {unit}" if unit == "B" else f"{n:.1f} {unit}"
+        n /= 1024
+    return f"{n:.2f} TiB"
+
+
+def sql_str(s: str) -> str:
+    return "'" + s.replace("\\", "\\\\").replace("'", "\\'") + "'"
+
+
+class ClickHouse:
+    """Minimal client for the HTTP interface"""
+
+    def __init__(self, url: str, user: str, password: str):
+        self.url = url.rstrip("/") + "/"
+        self.user = user
+        self.password = password
+
+    def query(self, sql: str, settings: dict = None, timeout: float = 600) -> str:
+        url = self.url
+        if settings:
+            url += "?" + urllib.parse.urlencode(settings)
+        req = urllib.request.Request(url, data=sql.encode(), method="POST")
+        req.add_header("X-ClickHouse-User", self.user)
+        if self.password:
+            req.add_header("X-ClickHouse-Key", self.password)
+        try:
+            with urllib.request.urlopen(req, timeout=timeout) as resp:
+                return resp.read().decode()
+        except urllib.error.HTTPError as e:
+            body = e.read().decode(errors="replace").strip()
+            raise RuntimeError(f"ClickHouse error: {body}") from None
+
+    def rows(self, sql: str, **kw) -> list:
+        out = self.query(sql + "\nFORMAT JSONEachRow", **kw)
+        return [json.loads(line) for line in out.splitlines() if line.strip()]
+
+
+class Store:
+    """The backup bucket, as seen by ClickHouse and by boto3"""
+
+    def __init__(self, conf: dict):
+        self.bucket = conf["s3_bucket"]
+        self.prefix = conf["s3_prefix"].strip("/")
+        self.key_id = conf["s3_access_key_id"]
+        self.secret = conf["s3_secret_access_key"]
+        endpoint = conf["s3_endpoint"].rstrip("/")
+        # ClickHouse may reach the store under another name (e.g. in tests)
+        ch_endpoint = conf.get("clickhouse_s3_endpoint", endpoint).rstrip("/")
+        self.ch_base = f"{ch_endpoint}/{self.bucket}"
+        botoconf = dict(s3={"addressing_style": "path"})
+        try:
+            # boto3 >= 1.36 adds checksums some S3-compatible stores reject
+            cfg = BotoConfig(
+                request_checksum_calculation="when_required",
+                response_checksum_validation="when_required",
+                **botoconf,
+            )
+        except TypeError:
+            cfg = BotoConfig(**botoconf)
+        self.s3 = boto3.client(
+            "s3",
+            endpoint_url=endpoint,
+            region_name=conf.get("s3_region", "us-east-1"),
+            aws_access_key_id=self.key_id,
+            aws_secret_access_key=self.secret,
+            config=cfg,
+        )
+
+    def table_prefix(self, database: str, table: str) -> str:
+        return f"{self.prefix}/{database}/{table}/"
+
+    def ch_s3(self, key: str) -> str:
+        """S3(...) argument for BACKUP/RESTORE and the s3 function.
+
+        No credentials: ClickHouse takes them from the <s3> endpoint in its
+        own config. An incremental backup records its base_backup argument
+        verbatim in its .backup file, so keys passed here would end up in
+        the bucket in plain text."""
+        return sql_str(f"{self.ch_base}/{key}")
+
+    def subdirs(self, prefix: str) -> list:
+        """Names of the "directories" directly below prefix"""
+        names = []
+        pager = self.s3.get_paginator("list_objects_v2")
+        for page in pager.paginate(Bucket=self.bucket, Prefix=prefix, Delimiter="/"):
+            for p in page.get("CommonPrefixes", []):
+                names.append(p["Prefix"][len(prefix) :].rstrip("/"))
+            for o in page.get("Contents", []):
+                names.append(o["Key"][len(prefix) :])
+        return names
+
+    def exists(self, key: str) -> bool:
+        try:
+            self.s3.head_object(Bucket=self.bucket, Key=key)
+            return True
+        except self.s3.exceptions.ClientError as e:
+            if e.response.get("Error", {}).get("Code") in ("404", "NoSuchKey", "NotFound"):
+                return False
+            raise
+
+    def put_empty(self, key: str) -> None:
+        self.s3.put_object(Bucket=self.bucket, Key=key, Body=b"")
+
+    def delete_key(self, key: str) -> None:
+        self.s3.delete_object(Bucket=self.bucket, Key=key)
+
+    def delete_prefix(self, prefix: str) -> int:
+        assert prefix.endswith("/") and prefix.startswith(self.prefix + "/")
+        n = 0
+        pager = self.s3.get_paginator("list_objects_v2")
+        for page in pager.paginate(Bucket=self.bucket, Prefix=prefix):
+            keys = [{"Key": o["Key"]} for o in page.get("Contents", [])]
+            if keys:
+                self.s3.delete_objects(Bucket=self.bucket, Delete={"Objects": keys, "Quiet": True})
+                n += len(keys)
+        return n
+
+    def chains(self, database: str, table: str) -> dict:
+        """{chain: {"backups": {backup: complete}, "keep": [labels]}}"""
+        tprefix = self.table_prefix(database, table)
+        out = {}
+        for chain in self.subdirs(tprefix):
+            if not chain:
+                continue
+            entry = {"backups": {}, "keep": []}
+            for name in self.subdirs(f"{tprefix}{chain}/"):
+                if name.startswith("KEEP-"):
+                    entry["keep"].append(name[len("KEEP-") :])
+                elif name:
+                    marker = f"{tprefix}{chain}/{name}/.backup"
+                    entry["backups"][name] = self.exists(marker)
+            out[chain] = entry
+        return out
+
+
+def write_atomically(path: str, text: str) -> None:
+    """node_exporter must never read a half-written file"""
+    tmp = f"{path}.{os.getpid()}.tmp"
+    with open(tmp, "w") as f:
+        f.write(text)
+    os.chmod(tmp, 0o644)
+    os.replace(tmp, path)
+
+
+def label_value(s: str) -> str:
+    return s.replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n")
+
+
+class Metrics:
+    """Backup results for Prometheus, as a node_exporter textfile. The job's
+    name goes in a "backup" label: the scrape already sets "job".
+
+    The latest result of every table is kept in a JSON state file, so a run
+    that backs up some tables doesn't drop the others' metrics."""
+
+    GAUGES = [
+        ("last_success_timestamp_seconds", "last_success",
+         "When the table's last successful backup finished"),
+        ("last_attempt_timestamp_seconds", "last_attempt",
+         "When the table's last backup attempt finished"),
+        ("last_attempt_failed", "last_failed",
+         "1 if the table's last backup attempt failed"),
+        ("last_duration_seconds", "duration",
+         "How long the table's last successful backup took"),
+        ("last_size_bytes", "total_size",
+         "Size of the table's last successful backup, as system.backups reports it"),
+        ("last_files", "num_files",
+         "Files in the table's last successful backup"),
+    ]
+
+    def __init__(self, metrics_file: str = "", state_file: str = ""):
+        self.metrics_file = metrics_file
+        self.state_file = state_file
+        self.state = {"tables": {}, "running": {}}
+        if state_file and os.path.exists(state_file):
+            with open(state_file) as f:
+                self.state = json.load(f)
+
+    def table_result(self, job: str, table: str, ok: bool, st: dict = None, duration_s: float = 0) -> None:
+        entry = self.state["tables"].setdefault(f"{job}/{table}", {"job": job, "table": table})
+        now = time.time()
+        entry["last_attempt"] = now
+        entry["last_failed"] = 0 if ok else 1
+        if ok:
+            entry["last_success"] = now
+            entry["duration"] = round(duration_s, 1)
+            entry["total_size"] = int(st.get("total_size", 0))
+            entry["num_files"] = int(st.get("num_files", 0))
+        self.save()
+
+    def job_running(self, job: str, running: bool) -> None:
+        self.state["running"][job] = 1 if running else 0
+        self.save()
+
+    def render(self) -> str:
+        p = "ooni_clickhouse_backup_"
+        lines = [
+            f"# HELP {p}running 1 while the job is backing up tables",
+            f"# TYPE {p}running gauge",
+        ]
+        for job, v in sorted(self.state["running"].items()):
+            lines.append(f'{p}running{{backup="{label_value(job)}"}} {v}')
+        for name, field, help_ in self.GAUGES:
+            lines += [f"# HELP {p}{name} {help_}", f"# TYPE {p}{name} gauge"]
+            for entry in sorted(self.state["tables"].values(), key=lambda e: (e["job"], e["table"])):
+                if field in entry:
+                    labels = f'backup="{label_value(entry["job"])}",table="{label_value(entry["table"])}"'
+                    lines.append(f"{p}{name}{{{labels}}} {entry[field]}")
+        return "\n".join(lines) + "\n"
+
+    def save(self) -> None:
+        if self.state_file:
+            os.makedirs(os.path.dirname(self.state_file), exist_ok=True)
+            write_atomically(self.state_file, json.dumps(self.state, indent=1))
+        if self.metrics_file:
+            write_atomically(self.metrics_file, self.render())
+
+
+def complete_chains(chains: dict) -> list:
+    """Chains holding at least one complete backup, oldest first"""
+    return sorted(c for c, e in chains.items() if any(e["backups"].values()))
+
+
+def latest_backup(chains: dict):
+    """(chain, backup) of the newest complete backup, or (None, None)"""
+    done = complete_chains(chains)
+    if not done:
+        return None, None
+    chain = done[-1]
+    backups = sorted(b for b, ok in chains[chain]["backups"].items() if ok)
+    return chain, backups[-1]
+
+
+class BackupJob:
+    def __init__(self, conf: dict, metrics: Metrics = None):
+        self.conf = conf
+        self.name = conf.get("name", "default")
+        self.metrics = metrics or Metrics()
+        self.db = conf.get("database", "ooni")
+        self.ch = ClickHouse(
+            conf.get("clickhouse_url", "http://127.0.0.1:8123/"),
+            conf.get("clickhouse_user", "default"),
+            conf.get("clickhouse_password", ""),
+        )
+        self.store = Store(conf)
+        self.full_interval_s = float(conf.get("full_interval_days", 30)) * 86400
+        self.keep_chains = max(1, int(conf.get("keep_chains", 1)))
+        # fnmatch patterns; an empty include list means every table
+        self.include = conf.get("include_tables", [])
+        self.exclude = conf.get("exclude_tables", [])
+        self.poll_s = float(conf.get("poll_interval_seconds", 30))
+        # Query settings for the uploads, e.g. R2 needs every part of a
+        # multipart upload but the last to be the same size
+        self.settings = {k: str(v) for k, v in conf.get("s3_settings", {}).items()}
+        bw = int(conf.get("max_backup_bandwidth", 0))
+        if bw:
+            # A query setting, not a BACKUP setting: 24.8 rejects it inside
+            # BACKUP ... SETTINGS. Measured throughput is ~1/3 of this value.
+            self.settings["max_backup_bandwidth"] = str(bw)
+
+    def tables(self, only: list) -> tuple:
+        """([(name, engine)] to back up, smallest first; [missing names])"""
+        rows = self.ch.rows(
+            f"""
+            SELECT t.name AS name, t.engine AS engine,
+                   coalesce(p.bytes, 0) AS bytes
+            FROM system.tables AS t
+            LEFT JOIN (
+                SELECT table, sum(bytes_on_disk) AS bytes
+                FROM system.parts
+                WHERE active AND database = {sql_str(self.db)}
+                GROUP BY table
+            ) AS p ON p.table = t.name
+            WHERE t.database = {sql_str(self.db)} AND NOT t.is_temporary
+            ORDER BY bytes, name
+            """
+        )
+        out = []
+        for r in rows:
+            name, engine = r["name"], r["engine"]
+            if only and name not in only:
+                continue
+            if name.startswith(".inner"):
+                continue  # backed up together with its materialized view
+            if not self.includes(name):
+                continue  # another job's table
+            if self.excludes(name):
+                log.info(f"{self.name}: {name}: excluded")
+                continue
+            if not SAFE_NAME.match(name):
+                log.error(f"{name}: unsupported characters in table name, skipping")
+                continue
+            out.append((name, engine))
+        # Tables asked for with --table that are this job's but don't exist
+        missing = sorted(
+            n for n in set(only) - {n for n, _ in out} if self.includes(n) and not self.excludes(n)
+        )
+        for name in missing:
+            log.error(f"{self.name}: {name}: not found")
+        return out, missing
+
+    def includes(self, table: str) -> bool:
+        return not self.include or any(fnmatch.fnmatchcase(table, p) for p in self.include)
+
+    def excludes(self, table: str) -> bool:
+        return any(fnmatch.fnmatchcase(table, p) for p in self.exclude)
+
+    def running_backups(self) -> list:
+        """Backups still running server-side, e.g. from a killed run"""
+        rows = self.ch.rows(
+            "SELECT id, name FROM system.backups WHERE status = 'CREATING_BACKUP'"
+        )
+        return [r for r in rows if self.store.ch_base in r["name"]]
+
+    def wait(self, op_id: str, what: str) -> dict:
+        t0 = time.monotonic()
+        last_log = t0
+        while True:
+            rows = self.ch.rows(
+                f"""
+                SELECT status, error, num_files, total_size, uncompressed_size
+                FROM system.backups WHERE id = {sql_str(op_id)}
+                """
+            )
+            if not rows:
+                raise RuntimeError(f"{what}: gone from system.backups (server restart?)")
+            st = rows[0]
+            if st["status"] == "BACKUP_CREATED":
+                return st
+            if st["status"] != "CREATING_BACKUP":
+                raise RuntimeError(f"{what}: {st['status']}: {st['error']}")
+            now = time.monotonic()
+            if now - last_log >= 1800:
+                log.info(f"{what}: still running after {(now - t0) / 3600:.1f}h")
+                last_log = now
+            time.sleep(self.poll_s)
+
+    def cleanup_incomplete(self, table: str, chains: dict) -> None:
+        tprefix = self.store.table_prefix(self.db, table)
+        for chain, entry in chains.items():
+            if not any(entry["backups"].values()) and not entry["keep"]:
+                n = self.store.delete_prefix(f"{tprefix}{chain}/")
+                log.info(f"{table}: removed incomplete chain {chain} ({n} objects)")
+                continue
+            for backup, ok in entry["backups"].items():
+                if not ok:
+                    n = self.store.delete_prefix(f"{tprefix}{chain}/{backup}/")
+                    log.info(f"{table}: removed incomplete backup {chain}/{backup} ({n} objects)")
+
+    def prune(self, table: str) -> None:
+        """Keep the newest keep_chains complete chains plus pinned ones"""
+        chains = self.store.chains(self.db, table)
+        done = complete_chains(chains)
+        tprefix = self.store.table_prefix(self.db, table)
+        for chain in done[: -self.keep_chains]:
+            if chains[chain]["keep"]:
+                continue
+            n = self.store.delete_prefix(f"{tprefix}{chain}/")
+            log.info(f"{table}: pruned chain {chain} ({n} objects)")
+
+    def backup_table(self, table: str, engine: str, force_full: bool, keep: str) -> None:
+        chains = self.store.chains(self.db, table)
+        self.cleanup_incomplete(table, chains)
+        chains = self.store.chains(self.db, table)
+        base_chain, base = latest_backup(chains)
+
+        now = utcnow()
+        export = engine in EXPORT_ENGINES
+        full = (
+            force_full
+            or base is None
+            or export  # tiny tables: a fresh full backup every run
+            or (now - parse_ts(base_chain)).total_seconds() >= self.full_interval_s
+        )
+        chain = ts(now) if full else base_chain
+        name = ts(now)
+        key = f"{self.store.table_prefix(self.db, table)}{chain}/{name}"
+        kind = "full" if full else f"incremental on {base_chain}/{base}"
+        what = f"{table}: {chain}/{name} ({kind})"
+
+        if export:
+            log.info(f"{table}: exporting {engine} rows")
+            self.ch.query(
+                f"""
+                INSERT INTO FUNCTION s3({self.store.ch_s3(key + "/data.native.zst")}, 'Native')
+                SELECT * FROM `{self.db}`.`{table}`
+                """,
+                timeout=3600,
+            )
+
+        sql = f"BACKUP TABLE `{self.db}`.`{table}` TO S3({self.store.ch_s3(key)})"
+        if not full:
+            base_key = f"{self.store.table_prefix(self.db, table)}{base_chain}/{base}"
+            sql += f" SETTINGS base_backup = S3({self.store.ch_s3(base_key)})"
+        sql += " ASYNC"
+
+        log.info(f"{what}: starting")
+        t0 = time.monotonic()
+        out = self.ch.query(sql + "\nFORMAT JSONEachRow", settings=self.settings)
+        op_id = json.loads(out.splitlines()[0])["id"]
+        st = self.wait(op_id, what)
+        secs = time.monotonic() - t0
+        log.info(
+            f"{what}: done in {secs / 60:.1f} min, {st['num_files']} files, "
+            f"{size(int(st['uncompressed_size']))} new data"
+        )
+        self.metrics.table_result(self.name, table, True, st, secs)
+
+        if keep:
+            self.store.put_empty(f"{self.store.table_prefix(self.db, table)}{chain}/KEEP-{keep}")
+            log.info(f"{table}: chain {chain} pinned as {keep}")
+        self.prune(table)
+
+    def run(self, only: list, force_full: bool, keep: str) -> int:
+        running = self.running_backups()
+        if running:
+            for r in running:
+                log.error(f"backup {r['id']} is still running on the server")
+            log.error("not starting: wait for it to finish, or for the server to drop it")
+            return 1
+
+        tables, failed = self.tables(only)
+        self.metrics.job_running(self.name, True)
+        try:
+            for table, engine in tables:
+                try:
+                    self.backup_table(table, engine, force_full, keep)
+                except Exception as e:
+                    log.error(f"{self.name}: {table}: backup failed: {e}")
+                    self.metrics.table_result(self.name, table, False)
+                    failed.append(table)
+        finally:
+            self.metrics.job_running(self.name, False)
+
+        if failed:
+            log.error(f"{self.name}: failed tables: {', '.join(failed)}")
+            return 1
+        log.info(f"{self.name}: all tables backed up")
+        return 0
+
+    def list(self, only: list) -> int:
+        tables = only or sorted(
+            n for n in self.store.subdirs(f"{self.store.prefix}/{self.db}/") if n
+        )
+        for table in tables:
+            chains = self.store.chains(self.db, table)
+            for chain in sorted(chains):
+                entry = chains[chain]
+                keep = f"  KEEP: {', '.join(entry['keep'])}" if entry["keep"] else ""
+                print(f"{table}/{chain}{keep}")
+                for backup, ok in sorted(entry["backups"].items()):
+                    status = "" if ok else "  (incomplete)"
+                    print(f"    {backup}{status}")
+            chain, backup = latest_backup(chains)
+            if backup:
+                key = f"{self.store.table_prefix(self.db, table)}{chain}/{backup}"
+                print(f"  latest: {self.store.ch_base}/{key}")
+        return 0
+
+    def release(self, label: str) -> int:
+        n = 0
+        for table in self.store.subdirs(f"{self.store.prefix}/{self.db}/"):
+            if not table:
+                continue
+            for chain, entry in self.store.chains(self.db, table).items():
+                if label in entry["keep"]:
+                    tprefix = self.store.table_prefix(self.db, table)
+                    self.store.delete_key(f"{tprefix}{chain}/KEEP-{label}")
+                    n += 1
+        log.info(f"released {n} chain(s) pinned as {label}; the next run prunes them")
+        return 0
+
+
+def job_confs(conf: dict, names: list) -> list:
+    """Each job's config: the top-level keys, overridden by the job's own.
+
+    A config without "jobs" is one job named "default". A job's
+    exclude_tables add to the top-level ones."""
+    common = {k: v for k, v in conf.items() if k != "jobs"}
+    out = []
+    for job in conf.get("jobs") or [{"name": "default"}]:
+        merged = {**common, **job}
+        merged["exclude_tables"] = common.get("exclude_tables", []) + job.get("exclude_tables", [])
+        if not names or merged["name"] in names:
+            out.append(merged)
+    unknown = set(names) - {j["name"] for j in out}
+    if unknown:
+        raise SystemExit(f"unknown job(s): {', '.join(sorted(unknown))}")
+    return out
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--config", default=DEFAULT_CONFIG)
+    sub = ap.add_subparsers(dest="command", required=True)
+    p = sub.add_parser("run", help="back up every table")
+    p.add_argument("--full", action="store_true", help="start new chains")
+    p.add_argument("--table", action="append", default=[], help="only this table (repeatable)")
+    p.add_argument("--keep", help="pin the chains used by this run under LABEL")
+    p = sub.add_parser("list", help="show backups in the bucket")
+    p.add_argument("--table", action="append", default=[])
+    p = sub.add_parser("release", help="unpin chains pinned under LABEL")
+    p.add_argument("label")
+    for p in sub.choices.values():
+        p.add_argument("--job", action="append", default=[], help="only this job (repeatable)")
+    args = ap.parse_args()
+
+    logging.basicConfig(
+        stream=sys.stderr, level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s"
+    )
+    log.setLevel(logging.INFO)
+
+    for label in [getattr(args, "keep", None), getattr(args, "label", None)]:
+        if label and not SAFE_NAME.match(label.replace(".", "_").replace("-", "_")):
+            ap.error(f"invalid label: {label}")
+
+    with open(args.config) as f:
+        conf = json.load(f)
+    metrics = Metrics(conf.get("metrics_file", ""), conf.get("state_file", ""))
+    jobs = [BackupJob(j, metrics) for j in job_confs(conf, args.job)]
+
+    if args.command == "list":
+        for job in jobs:
+            job.list(args.table)
+        return 0
+
+    if args.command == "run":
+        orphans = [t for t in args.table if not any(j.includes(t) and not j.excludes(t) for j in jobs)]
+        if orphans:
+            ap.error(f"no job backs up: {', '.join(orphans)}")
+
+    # Exit cleanly on SIGTERM; a BACKUP already sent keeps running
+    # server-side, and the next run waits for it (see running_backups).
+    signal.signal(signal.SIGTERM, lambda *_: sys.exit(1))
+    lock_path = conf.get("lock_file", "/run/lock/ooni-clickhouse-backup.lock")
+    with open(lock_path, "w") as lock:
+        try:
+            fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
+        except BlockingIOError:
+            log.error(f"another instance holds {lock_path}")
+            return 1
+        rc = 0
+        for job in jobs:
+            if args.command == "run":
+                # A failed job doesn't stop the next one
+                rc |= job.run(args.table, args.full, args.keep)
+            else:
+                rc |= job.release(args.label)
+        return rc
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/ansible/roles/clickhouse_backup/tasks/main.yml b/ansible/roles/clickhouse_backup/tasks/main.yml
new file mode 100644
index 00000000..77afea76
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tasks/main.yml
@@ -0,0 +1,75 @@
+---
+- name: Check backup bucket settings
+  ansible.builtin.assert:
+    that:
+      - clickhouse_backup_s3_endpoint | length > 0
+      - clickhouse_backup_s3_access_key_id | length > 0
+      - clickhouse_backup_s3_secret_access_key | length > 0
+      - clickhouse_backup_jobs | length > 0
+      - clickhouse_backup_jobs | rejectattr('s3_bucket', 'truthy') | list | length == 0
+      - clickhouse_backup_jobs | map(attribute='name') | unique | list | length == clickhouse_backup_jobs | length
+    fail_msg: set clickhouse_backup_s3_endpoint, the bucket credentials, and an s3_bucket and a unique name for every job
+
+- name: Create the directories for the backup state and the node_exporter textfile
+  ansible.builtin.file:
+    path: "{{ item }}"
+    state: directory
+    owner: root
+    group: root
+    mode: "0755"
+  loop:
+    - "{{ clickhouse_backup_state_file | dirname }}"
+    - "{{ clickhouse_backup_metrics_file | dirname }}"
+
+- name: Install dependencies for the backup script
+  ansible.builtin.apt:
+    pkg:
+      - python3-boto3
+
+- name: Give ClickHouse the backup bucket credentials
+  ansible.builtin.template:
+    src: ooni_backup_s3.xml.j2
+    dest: /etc/clickhouse-server/config.d/ooni_backup_s3.xml
+    owner: clickhouse
+    group: clickhouse
+    mode: "0600"
+
+- name: Ensure /etc/ooni exists
+  ansible.builtin.file:
+    path: /etc/ooni
+    state: directory
+    mode: "0755"
+
+- name: Configure the backup script
+  ansible.builtin.template:
+    src: clickhouse-backup.json.j2
+    dest: /etc/ooni/clickhouse-backup.json
+    owner: root
+    group: root
+    mode: "0600"
+
+- name: Install the backup script
+  ansible.builtin.copy:
+    src: ooni_clickhouse_backup.py
+    dest: /usr/local/bin/ooni-clickhouse-backup
+    owner: root
+    group: root
+    mode: "0755"
+
+- name: Install the backup service and timer
+  ansible.builtin.template:
+    src: "{{ item }}"
+    dest: "/etc/systemd/system/{{ item }}"
+    owner: root
+    group: root
+    mode: "0644"
+  loop:
+    - ooni-clickhouse-backup.service
+    - ooni-clickhouse-backup.timer
+
+- name: Set the backup timer state
+  ansible.builtin.systemd:
+    name: ooni-clickhouse-backup.timer
+    daemon_reload: true
+    state: "{{ 'started' if clickhouse_backup_enabled else 'stopped' }}"
+    enabled: "{{ clickhouse_backup_enabled }}"
diff --git a/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2 b/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2
new file mode 100644
index 00000000..da0413f6
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2
@@ -0,0 +1,19 @@
+{{ {
+  "clickhouse_url": clickhouse_backup_clickhouse_url,
+  "clickhouse_user": clickhouse_backup_clickhouse_user,
+  "clickhouse_password": clickhouse_backup_clickhouse_password,
+  "database": clickhouse_backup_database,
+  "s3_endpoint": clickhouse_backup_s3_endpoint,
+  "s3_region": clickhouse_backup_s3_region,
+  "s3_prefix": clickhouse_backup_s3_prefix,
+  "s3_access_key_id": clickhouse_backup_s3_access_key_id,
+  "s3_secret_access_key": clickhouse_backup_s3_secret_access_key,
+  "s3_settings": clickhouse_backup_s3_settings,
+  "full_interval_days": clickhouse_backup_full_interval_days,
+  "keep_chains": clickhouse_backup_keep_chains,
+  "max_backup_bandwidth": clickhouse_backup_max_bandwidth,
+  "exclude_tables": clickhouse_backup_exclude_tables,
+  "metrics_file": clickhouse_backup_metrics_file,
+  "state_file": clickhouse_backup_state_file,
+  "jobs": clickhouse_backup_jobs,
+} | to_nice_json }}
diff --git a/ansible/roles/clickhouse_backup/templates/ooni-clickhouse-backup.service b/ansible/roles/clickhouse_backup/templates/ooni-clickhouse-backup.service
new file mode 100644
index 00000000..d414389d
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/templates/ooni-clickhouse-backup.service
@@ -0,0 +1,14 @@
+[Unit]
+Description=Back up ClickHouse tables to object storage
+Wants=ooni-clickhouse-backup.timer
+After=clickhouse-server.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/ooni-clickhouse-backup run
+SyslogIdentifier=ooni-clickhouse-backup
+# A new full backup of the largest tables takes more than a day
+TimeoutStartSec=infinity
+
+[Install]
+WantedBy=multi-user.target
diff --git a/ansible/roles/clickhouse_backup/templates/ooni-clickhouse-backup.timer b/ansible/roles/clickhouse_backup/templates/ooni-clickhouse-backup.timer
new file mode 100644
index 00000000..0651c78e
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/templates/ooni-clickhouse-backup.timer
@@ -0,0 +1,11 @@
+[Unit]
+Description=Back up ClickHouse tables to object storage
+Requires=ooni-clickhouse-backup.service
+
+[Timer]
+Unit=ooni-clickhouse-backup.service
+OnCalendar={{ clickhouse_backup_on_calendar }}
+Persistent=true
+
+[Install]
+WantedBy=timers.target
diff --git a/ansible/roles/clickhouse_backup/templates/ooni_backup_s3.xml.j2 b/ansible/roles/clickhouse_backup/templates/ooni_backup_s3.xml.j2
new file mode 100644
index 00000000..808c000b
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/templates/ooni_backup_s3.xml.j2
@@ -0,0 +1,19 @@
+<?xml version="1.0"?>
+<!-- {{ ansible_managed }}
+     Credentials for the backup buckets, one endpoint per bucket. BACKUP/RESTORE
+     and the s3 function use them for any URL under the endpoint, so queries
+     never carry keys: an incremental backup stores its base_backup argument
+     in plain text in the bucket. ClickHouse reloads this file without a
+     restart. -->
+<clickhouse>
+    <s3>
+{% for bucket in clickhouse_backup_jobs | map(attribute='s3_bucket') | unique %}
+        <ooni_backup_{{ loop.index }}>
+            <endpoint>{{ clickhouse_backup_s3_endpoint | regex_replace('/$', '') }}/{{ bucket }}/</endpoint>
+            <region>{{ clickhouse_backup_s3_region }}</region>
+            <access_key_id>{{ clickhouse_backup_s3_access_key_id | e }}</access_key_id>
+            <secret_access_key>{{ clickhouse_backup_s3_secret_access_key | e }}</secret_access_key>
+        </ooni_backup_{{ loop.index }}>
+{% endfor %}
+    </s3>
+</clickhouse>
diff --git a/ansible/roles/prometheus/files/alert_rules.yml b/ansible/roles/prometheus/files/alert_rules.yml
index 1e704a93..354eeb6d 100644
--- a/ansible/roles/prometheus/files/alert_rules.yml
+++ b/ansible/roles/prometheus/files/alert_rules.yml
@@ -99,9 +99,10 @@ groups:
     annotations:
       summary: 'Broken `textfile` at {{ $labels.instance }}'
 
-  # all the textfile are updated at least hourly
+  # all the textfile are updated at least hourly, except the ClickHouse
+  # backup results, which change after each backup (see ClickHouseBackupStale)
   - alert: TextfileJam
-    expr: (node_textfile_mtime - ON(instance,job) GROUP_LEFT() node_time) / -3600 > 2
+    expr: (node_textfile_mtime{file!~"ooni_clickhouse_backup.*"} - ON(instance,job) GROUP_LEFT() node_time) / -3600 > 2
     annotations:
       summary: 'Stale `textfile` at {{ $labels.instance }}'
       description: '`{{ $labels.file }}` is {{ printf "%.0f" $value }}h old.'
@@ -144,4 +145,38 @@ groups:
     expr: rate(alertmanager_notifications_failed_total[1m]) > 0
     annotations:
       summary: 'Alertmanager {{ $labels.instance }} fails {{ $labels.integration }} notifications.'
+
+# Results written by ooni-clickhouse-backup (ansible role clickhouse_backup)
+# to node_exporter's textfile directory on data1 and data3
+- name: ClickHouse backups
+  rules:
+  - alert: ClickHouseBackupFailed
+    expr: ooni_clickhouse_backup_last_attempt_failed == 1
+    annotations:
+      summary: 'ClickHouse backup of {{ $labels.table }} ({{ $labels.backup }}) failed on {{ $labels.instance }}'
+      description: 'See `journalctl --identifier ooni-clickhouse-backup` on {{ $labels.instance }}. The next daily run retries it.'
+
+  # Backups run daily; a new full backup of obs_web can take a day and a half
+  - alert: ClickHouseBackupStale
+    expr: time() - ooni_clickhouse_backup_last_success_timestamp_seconds > 3 * 86400
+    annotations:
+      summary: 'No ClickHouse backup of {{ $labels.table }} ({{ $labels.backup }}) for {{ $value | humanizeDuration }}'
+      description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on {{ $labels.instance }}.'
+
+  # The job hasn't written any result since its host was set up, or the file is gone
+  - alert: ClickHouseBackupMissing
+    expr: absent(ooni_clickhouse_backup_running{instance="data3.htz-fsn.prod.ooni.nu"}) or absent(ooni_clickhouse_backup_running{instance="data1.htz-fsn.prod.ooni.nu"})
+    for: 26h
+    annotations:
+      summary: 'No ClickHouse backup results from data1 or data3 for over a day'
+
+  # data3 takes the backups. Above half its CPUs (load per CPU > 0.5), the
+  # backup or merges are crowding the node; lower clickhouse_backup_max_bandwidth
+  # or move the backup to another replica.
+  - alert: ClickHouseBackupHostLoadHigh
+    expr: node_load5{instance="data3.htz-fsn.prod.ooni.nu"} / on (instance) count by (instance) (node_cpu_seconds_total{mode="idle", instance="data3.htz-fsn.prod.ooni.nu"}) > 0.5
+    for: 15m
+    annotations:
+      summary: 'data3 load is {{ $value | humanizePercentage }} of its CPUs'
+      description: 'Backup running: check ooni_clickhouse_backup_running on data3.'
 ...
diff --git a/ansible/roles/prometheus_node_exporter/defaults/main.yml b/ansible/roles/prometheus_node_exporter/defaults/main.yml
index 0cdc0011..bb75dd55 100644
--- a/ansible/roles/prometheus_node_exporter/defaults/main.yml
+++ b/ansible/roles/prometheus_node_exporter/defaults/main.yml
@@ -14,6 +14,9 @@ node_exporter_host: 'localhost'
 node_exporter_port: 8100
 node_exporter_htpasswd: '/etc/ooni/prometheus_passwd'
 node_exporter_options: ''
+# Directory for the textfile collector (*.prom files written by other jobs,
+# e.g. clickhouse_backup). Empty: no textfile collector directory.
+node_exporter_textfile_dir: ''
 
 node_exporter_state: started
 node_exporter_enabled: true
diff --git a/ansible/roles/prometheus_node_exporter/tasks/install.yml b/ansible/roles/prometheus_node_exporter/tasks/install.yml
index 1ff74425..6e042260 100644
--- a/ansible/roles/prometheus_node_exporter/tasks/install.yml
+++ b/ansible/roles/prometheus_node_exporter/tasks/install.yml
@@ -68,6 +68,18 @@
     - node_exporter
     - secrets
 
+- name: Create the textfile collector directory.
+  file:
+    path: "{{ node_exporter_textfile_dir }}"
+    state: directory
+    owner: root
+    group: root
+    mode: 0755
+  when: node_exporter_textfile_dir | length > 0
+  tags:
+    - prometheus_node_exporter
+    - config
+
 - name: Copy the node_exporter systemd unit file.
   template:
     src: node_exporter.service.j2
diff --git a/ansible/roles/prometheus_node_exporter/templates/node_exporter.service.j2 b/ansible/roles/prometheus_node_exporter/templates/node_exporter.service.j2
index 42cb98cc..31a822b0 100644
--- a/ansible/roles/prometheus_node_exporter/templates/node_exporter.service.j2
+++ b/ansible/roles/prometheus_node_exporter/templates/node_exporter.service.j2
@@ -4,7 +4,7 @@ Description=NodeExporter
 [Service]
 TimeoutStartSec=0
 User=node_exporter
-ExecStart={{ node_exporter_bin_path }} --web.listen-address={{ node_exporter_host }}:{{ node_exporter_port }} {{ node_exporter_options }}
+ExecStart={{ node_exporter_bin_path }} --web.listen-address={{ node_exporter_host }}:{{ node_exporter_port }} {{ node_exporter_options }}{% if node_exporter_textfile_dir %} --collector.textfile.directory={{ node_exporter_textfile_dir }}{% endif %}
 Restart={{ node_exporter_restart }}
 
 [Install]
diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index c1b14ffa..6e8a4372 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -176,6 +176,161 @@ Future implementation is tracked in:
 [Implement Grafana dashboard and alarms backup](#implement-grafana-dashboard-and-alarms-backup)&thinsp;🐞
 
 
+## ClickHouse backup runbook
+The `ooni` database on the ClickHouse cluster is backed up to Cloudflare R2
+by the `clickhouse_backup` Ansible role, using ClickHouse's native `BACKUP`.
+Each host runs the jobs set in its host_vars, one after the other:
+
+| Host | Job | Bucket | Tables |
+|---|---|---|---|
+| data3 | `measurements` | `ooni-clickhouse-backup` | every replicated table, minus the private ones and data1's |
+| data3 | `private` | `ooni-clickhouse-backup-private` | `clickhouse_backup_private_tables`: `msmt_feedback`, `faulty_measurements`, `url_priorities` |
+| data1 | `data1` | `ooni-clickhouse-backup` | `clickhouse_backup_data1_tables`: plain MergeTree tables whose data only data1 holds |
+
+data3 takes the replicated tables because it isn't behind clickhouseproxy
+and doesn't run Airflow; replicated tables hold the same data on every node.
+Private tables have their own bucket because R2 tokens grant access per
+bucket, and researchers may be given read access to the measurement tables.
+When a data1-only table becomes replicated, remove it from
+`clickhouse_backup_data1_tables` and data3 picks it up.
+
+- `ooni-clickhouse-backup.timer` runs `/usr/local/bin/ooni-clickhouse-backup run`
+  daily at 02:30 UTC.
+- Each table has its own chain: a full backup, then daily incrementals. A
+  table starts a new chain every `clickhouse_backup_full_interval_days` (30).
+  Once a new chain is complete, older chains are deleted unless pinned.
+- EmbeddedRocksDB and Join tables (`fingerprints_*`, `session_expunge`,
+  `test_groups`): `BACKUP` saves only their schema, so their rows are also
+  exported to `data.native.zst` inside each backup.
+- R2 rejects multipart uploads whose parts (all but the last) differ in
+  size, so `clickhouse_backup_s3_settings` fixes ClickHouse's part size at
+  128 MiB (`s3_upload_part_size_multiply_factor: 1`). R2 allows 10,000 parts,
+  so one backup file can reach 1.25 TiB.
+- Two complete chains are kept per table (`clickhouse_backup_keep_chains`),
+  so a month or two of restore points.
+- Bandwidth is capped with `clickhouse_backup_max_bandwidth`. On 24.8 the
+  actual rate is about a third of the cap: 120 MB/s gives ~40 MB/s. A new
+  full backup of everything (~5.6 TB) takes about 39 hours at that rate.
+
+Layout in the bucket:
+
+```
+prod/ooni/<table>/<chain>/<backup>/          native backup (first one is full)
+prod/ooni/<table>/<chain>/KEEP-<label>       pinned chain, never pruned
+```
+
+Never put S3 keys in `BACKUP`/`RESTORE`/`s3()` queries: an incremental
+backup stores its `base_backup` argument, keys included, in plain text in
+its `.backup` file. ClickHouse gets the keys from
+`/etc/clickhouse-server/config.d/ooni_backup_s3.xml`, which it reloads
+without a restart.
+
+### Setting up
+1. In the Cloudflare dashboard, create the R2 buckets `ooni-clickhouse-backup`
+   and `ooni-clickhouse-backup-private` (location hint: Western Europe), and
+   an R2 API token with Object Read & Write on both.
+2. Store the S3 endpoint (`https://<account id>.r2.cloudflarestorage.com`) and
+   the token's keys in SSM as
+   `/oonidevops/secrets/clickhouse_backup_r2_endpoint`,
+   `/oonidevops/secrets/clickhouse_backup_r2_access_key_id` and
+   `/oonidevops/secrets/clickhouse_backup_r2_secret_access_key`.
+3. Deploy node_exporter first, for its textfile directory, then the backups:
+   ```bash
+   ./play -i inventory deploy-clickhouse.yml -l data1.htz-fsn.prod.ooni.nu,data3.htz-fsn.prod.ooni.nu -t node_exporter --diff
+   ./play -i inventory deploy-clickhouse.yml -l data1.htz-fsn.prod.ooni.nu,data3.htz-fsn.prod.ooni.nu -t clickhouse_backup --diff
+   ```
+   and the alert rules with `deploy-monitoring.yml`.
+4. The timer takes the first (full) backup at its next 02:30 UTC. A full
+   backup of everything takes about 39 hours, so to have it run over a
+   weekend, deploy on Friday after 02:30 UTC. To start one now instead:
+   ```bash
+   sudo systemctl start --no-block ooni-clickhouse-backup.service
+   sudo journalctl -f --identifier ooni-clickhouse-backup
+   ```
+
+### Monitoring
+After each table, the script writes its result to
+`/var/lib/node_exporter/textfile_collector/ooni_clickhouse_backup.prom`
+(metrics `ooni_clickhouse_backup_*`, labels `backup` = job and `table`).
+Alerts, in `alert_rules.yml`:
+
+- `ClickHouseBackupFailed`: a table's last attempt failed.
+- `ClickHouseBackupStale`: a table hasn't been backed up for 3 days.
+- `ClickHouseBackupMissing`: data1 or data3 hasn't written results for a day.
+- `ClickHouseBackupHostLoadHigh`: data3's 5-minute load is over half its
+  CPUs for 15 minutes.
+
+```bash
+sudo systemctl status ooni-clickhouse-backup.service
+sudo journalctl --identifier ooni-clickhouse-backup --since yesterday
+sudo ooni-clickhouse-backup list
+```
+
+The unit fails if any table fails; the others are still backed up. If the
+script is stopped, a `BACKUP` already sent keeps running inside ClickHouse.
+The next run refuses to start until it finishes:
+
+```sql
+SELECT id, status, start_time, error FROM system.backups ORDER BY start_time DESC LIMIT 10
+```
+
+### Before upgrading ClickHouse
+A server can restore backups taken by the same or an older version, but not
+by a newer one. On 24.8, restoring a backup taken on 26.8 crashed the server
+(segfault in `restorePartFromBackup`). After an upgrade step that changes the
+on-disk format (25.8, 26.3), the only way back is restoring a backup taken
+before that step, so:
+
+1. Disable the timer with `clickhouse_backup_enabled: false` (or
+   `sudo systemctl stop ooni-clickhouse-backup.timer`) and check that
+   `system.backups` shows nothing running.
+2. Take a pinned backup:
+   ```bash
+   sudo ooni-clickhouse-backup run --keep pre-25.8.29.51
+   ```
+3. Upgrade all three nodes. Don't run backups while nodes are on different
+   versions.
+4. Re-enable the timer. Once the whole upgrade is confirmed good, start
+   new chains on the new version and unpin the old ones:
+   ```bash
+   sudo ooni-clickhouse-backup run --full
+   sudo ooni-clickhouse-backup release pre-25.8.29.51
+   ```
+
+Data written after a pinned backup isn't in it. If you roll back, that
+window has to be filled again by reprocessing those days.
+
+### Restoring
+The target server needs `ooni_backup_s3.xml` in `config.d` (the role
+installs it). `ooni-clickhouse-backup list` prints the latest backup URL of
+each table. Restoring the latest incremental restores the whole chain.
+
+1. The database isn't part of a table backup:
+   ```sql
+   CREATE DATABASE IF NOT EXISTS ooni
+   ```
+2. Restore tables before the materialized views that read from them, on
+   one node. A replicated table that still exists makes `RESTORE` fail with
+   `REPLICA_ALREADY_EXISTS`: drop it (`DROP TABLE ... SYNC`) or restore on
+   another server.
+   ```sql
+   RESTORE TABLE ooni.obs_web FROM S3('https://<account id>.r2.cloudflarestorage.com/ooni-clickhouse-backup/prod/ooni/obs_web/<chain>/<backup>')
+   ```
+3. For EmbeddedRocksDB and Join tables, load the rows too:
+   ```sql
+   INSERT INTO ooni.fingerprints_dns
+   SELECT * FROM s3('<backup URL>/data.native.zst', 'Native')
+   ```
+4. On the other replicas, create replicated tables with the same
+   `SHOW CREATE TABLE` statement; they fetch the data from the restored node.
+   Non-replicated tables whose data only data1 holds were backed up from
+   data1 (job `data1`); restore them there.
+5. Compare row counts and checksums with what you expect, e.g.
+   ```sql
+   SELECT count(), sum(cityHash64(toString(tuple(*)))) FROM ooni.obs_web
+   ```
+
+
 ## Grafana editing
 This runbook describes adding new dashboards, panels and alerts in
 [Grafana](#grafana)&thinsp;🔧
diff --git a/scripts/dump-tables-ch.sh b/scripts/dump-tables-ch.sh
deleted file mode 100755
index efd11d70..00000000
--- a/scripts/dump-tables-ch.sh
+++ /dev/null
@@ -1,46 +0,0 @@
-#!/bin/bash
-set -euxo pipefail
-# This is to be run manually on the clickhouse host to dump schemas and table dumps
-# You may want to make some tweaks to the dumping rules in order avoid dumping
-# too much data (eg. fastpath)
-# You should then scp the data over to the target host manually, by running:
-# $ scp * clickhouse-instance2:/var/lib/clickhouse/ooni-dumps/
-TABLES=(
-"fastpath"
-"jsonl"
-"url_priorities"
-"citizenlab"
-"citizenlab_flip"
-"test_groups"
-"accounts"
-"session_expunge"
-"msmt_feedback"
-"fingerprints_dns"
-"fingerprints_http"
-"asnmeta"
-"counters_test_list"
-"counters_asn_test_list"
-"incidents"
-"oonirun"
-)
-
-dump_dir="./dumps"
-current_date=$(date +%Y%m%d)
-
-# Directory to store the dumps
-mkdir -p "$dump_dir"
-
-# Iterate over each table
-for table in "${TABLES[@]}"; do
-    # Define file names for schema and data dump
-    schema_file="${dump_dir}/${current_date}-${table}_schema.sql"
-    data_file="${dump_dir}/${current_date}-${table}_dump.clickhouse"
-
-    # Dump the table schema
-    echo "[+] dumping schema $schema_file"
-    clickhouse-client --query="SHOW CREATE TABLE ${table} FORMAT TabSeparatedRaw" > "$schema_file"
-
-    # Dump the table data in ClickHouse native format
-    echo "[+] dumping table data $data_file"
-    clickhouse-client --query="SELECT * FROM ${table} INTO OUTFILE '${data_file}' FORMAT Native"
-done
diff --git a/scripts/restore-dumps.sh b/scripts/restore-dumps.sh
deleted file mode 100755
index df67126e..00000000
--- a/scripts/restore-dumps.sh
+++ /dev/null
@@ -1,35 +0,0 @@
-#!/bin/bash
-set -euxo pipefail
-# Restore schema and sampled dumps to running clickhouse instance
-# This script is to be run manually from with CWD set to contains the dumps and
-# schema files generated from `dump-tables-ch.sh`
-# See dump-tables-ch.sh for instruction on it's usage
-for schema_file in *schema.sql;do
-    cat $schema_file | clickhouse-client;
-done
-
-dump_ts="20240202"
-TABLES=(
-"jsonl"
-"url_priorities"
-"citizenlab"
-"citizenlab_flip"
-"test_groups"
-"accounts"
-"session_expunge"
-"msmt_feedback"
-"fingerprints_dns"
-"fingerprints_http"
-"asnmeta"
-"counters_test_list"
-"counters_asn_test_list"
-"incidents"
-"oonirun"
-)
-for table in "${TABLES[@]}"; do
-    echo "Restoring ${table}"
-    cat ${dump_ts}-${table}_dump.clickhouse | clickhouse-client --query="INSERT INTO ${table} FORMAT Native"
-done
-
-echo "Restoring fastpath"
-gzip -cd 20240109T1314-fastpath.clickhouse.gz | clickhouse-client --query="INSERT INTO fastpath FORMAT Native"

From de4e8a3113eec87e43261390b3069caa342f8c06 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 06:01:18 +0000
Subject: [PATCH 02/32] clickhouse_backup: start a new full backup each
 calendar month

A table started a new chain once its chain was 30 days old, counted
from when its full began. With weekly runs that gives a full every 5
weeks, and a threshold of 28 days would slip a week whenever a table's
backup started a few minutes earlier than it did in the full.

A table now starts a new chain at its first backup in each calendar
month (UTC): the 1st with daily runs, the first Sunday with weekly
ones. A full that fails is retried at the next run, as the latest
complete chain is still from the month before.
clickhouse_backup_full_interval_days is gone.
---
 ansible/roles/clickhouse_backup/defaults/main.yml        | 2 --
 .../clickhouse_backup/files/ooni_clickhouse_backup.py    | 9 +++++----
 .../templates/clickhouse-backup.json.j2                  | 1 -
 docs/Runbooks.md                                         | 3 ++-
 4 files changed, 7 insertions(+), 8 deletions(-)

diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml
index ed9b6a9b..c2165e59 100644
--- a/ansible/roles/clickhouse_backup/defaults/main.yml
+++ b/ansible/roles/clickhouse_backup/defaults/main.yml
@@ -43,8 +43,6 @@ clickhouse_backup_jobs:
     include_tables: []
     exclude_tables: []
 
-# A new full backup per table every this many days; incrementals in between
-clickhouse_backup_full_interval_days: 30
 # Complete chains kept per table, besides chains pinned with --keep: the
 # current one and the one before, so a month or two of restore points
 clickhouse_backup_keep_chains: 2
diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
index a96cda7c..9ab88fe2 100644
--- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
+++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
@@ -20,8 +20,10 @@
 
 <chain> and <backup> are UTC timestamps (20261013T023000Z). The first backup
 of a chain is a full backup; every later one is incremental on top of the
-previous one. A backup is complete once ClickHouse has written its .backup
-file, which it does last.
+previous one. A table starts a new chain at its first backup in each
+calendar month (UTC): the 1st with daily runs, the first Sunday with weekly
+ones. A backup is complete once ClickHouse has written its .backup file,
+which it does last.
 
 Why per table: a new full backup of obs_web takes more than a day at our
 bandwidth cap, so one failure shouldn't throw away the other tables' work,
@@ -334,7 +336,6 @@ def __init__(self, conf: dict, metrics: Metrics = None):
             conf.get("clickhouse_password", ""),
         )
         self.store = Store(conf)
-        self.full_interval_s = float(conf.get("full_interval_days", 30)) * 86400
         self.keep_chains = max(1, int(conf.get("keep_chains", 1)))
         # fnmatch patterns; an empty include list means every table
         self.include = conf.get("include_tables", [])
@@ -461,7 +462,7 @@ def backup_table(self, table: str, engine: str, force_full: bool, keep: str) ->
             force_full
             or base is None
             or export  # tiny tables: a fresh full backup every run
-            or (now - parse_ts(base_chain)).total_seconds() >= self.full_interval_s
+            or parse_ts(base_chain).strftime("%Y%m") != now.strftime("%Y%m")
         )
         chain = ts(now) if full else base_chain
         name = ts(now)
diff --git a/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2 b/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2
index da0413f6..bc591a0e 100644
--- a/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2
+++ b/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2
@@ -9,7 +9,6 @@
   "s3_access_key_id": clickhouse_backup_s3_access_key_id,
   "s3_secret_access_key": clickhouse_backup_s3_secret_access_key,
   "s3_settings": clickhouse_backup_s3_settings,
-  "full_interval_days": clickhouse_backup_full_interval_days,
   "keep_chains": clickhouse_backup_keep_chains,
   "max_backup_bandwidth": clickhouse_backup_max_bandwidth,
   "exclude_tables": clickhouse_backup_exclude_tables,
diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index 6e8a4372..fb3c20e5 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -197,7 +197,8 @@ When a data1-only table becomes replicated, remove it from
 - `ooni-clickhouse-backup.timer` runs `/usr/local/bin/ooni-clickhouse-backup run`
   daily at 02:30 UTC.
 - Each table has its own chain: a full backup, then daily incrementals. A
-  table starts a new chain every `clickhouse_backup_full_interval_days` (30).
+  table starts a new chain with its first backup of each calendar month
+  (UTC), which is the first Sunday of the month if the timer runs weekly.
   Once a new chain is complete, older chains are deleted unless pinned.
 - EmbeddedRocksDB and Join tables (`fingerprints_*`, `session_expunge`,
   `test_groups`): `BACKUP` saves only their schema, so their rows are also

From 1077400478d6e7e236d2b3dff28fdc5f30ee5e1a Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 06:01:25 +0000
Subject: [PATCH 03/32] clickhouse_backup: don't start a backup when the timer
 starts

Requires= in the timer unit started ooni-clickhouse-backup.service
whenever the timer started: at every boot of data1 and data3, and on
every deploy that restarts the timer. Backups now start only at
OnCalendar; Persistent=true still runs one missed while the host was
down.
---
 .../clickhouse_backup/templates/ooni-clickhouse-backup.timer     | 1 -
 1 file changed, 1 deletion(-)

diff --git a/ansible/roles/clickhouse_backup/templates/ooni-clickhouse-backup.timer b/ansible/roles/clickhouse_backup/templates/ooni-clickhouse-backup.timer
index 0651c78e..5b599270 100644
--- a/ansible/roles/clickhouse_backup/templates/ooni-clickhouse-backup.timer
+++ b/ansible/roles/clickhouse_backup/templates/ooni-clickhouse-backup.timer
@@ -1,6 +1,5 @@
 [Unit]
 Description=Back up ClickHouse tables to object storage
-Requires=ooni-clickhouse-backup.service
 
 [Timer]
 Unit=ooni-clickhouse-backup.service

From 979eba2c8f4a53dff76c07816ff18c6d178e4600 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 06:01:25 +0000
Subject: [PATCH 04/32] clickhouse_backup: alert when a job's run completes

The alerts only said when something went wrong. At the end of a run
the script now records each job's result (ooni_clickhouse_backup_last_
run_*: when it finished, how long it took, tables backed up, in full,
and failed), and ClickHouseBackupCompleted fires for an hour after a
run with no failed table.

It goes to a new Alertmanager receiver, slack-notice: one message per
job run in #ooni-bots, without the [FIRING] prefix, a "resolved"
message or email. ClickHouseBackupFailed now says that nothing was
deleted and that the next run retries the table.
---
 .../files/ooni_clickhouse_backup.py           | 31 +++++++++++++++++--
 .../roles/prometheus/files/alert_rules.yml    | 14 ++++++++-
 .../files/templates/ooni.tmpl                 |  4 +++
 .../templates/alertmanager.yml                | 11 +++++++
 docs/Runbooks.md                              |  5 ++-
 5 files changed, 61 insertions(+), 4 deletions(-)

diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
index 9ab88fe2..69a5dd70 100644
--- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
+++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
@@ -45,6 +45,8 @@
 After each table, the result goes to a Prometheus textfile (metrics_file in
 the config) for node_exporter: last success and attempt per table, whether
 the last attempt failed, its duration and size, and whether a job is running.
+At the end of a run, each job's result: when it finished, how long it took,
+and how many tables it backed up, as full backups and in all, and failed.
 
 Monitor with:  sudo journalctl -f --identifier ooni-clickhouse-backup
 """
@@ -261,6 +263,14 @@ class Metrics:
          "Files in the table's last successful backup"),
     ]
 
+    RUN_GAUGES = [
+        ("last_run_finished_timestamp_seconds", "finished", "When the job's last run finished"),
+        ("last_run_duration_seconds", "duration", "How long the job's last run took"),
+        ("last_run_tables", "tables", "Tables the job's last run backed up"),
+        ("last_run_full_backups", "fulls", "Tables the job's last run backed up in full"),
+        ("last_run_failed_tables", "failed", "Tables whose backup failed in the job's last run"),
+    ]
+
     def __init__(self, metrics_file: str = "", state_file: str = ""):
         self.metrics_file = metrics_file
         self.state_file = state_file
@@ -268,6 +278,7 @@ def __init__(self, metrics_file: str = "", state_file: str = ""):
         if state_file and os.path.exists(state_file):
             with open(state_file) as f:
                 self.state = json.load(f)
+        self.state.setdefault("runs", {})
 
     def table_result(self, job: str, table: str, ok: bool, st: dict = None, duration_s: float = 0) -> None:
         entry = self.state["tables"].setdefault(f"{job}/{table}", {"job": job, "table": table})
@@ -285,6 +296,13 @@ def job_running(self, job: str, running: bool) -> None:
         self.state["running"][job] = 1 if running else 0
         self.save()
 
+    def run_result(self, job: str, duration_s: float, tables: int, fulls: int, failed: int) -> None:
+        self.state["runs"][job] = {
+            "finished": time.time(), "duration": round(duration_s, 1),
+            "tables": tables, "fulls": fulls, "failed": failed,
+        }
+        self.save()
+
     def render(self) -> str:
         p = "ooni_clickhouse_backup_"
         lines = [
@@ -293,6 +311,10 @@ def render(self) -> str:
         ]
         for job, v in sorted(self.state["running"].items()):
             lines.append(f'{p}running{{backup="{label_value(job)}"}} {v}')
+        for name, field, help_ in self.RUN_GAUGES:
+            lines += [f"# HELP {p}{name} {help_}", f"# TYPE {p}{name} gauge"]
+            for job, run in sorted(self.state["runs"].items()):
+                lines.append(f'{p}{name}{{backup="{label_value(job)}"}} {run[field]}')
         for name, field, help_ in self.GAUGES:
             lines += [f"# HELP {p}{name} {help_}", f"# TYPE {p}{name} gauge"]
             for entry in sorted(self.state["tables"].values(), key=lambda e: (e["job"], e["table"])):
@@ -450,7 +472,8 @@ def prune(self, table: str) -> None:
             n = self.store.delete_prefix(f"{tprefix}{chain}/")
             log.info(f"{table}: pruned chain {chain} ({n} objects)")
 
-    def backup_table(self, table: str, engine: str, force_full: bool, keep: str) -> None:
+    def backup_table(self, table: str, engine: str, force_full: bool, keep: str) -> bool:
+        """Back up one table; True if that was a full backup"""
         chains = self.store.chains(self.db, table)
         self.cleanup_incomplete(table, chains)
         chains = self.store.chains(self.db, table)
@@ -502,6 +525,7 @@ def backup_table(self, table: str, engine: str, force_full: bool, keep: str) ->
             self.store.put_empty(f"{self.store.table_prefix(self.db, table)}{chain}/KEEP-{keep}")
             log.info(f"{table}: chain {chain} pinned as {keep}")
         self.prune(table)
+        return full
 
     def run(self, only: list, force_full: bool, keep: str) -> int:
         running = self.running_backups()
@@ -512,17 +536,20 @@ def run(self, only: list, force_full: bool, keep: str) -> int:
             return 1
 
         tables, failed = self.tables(only)
+        t0, done, fulls = time.monotonic(), 0, 0
         self.metrics.job_running(self.name, True)
         try:
             for table, engine in tables:
                 try:
-                    self.backup_table(table, engine, force_full, keep)
+                    fulls += self.backup_table(table, engine, force_full, keep)
+                    done += 1
                 except Exception as e:
                     log.error(f"{self.name}: {table}: backup failed: {e}")
                     self.metrics.table_result(self.name, table, False)
                     failed.append(table)
         finally:
             self.metrics.job_running(self.name, False)
+        self.metrics.run_result(self.name, time.monotonic() - t0, done, fulls, len(failed))
 
         if failed:
             log.error(f"{self.name}: failed tables: {', '.join(failed)}")
diff --git a/ansible/roles/prometheus/files/alert_rules.yml b/ansible/roles/prometheus/files/alert_rules.yml
index 354eeb6d..d008c94d 100644
--- a/ansible/roles/prometheus/files/alert_rules.yml
+++ b/ansible/roles/prometheus/files/alert_rules.yml
@@ -154,7 +154,19 @@ groups:
     expr: ooni_clickhouse_backup_last_attempt_failed == 1
     annotations:
       summary: 'ClickHouse backup of {{ $labels.table }} ({{ $labels.backup }}) failed on {{ $labels.instance }}'
-      description: 'See `journalctl --identifier ooni-clickhouse-backup` on {{ $labels.instance }}. The next daily run retries it.'
+      description: 'Nothing was deleted: the older backups of {{ $labels.table }} are kept. The next run retries it. See `journalctl --identifier ooni-clickhouse-backup` on {{ $labels.instance }}.'
+
+  # A job's run finished with every table backed up. One Slack message per
+  # job run, no email and no "resolved" message: Alertmanager routes it to
+  # slack-notice. It clears an hour after the run.
+  - alert: ClickHouseBackupCompleted
+    expr: |
+      ooni_clickhouse_backup_last_run_duration_seconds
+        and on (instance, backup) (time() - ooni_clickhouse_backup_last_run_finished_timestamp_seconds < 3600)
+        and on (instance, backup) ooni_clickhouse_backup_last_run_failed_tables == 0
+    annotations:
+      summary: 'ClickHouse backup {{ $labels.backup }} on {{ $labels.instance }} completed in {{ $value | humanizeDuration }}'
+      description: '{{ with printf `ooni_clickhouse_backup_last_run_tables{instance="%s",backup="%s"}` $labels.instance $labels.backup | query }}{{ . | first | value }}{{ end }} tables backed up, {{ with printf `ooni_clickhouse_backup_last_run_full_backups{instance="%s",backup="%s"}` $labels.instance $labels.backup | query }}{{ . | first | value }}{{ end }} of them in full. Nothing to do.'
 
   # Backups run daily; a new full backup of obs_web can take a day and a half
   - alert: ClickHouseBackupStale
diff --git a/ansible/roles/prometheus_alertmanager/files/templates/ooni.tmpl b/ansible/roles/prometheus_alertmanager/files/templates/ooni.tmpl
index c8ad084a..1268b5f2 100644
--- a/ansible/roles/prometheus_alertmanager/files/templates/ooni.tmpl
+++ b/ansible/roles/prometheus_alertmanager/files/templates/ooni.tmpl
@@ -4,3 +4,7 @@
 {{ define "slack.ooni.title" }}
 {{ range $i, $el := .Alerts }}{{ if gt $i 0 }}
 {{ end }}{{ if eq $el.Status "resolved" }}[RESOLVED] {{ else }}[FIRING] {{ end }}{{ $el.Annotations.summary }}{{ end }}{{ end }}
+
+{{ define "slack.ooni.notice.title" }}
+{{ range $i, $el := .Alerts }}{{ if gt $i 0 }}
+{{ end }}{{ $el.Annotations.summary }}{{ end }}{{ end }}
diff --git a/ansible/roles/prometheus_alertmanager/templates/alertmanager.yml b/ansible/roles/prometheus_alertmanager/templates/alertmanager.yml
index eb36cf18..6d0136f4 100755
--- a/ansible/roles/prometheus_alertmanager/templates/alertmanager.yml
+++ b/ansible/roles/prometheus_alertmanager/templates/alertmanager.yml
@@ -42,6 +42,11 @@ route:
   # All the above attributes are inherited by all child routes and can
   # overwritten on each.
   routes:
+  # Notices of things that went right: Slack only, posted once
+  - receiver: slack-notice
+    match: {alertname: ClickHouseBackupCompleted}
+    group_by: ['alertname', 'instance', 'backup']
+    repeat_interval: 24h
   - receiver: team-email
     repeat_interval: 24h
     match: {severity: info}
@@ -69,6 +74,12 @@ receivers:
     title: '{% raw %}{{ template "slack.ooni.title" . }}{% endraw %}'
     channel: '#ooni-bots'
 
+- name: 'slack-notice' # no email, no "resolved" message
+  slack_configs:
+  - send_resolved: false
+    text: '{% raw %}{{ template "slack.ooni.text" . }}{% endraw %}'
+    title: '{% raw %}{{ template "slack.ooni.notice.title" . }}{% endraw %}'
+    channel: '#ooni-bots'
 - name: 'team-email' # no slack
   email_configs:
 {% for u in ['arturo', 'mehul'] %}
diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index fb3c20e5..02b8a82f 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -255,7 +255,10 @@ After each table, the script writes its result to
 (metrics `ooni_clickhouse_backup_*`, labels `backup` = job and `table`).
 Alerts, in `alert_rules.yml`:
 
-- `ClickHouseBackupFailed`: a table's last attempt failed.
+- `ClickHouseBackupFailed`: a table's last attempt failed. Nothing is
+  deleted, and the next run retries it.
+- `ClickHouseBackupCompleted`: a job's run backed up every table. One Slack
+  message per run in `#ooni-bots`, no email.
 - `ClickHouseBackupStale`: a table hasn't been backed up for 3 days.
 - `ClickHouseBackupMissing`: data1 or data3 hasn't written results for a day.
 - `ClickHouseBackupHostLoadHigh`: data3's 5-minute load is over half its

From c76e14e317f46930fac40bb8f2ba07649a695cc1 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 08:43:45 +0000
Subject: [PATCH 05/32] clickhouse_backup: word the completion notice for any
 number of tables

"1 tables backed up, 1 of them in full" now reads "Tables backed up: 1,
in full: 1".
---
 ansible/roles/prometheus/files/alert_rules.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ansible/roles/prometheus/files/alert_rules.yml b/ansible/roles/prometheus/files/alert_rules.yml
index d008c94d..66eb2e3c 100644
--- a/ansible/roles/prometheus/files/alert_rules.yml
+++ b/ansible/roles/prometheus/files/alert_rules.yml
@@ -166,7 +166,7 @@ groups:
         and on (instance, backup) ooni_clickhouse_backup_last_run_failed_tables == 0
     annotations:
       summary: 'ClickHouse backup {{ $labels.backup }} on {{ $labels.instance }} completed in {{ $value | humanizeDuration }}'
-      description: '{{ with printf `ooni_clickhouse_backup_last_run_tables{instance="%s",backup="%s"}` $labels.instance $labels.backup | query }}{{ . | first | value }}{{ end }} tables backed up, {{ with printf `ooni_clickhouse_backup_last_run_full_backups{instance="%s",backup="%s"}` $labels.instance $labels.backup | query }}{{ . | first | value }}{{ end }} of them in full. Nothing to do.'
+      description: 'Tables backed up: {{ with printf `ooni_clickhouse_backup_last_run_tables{instance="%s",backup="%s"}` $labels.instance $labels.backup | query }}{{ . | first | value }}{{ end }}, in full: {{ with printf `ooni_clickhouse_backup_last_run_full_backups{instance="%s",backup="%s"}` $labels.instance $labels.backup | query }}{{ . | first | value }}{{ end }}. Nothing to do.'
 
   # Backups run daily; a new full backup of obs_web can take a day and a half
   - alert: ClickHouseBackupStale

From fee921baa3cedce56abad297eea8b68cb9ddfe97 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 08:43:45 +0000
Subject: [PATCH 06/32] clickhouse_backup: test the role end to end in
 containers, in CI when it changes

tests/run.sh renders the role's configs from its templates, and
Alertmanager's from its role, with Ansible, then:
- promtool unit-tests the backup alerts in alert_rules.yml, and amtool
  checks Alertmanager's config and that the alerts reach the right
  receivers;
- test_schedule.py checks when full backups start and what is pruned,
  over months of daily and weekly runs with failures;
- test_e2e.py runs the script against ClickHouse 24.8.6.70, MinIO for
  R2, node_exporter, Prometheus 2.42.0 and Alertmanager 0.25.0 (the
  versions in production) and a stand-in for Slack, in docker compose:
  full and incremental backups and a restore; a failed run; a month's
  new full interrupted by a ClickHouse restart, which deletes nothing,
  then completed with the oldest chain pruned; and the Slack messages
  and emails each causes.

The workflow runs it on pull requests and pushes to main that change
this role, the alert rules or Alertmanager's role. About 7 minutes.
---
 .github/workflows/test_clickhouse_backup.yml  |  29 +++
 .../roles/clickhouse_backup/tests/.gitignore  |   1 +
 .../roles/clickhouse_backup/tests/README.md   |  35 +++
 .../tests/alert_rules_test.yml                |  44 ++++
 .../clickhouse_backup/tests/clickhouse.xml    |   7 +
 .../roles/clickhouse_backup/tests/compose.yml |  66 +++++
 .../clickhouse_backup/tests/prometheus.yml    |  14 ++
 .../roles/clickhouse_backup/tests/render.yml  |  34 +++
 ansible/roles/clickhouse_backup/tests/run.sh  |  34 +++
 .../clickhouse_backup/tests/runner.Dockerfile |   3 +
 .../roles/clickhouse_backup/tests/slack.py    |  27 ++
 .../roles/clickhouse_backup/tests/test_e2e.py | 234 ++++++++++++++++++
 .../clickhouse_backup/tests/test_schedule.py  | 120 +++++++++
 .../roles/clickhouse_backup/tests/vars.yml    |  34 +++
 14 files changed, 682 insertions(+)
 create mode 100644 .github/workflows/test_clickhouse_backup.yml
 create mode 100644 ansible/roles/clickhouse_backup/tests/.gitignore
 create mode 100644 ansible/roles/clickhouse_backup/tests/README.md
 create mode 100644 ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
 create mode 100644 ansible/roles/clickhouse_backup/tests/clickhouse.xml
 create mode 100644 ansible/roles/clickhouse_backup/tests/compose.yml
 create mode 100644 ansible/roles/clickhouse_backup/tests/prometheus.yml
 create mode 100644 ansible/roles/clickhouse_backup/tests/render.yml
 create mode 100755 ansible/roles/clickhouse_backup/tests/run.sh
 create mode 100644 ansible/roles/clickhouse_backup/tests/runner.Dockerfile
 create mode 100644 ansible/roles/clickhouse_backup/tests/slack.py
 create mode 100644 ansible/roles/clickhouse_backup/tests/test_e2e.py
 create mode 100644 ansible/roles/clickhouse_backup/tests/test_schedule.py
 create mode 100644 ansible/roles/clickhouse_backup/tests/vars.yml

diff --git a/.github/workflows/test_clickhouse_backup.yml b/.github/workflows/test_clickhouse_backup.yml
new file mode 100644
index 00000000..04a73ed6
--- /dev/null
+++ b/.github/workflows/test_clickhouse_backup.yml
@@ -0,0 +1,29 @@
+# Tests the ClickHouse backup role, its alerts and their Alertmanager routes
+# (ansible/roles/clickhouse_backup/tests), only when one of them changes
+name: Test ClickHouse backups
+on:
+  pull_request:
+    paths:
+      - "ansible/roles/clickhouse_backup/**"
+      - "ansible/roles/prometheus/files/alert_rules.yml"
+      - "ansible/roles/prometheus_alertmanager/**"
+      - ".github/workflows/test_clickhouse_backup.yml"
+  push:
+    branches: [main]
+    paths:
+      - "ansible/roles/clickhouse_backup/**"
+      - "ansible/roles/prometheus/files/alert_rules.yml"
+      - "ansible/roles/prometheus_alertmanager/**"
+      - ".github/workflows/test_clickhouse_backup.yml"
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - run: pip install ansible-core==2.17.7 boto3==1.26.27
+      - run: ansible/roles/clickhouse_backup/tests/run.sh
diff --git a/ansible/roles/clickhouse_backup/tests/.gitignore b/ansible/roles/clickhouse_backup/tests/.gitignore
new file mode 100644
index 00000000..9d931c43
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tests/.gitignore
@@ -0,0 +1 @@
+work/
diff --git a/ansible/roles/clickhouse_backup/tests/README.md b/ansible/roles/clickhouse_backup/tests/README.md
new file mode 100644
index 00000000..d2a97e6d
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tests/README.md
@@ -0,0 +1,35 @@
+# Tests for the clickhouse_backup role
+
+`./run.sh` runs them all, in about seven minutes:
+
+- `alert_rules_test.yml`: promtool unit tests of the backup alerts in
+  `roles/prometheus/files/alert_rules.yml`, then amtool checks that
+  Alertmanager's config is valid and routes them to the right receivers.
+- `test_schedule.py`: which runs start a full backup and what gets pruned,
+  over months of daily and weekly runs with failures, ClickHouse and the
+  bucket stubbed.
+- `test_e2e.py`: the real thing, in the containers of `compose.yml`:
+  ClickHouse at production's version, MinIO standing in for R2, the script
+  in a Debian bookworm Python with bookworm's boto3, node_exporter,
+  Prometheus with the repo's alert rules, Alertmanager with the repo's
+  routes, and a stand-in for Slack. Full and incremental backups, a
+  restore, a failed run, an interrupted full, the month's new full and
+  pruning, and the Slack messages and emails each one causes.
+
+The configs are rendered from the role's templates into `work/` by
+`render.yml`, with the variables in `vars.yml`.
+
+Needs docker compose, python3 with boto3, and ansible-core. With rootless
+podman:
+
+```bash
+DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock ./run.sh
+```
+
+`KEEP=1 ./run.sh` leaves the containers up afterwards (Prometheus on
+127.0.0.1:19990, Alertmanager on 19993); remove them with
+`docker compose -f compose.yml down -v`.
+
+GitHub runs it (`.github/workflows/test_clickhouse_backup.yml`) when a pull
+request or a push to main changes this role, the alert rules or
+Alertmanager's role.
diff --git a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
new file mode 100644
index 00000000..ce753a0f
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
@@ -0,0 +1,44 @@
+# promtool unit tests for the ClickHouse backup alerts in alert_rules.yml
+rule_files: [alert_rules.yml]
+evaluation_interval: 1m
+tests:
+  - interval: 1m
+    input_series:
+      # measurements finished at t=0 after 30 min, all 14 tables backed up, 1 in full
+      - {series: 'ooni_clickhouse_backup_last_run_finished_timestamp_seconds{instance="data3", backup="measurements"}', values: '0x180'}
+      - {series: 'ooni_clickhouse_backup_last_run_duration_seconds{instance="data3", backup="measurements"}', values: '1800x180'}
+      - {series: 'ooni_clickhouse_backup_last_run_failed_tables{instance="data3", backup="measurements"}', values: '0x180'}
+      - {series: 'ooni_clickhouse_backup_last_run_tables{instance="data3", backup="measurements"}', values: '14x180'}
+      - {series: 'ooni_clickhouse_backup_last_run_full_backups{instance="data3", backup="measurements"}', values: '1x180'}
+      # private finished at the same time with a table failed: fastpath, last backed up 4 days ago
+      - {series: 'ooni_clickhouse_backup_last_run_finished_timestamp_seconds{instance="data3", backup="private"}', values: '0x180'}
+      - {series: 'ooni_clickhouse_backup_last_run_duration_seconds{instance="data3", backup="private"}', values: '60x180'}
+      - {series: 'ooni_clickhouse_backup_last_run_failed_tables{instance="data3", backup="private"}', values: '1x180'}
+      - {series: 'ooni_clickhouse_backup_last_attempt_failed{instance="data3", backup="private", table="fastpath"}', values: '1x180'}
+      - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data3", backup="private", table="fastpath"}', values: '-345600x180'}
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: ClickHouseBackupCompleted
+        exp_alerts:
+          - exp_labels: {instance: data3, backup: measurements}
+            exp_annotations:
+              summary: 'ClickHouse backup measurements on data3 completed in 30m 0s'
+              description: 'Tables backed up: 14, in full: 1. Nothing to do.'
+      # an hour after the run it clears
+      - eval_time: 2h
+        alertname: ClickHouseBackupCompleted
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: ClickHouseBackupFailed
+        exp_alerts:
+          - exp_labels: {instance: data3, backup: private, table: fastpath}
+            exp_annotations:
+              summary: 'ClickHouse backup of fastpath (private) failed on data3'
+              description: 'Nothing was deleted: the older backups of fastpath are kept. The next run retries it. See `journalctl --identifier ooni-clickhouse-backup` on data3.'
+      - eval_time: 10m
+        alertname: ClickHouseBackupStale
+        exp_alerts:
+          - exp_labels: {instance: data3, backup: private, table: fastpath}
+            exp_annotations:
+              summary: 'No ClickHouse backup of fastpath (private) for 4d 0h 10m 0s'
+              description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on data3.'
diff --git a/ansible/roles/clickhouse_backup/tests/clickhouse.xml b/ansible/roles/clickhouse_backup/tests/clickhouse.xml
new file mode 100644
index 00000000..918706bd
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tests/clickhouse.xml
@@ -0,0 +1,7 @@
+<clickhouse>
+    <!-- Keep the test server small -->
+    <metric_log remove="1"/>
+    <asynchronous_metric_log remove="1"/>
+    <trace_log remove="1"/>
+    <max_server_memory_usage_to_ram_ratio>0.5</max_server_memory_usage_to_ram_ratio>
+</clickhouse>
diff --git a/ansible/roles/clickhouse_backup/tests/compose.yml b/ansible/roles/clickhouse_backup/tests/compose.yml
new file mode 100644
index 00000000..e90a9881
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tests/compose.yml
@@ -0,0 +1,66 @@
+# The backup's environment: ClickHouse at production's version, MinIO in place
+# of R2, the backup script in "runner", node_exporter reading its textfile,
+# Prometheus with the repo's alert rules, Alertmanager with the repo's routes,
+# and a stand-in for Slack. run.sh renders work/ first.
+name: chbackup-test
+services:
+  minio:
+    image: docker.io/pgsty/minio:RELEASE.2026-08-04T00-00-00Z
+    command: server /data
+    environment:
+      MINIO_ROOT_USER: testkey
+      MINIO_ROOT_PASSWORD: testsecret123
+    ports: ["127.0.0.1:19910:9000"]
+
+  clickhouse:
+    image: docker.io/clickhouse/clickhouse-server:24.8.6.70
+    environment:
+      CLICKHOUSE_USER: admin
+      CLICKHOUSE_PASSWORD: admin
+    volumes:
+      - ./work/ooni_backup_s3.xml:/etc/clickhouse-server/config.d/ooni_backup_s3.xml:ro,z
+      - ./clickhouse.xml:/etc/clickhouse-server/config.d/test.xml:ro,z
+    ulimits:
+      nofile: 262144
+    mem_limit: 2g
+    ports: ["127.0.0.1:19911:8123"]
+
+  runner:
+    build: {context: ., dockerfile: runner.Dockerfile}
+    command: sleep infinity
+    volumes:
+      - ../files:/role:ro,z
+      - ./work:/work:ro,z
+      - ./work/textfile:/var/lib/node_exporter/textfile_collector:z
+
+  node_exporter:
+    image: quay.io/prometheus/node-exporter:v1.8.2
+    command: [--collector.disable-defaults, --collector.textfile, --collector.textfile.directory=/textfile]
+    volumes:
+      - ./work/textfile:/textfile:ro,z
+
+  # monitoring.ooni.org runs Debian's prometheus 2.42.0
+  prometheus:
+    image: docker.io/prom/prometheus:v2.42.0
+    command: [--config.file=/etc/prometheus/prometheus.yml]
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro,z
+      - ../../prometheus/files/alert_rules.yml:/etc/prometheus/alert_rules.yml:ro,z
+      - ./alert_rules_test.yml:/etc/prometheus/alert_rules_test.yml:ro,z
+    ports: ["127.0.0.1:19990:9090"]
+
+  # and Debian's alertmanager 0.25.0
+  alertmanager:
+    image: docker.io/prom/alertmanager:v0.25.0
+    command: [--config.file=/etc/alertmanager/alertmanager.yml]
+    volumes:
+      - ./work/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro,z
+      - ../../prometheus_alertmanager/files/templates:/etc/prometheus/alertmanager_templates:ro,z
+    ports: ["127.0.0.1:19993:9093"]
+
+  slack:
+    image: docker.io/library/python:3.11-slim-bookworm
+    command: [python3, /slack.py]
+    volumes:
+      - ./slack.py:/slack.py:ro,z
+    ports: ["127.0.0.1:19980:8080"]
diff --git a/ansible/roles/clickhouse_backup/tests/prometheus.yml b/ansible/roles/clickhouse_backup/tests/prometheus.yml
new file mode 100644
index 00000000..6e86326b
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tests/prometheus.yml
@@ -0,0 +1,14 @@
+global:
+  scrape_interval: 5s
+  evaluation_interval: 5s
+rule_files: [alert_rules.yml]
+alerting:
+  alertmanagers:
+    - static_configs: [{targets: ["alertmanager:9093"]}]
+scrape_configs:
+  # The backup's textfile, labelled as data3 like the production scrape
+  - job_name: node
+    static_configs: [{targets: ["node_exporter:9100"]}]
+    relabel_configs:
+      - target_label: instance
+        replacement: data3.htz-fsn.prod.ooni.nu
diff --git a/ansible/roles/clickhouse_backup/tests/render.yml b/ansible/roles/clickhouse_backup/tests/render.yml
new file mode 100644
index 00000000..a42ae1ef
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tests/render.yml
@@ -0,0 +1,34 @@
+# Renders the role's templates and Alertmanager's into work/, with vars.yml
+- hosts: localhost
+  connection: local
+  gather_facts: false
+  vars_files:
+    - ../defaults/main.yml
+    - vars.yml
+  tasks:
+    - name: Render the backup script's configs
+      ansible.builtin.template:
+        src: "{{ playbook_dir }}/../templates/clickhouse-backup.json.j2"
+        dest: "{{ playbook_dir }}/work/{{ item.name }}.json"
+        mode: "0644"
+      vars:
+        clickhouse_backup_jobs: "{{ item.jobs }}"
+        clickhouse_backup_max_bandwidth: "{{ item.bandwidth }}"
+      loop: "{{ test_configs }}"
+
+    - name: Render ClickHouse's bucket credentials, for every bucket of every config
+      ansible.builtin.template:
+        src: "{{ playbook_dir }}/../templates/ooni_backup_s3.xml.j2"
+        dest: "{{ playbook_dir }}/work/ooni_backup_s3.xml"
+        mode: "0644"
+      vars:
+        clickhouse_backup_jobs: "{{ test_configs | map(attribute='jobs') | flatten }}"
+
+    - name: Render Alertmanager's config, with Slack and the mail server stood in for
+      ansible.builtin.template:
+        src: "{{ playbook_dir }}/work/alertmanager.yml.j2"
+        dest: "{{ playbook_dir }}/work/alertmanager.yml"
+        mode: "0644"
+      vars:
+        alertmanager_slack_api_url: http://slack:8080/
+        am_mx_openobservatory: mx.invalid
diff --git a/ansible/roles/clickhouse_backup/tests/run.sh b/ansible/roles/clickhouse_backup/tests/run.sh
new file mode 100755
index 00000000..3f15659c
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tests/run.sh
@@ -0,0 +1,34 @@
+#!/bin/bash
+# Tests the clickhouse_backup role: renders its configs from the role's templates, checks the backup alerts
+# (promtool) and their Alertmanager routes (amtool), runs the schedule tests, then test_e2e.py against the
+# containers in compose.yml. Needs docker compose, python3 with boto3, and ansible-core. With rootless podman,
+# point DOCKER_HOST at its socket. KEEP=1 leaves the containers running.
+set -euo pipefail
+cd "$(dirname "$0")"
+export COMPOSE=${COMPOSE:-docker compose}
+export PYTHONDONTWRITEBYTECODE=1  # no __pycache__ left in the role's files/
+compose() { $COMPOSE -f compose.yml "$@"; }
+
+echo "== rendering the configs"
+rm -rf work && mkdir -p work/textfile
+# The template looks up openobservatory.org's MX record for the mail server; render.yml sets one instead
+sed '/lookup(.dig./d' ../../prometheus_alertmanager/templates/alertmanager.yml > work/alertmanager.yml.j2
+ansible-playbook -i localhost, render.yml > work/render.log || { cat work/render.log; exit 1; }
+
+echo "== alert rules and routes"
+compose run --rm --no-deps --entrypoint promtool prometheus test rules /etc/prometheus/alert_rules_test.yml
+compose run --rm --no-deps --entrypoint amtool alertmanager check-config /etc/alertmanager/alertmanager.yml
+route() {  # alert labels, receiver it must reach
+  compose run --rm --no-deps --entrypoint amtool alertmanager config routes test \
+    --config.file=/etc/alertmanager/alertmanager.yml --verify.receivers="$2" $1
+}
+route "alertname=ClickHouseBackupCompleted instance=data3 backup=measurements" slack-notice
+route "alertname=ClickHouseBackupFailed instance=data3 backup=measurements table=fastpath" team-all
+
+echo "== schedule"
+python3 -m unittest test_schedule
+
+echo "== end to end"
+[ -n "${KEEP:-}" ] || trap 'compose down -v --remove-orphans >/dev/null 2>&1' EXIT
+compose up -d --build --quiet-pull
+python3 test_e2e.py
diff --git a/ansible/roles/clickhouse_backup/tests/runner.Dockerfile b/ansible/roles/clickhouse_backup/tests/runner.Dockerfile
new file mode 100644
index 00000000..48ba2c22
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tests/runner.Dockerfile
@@ -0,0 +1,3 @@
+# Debian bookworm's python3 and python3-boto3, as on the ClickHouse hosts
+FROM docker.io/library/python:3.11-slim-bookworm
+RUN pip install --no-cache-dir boto3==1.26.27 && mkdir -p /run/lock
diff --git a/ansible/roles/clickhouse_backup/tests/slack.py b/ansible/roles/clickhouse_backup/tests/slack.py
new file mode 100644
index 00000000..85ff0e93
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tests/slack.py
@@ -0,0 +1,27 @@
+"""Stands in for Slack's incoming webhook: keeps what Alertmanager posts, and
+returns it all, as a JSON list, to GET."""
+import json
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+messages = []
+
+
+class Handler(BaseHTTPRequestHandler):
+    def do_POST(self):
+        messages.append(json.loads(self.rfile.read(int(self.headers["Content-Length"]))))
+        self.reply(b"ok")
+
+    def do_GET(self):
+        self.reply(json.dumps(messages).encode())
+
+    def reply(self, body):
+        self.send_response(200)
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, *args):
+        pass
+
+
+ThreadingHTTPServer(("", 8080), Handler).serve_forever()
diff --git a/ansible/roles/clickhouse_backup/tests/test_e2e.py b/ansible/roles/clickhouse_backup/tests/test_e2e.py
new file mode 100644
index 00000000..a43750dd
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tests/test_e2e.py
@@ -0,0 +1,234 @@
+"""End-to-end test of the backup in compose.yml's environment, after run.sh has rendered work/ and started it.
+In order:
+  1. a first run takes a full backup of every table, each job to its own bucket
+  2. ClickHouseBackupCompleted reaches Slack once per job, as a notice, without email
+  3. a second run takes incrementals, and a restore from them returns the same rows
+  4. a run whose bucket is missing fails, and ClickHouseBackupFailed reaches Slack and email
+  5. a new month's full interrupted by a ClickHouse restart deletes nothing; the next run
+     removes what it left, takes the full and prunes the oldest chain
+Prints each check; exits 1 at the first failure."""
+import json
+import os
+import shlex
+import subprocess
+import sys
+import time
+import urllib.request
+from datetime import datetime, timezone
+
+import boto3
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+COMPOSE = shlex.split(os.environ.get("COMPOSE", "docker compose")) + ["-f", os.path.join(HERE, "compose.yml")]
+MEASUREMENTS, PRIVATE = "ooni-clickhouse-backup", "ooni-clickhouse-backup-private"
+DATA3 = "data3.htz-fsn.prod.ooni.nu"  # the instance label prometheus.yml gives the textfile
+s3 = boto3.client(
+    "s3", endpoint_url="http://127.0.0.1:19910", region_name="us-east-1",
+    aws_access_key_id="testkey", aws_secret_access_key="testsecret123",
+)
+
+
+def check(ok, what):
+    print(f"  {'ok  ' if ok else 'FAIL'} {what}")
+    if not ok:
+        sys.exit(1)
+
+
+def http(url, data=None):
+    with urllib.request.urlopen(url, data=data, timeout=30) as r:
+        return r.read().decode()
+
+
+def ch(sql):
+    return http("http://127.0.0.1:19911/?user=admin&password=admin", sql.encode()).strip()
+
+
+def backup(config, *args):
+    """Run the backup script in the runner container, as the timer runs it on data3"""
+    cmd = COMPOSE + ["exec", "-T", "runner", "python3", "/role/ooni_clickhouse_backup.py",
+                     "--config", f"/work/{config}.json", *args]
+    return subprocess.run(cmd, capture_output=True, text=True)
+
+
+def keys(bucket, prefix):
+    pages = s3.get_paginator("list_objects_v2").paginate(Bucket=bucket, Prefix=prefix)
+    return [o["Key"] for page in pages for o in page.get("Contents", [])]
+
+
+def chains(bucket, table):
+    """{chain: {backup: complete}} under prod/ooni/<table>/"""
+    out = {}
+    for key in keys(bucket, f"prod/ooni/{table}/"):
+        chain, name = key.split("/")[3:5]
+        backups = out.setdefault(chain, {})
+        backups[name] = backups.get(name, False) or key.endswith(f"/{name}/.backup")
+    return out
+
+
+def copy_prefix(bucket, src, dst):
+    for key in keys(bucket, src):
+        s3.copy_object(Bucket=bucket, Key=dst + key[len(src):], CopySource={"Bucket": bucket, "Key": key})
+
+
+def delete_prefix(bucket, prefix):
+    for key in keys(bucket, prefix):
+        s3.delete_object(Bucket=bucket, Key=key)
+
+
+def wait_for(what, get, timeout=150):
+    """Poll get() until it returns something truthy"""
+    t0 = time.monotonic()
+    while time.monotonic() - t0 < timeout:
+        try:
+            value = get()
+        except OSError:
+            value = None
+        if value:
+            return value
+        time.sleep(2)
+    check(False, f"{what}, within {timeout}s")
+
+
+def slack():
+    """(title, text) of every message the Slack stand-in has received"""
+    return [(a["title"].strip(), a.get("text", "").strip()) for m in json.loads(http("http://127.0.0.1:19980/"))
+            for a in m.get("attachments", [])]
+
+
+def email_attempts():
+    lines = http("http://127.0.0.1:19993/metrics").splitlines()
+    return sum(float(l.split()[-1]) for l in lines
+               if l.startswith('alertmanager_notifications_total{integration="email"'))
+
+
+def months_ago(n):
+    now = datetime.now(timezone.utc)
+    y, m = divmod(now.year * 12 + now.month - 1 - n, 12)
+    return f"{y}{m + 1:02}01T023000Z"
+
+
+def setup():
+    print("== waiting for the services")
+    wait_for("ClickHouse", lambda: ch("SELECT 1") == "1")
+    wait_for("MinIO", lambda: http("http://127.0.0.1:19910/minio/health/live") == "")
+    wait_for("Prometheus", lambda: "Ready" in http("http://127.0.0.1:19990/-/ready"))
+    wait_for("Alertmanager", lambda: "OK" in http("http://127.0.0.1:19993/-/ready"))
+    wait_for("Slack stand-in", lambda: http("http://127.0.0.1:19980/") == "[]")
+    for bucket in (MEASUREMENTS, PRIVATE):
+        s3.create_bucket(Bucket=bucket)
+    for sql in [
+        "CREATE DATABASE ooni",
+        "CREATE TABLE ooni.fastpath (measurement_uid String, input String, anomaly LowCardinality(String)) "
+        "ENGINE = MergeTree ORDER BY measurement_uid",
+        "INSERT INTO ooni.fastpath SELECT toString(number), 'https://example.org/' || toString(number % 100), "
+        "if(number % 7 = 0, 't', 'f') FROM numbers(200000)",
+        # ~20 MB that doesn't compress: takes ~20 s to back up with work/slow.json
+        "CREATE TABLE ooni.obs_web (measurement_uid String, blob String) ENGINE = MergeTree ORDER BY measurement_uid",
+        "INSERT INTO ooni.obs_web SELECT toString(number), randomString(4096) FROM numbers(5000)",
+        "CREATE TABLE ooni.fingerprints_dns (name String, pattern String) ENGINE = EmbeddedRocksDB PRIMARY KEY name",
+        "INSERT INTO ooni.fingerprints_dns VALUES ('a', '1.2.3.4'), ('b', '5.6.7.8')",
+        "CREATE TABLE ooni.msmt_feedback (measurement_uid String, comment String) ENGINE = MergeTree ORDER BY measurement_uid",
+        "INSERT INTO ooni.msmt_feedback VALUES ('m1', 'blocked?')",
+        "CREATE TABLE ooni.labeling_frames (measurement_uid String) ENGINE = MergeTree ORDER BY measurement_uid",
+        "INSERT INTO ooni.labeling_frames VALUES ('x')",
+        "CREATE TABLE ooni.asnmeta_tmp (asn UInt32) ENGINE = MergeTree ORDER BY asn",  # excluded: *_tmp
+    ]:
+        ch(sql)
+
+
+def test_first_run():
+    print("== 1. first run: a full backup of every table")
+    r = backup("backup", "run")
+    check(r.returncode == 0, f"the run succeeds{'' if r.returncode == 0 else ': ' + r.stderr[-2000:]}")
+    for bucket, table in [(MEASUREMENTS, "fastpath"), (MEASUREMENTS, "obs_web"),
+                          (MEASUREMENTS, "fingerprints_dns"), (PRIVATE, "msmt_feedback")]:
+        c = chains(bucket, table)
+        check(len(c) == 1 and all(all(b.values()) for b in c.values()), f"{table}: one complete chain in {bucket}")
+    check(any(k.endswith("/data.native.zst") for k in keys(MEASUREMENTS, "prod/ooni/fingerprints_dns/")),
+          "fingerprints_dns (EmbeddedRocksDB): its rows are exported too")
+    check(not keys(MEASUREMENTS, "prod/ooni/msmt_feedback/"), "msmt_feedback isn't in the measurements bucket")
+    check(not keys(MEASUREMENTS, "prod/ooni/asnmeta_tmp/"), "asnmeta_tmp (excluded) isn't backed up")
+    check(not keys(MEASUREMENTS, "prod/ooni/labeling_frames/"), "labeling_frames (data1's) isn't backed up")
+
+
+def test_completed_alert(emails_before):
+    print("== 2. ClickHouseBackupCompleted")
+    for job, tables in (("measurements", 3), ("private", 1)):
+        title = f"ClickHouse backup {job} on {DATA3} completed"
+        msgs = wait_for(f"a Slack message: {title}", lambda: [m for m in slack() if m[0].startswith(title)])
+        check(len(msgs) == 1, f"one Slack message for {job}: {msgs[0][0]!r}")
+        check(f"Tables backed up: {tables}, in full: {tables}" in msgs[0][1], f"it says: {msgs[0][1]!r}")
+    check(not any(t.startswith("[FIRING]") for t, _ in slack()), "no [FIRING] in the titles")
+    check(email_attempts() == emails_before, "no email sent")
+
+
+def test_incremental_and_restore():
+    print("== 3. second run: incrementals, and a restore")
+    ch("INSERT INTO ooni.fastpath SELECT toString(number), 'https://example.org/new', 'f' FROM numbers(200000, 1000)")
+    r = backup("backup", "run")
+    check(r.returncode == 0, "the run succeeds")
+    (chain, backups), = chains(MEASUREMENTS, "fastpath").items()
+    check(len(backups) == 2 and all(backups.values()), f"fastpath: a full and an incremental in chain {chain}")
+    latest = f"http://minio:9000/{MEASUREMENTS}/prod/ooni/fastpath/{chain}/{max(backups)}"
+    ch("CREATE DATABASE restored")
+    ch(f"RESTORE TABLE ooni.fastpath AS restored.fastpath FROM S3('{latest}')")
+    digest = "SELECT count(), sum(cityHash64(measurement_uid, input, anomaly)) FROM {}"
+    check(ch(digest.format("restored.fastpath")) == ch(digest.format("ooni.fastpath")),
+          "the restored fastpath has the same rows (201000)")
+
+
+def test_failure_alert(emails_before):
+    print("== 4. a run that fails: ClickHouseBackupFailed")
+    r = backup("broken", "run")
+    check(r.returncode == 1, "the run exits 1")
+    title = f"[FIRING] ClickHouse backup of labeling_frames (broken) failed on {DATA3}"
+    msgs = wait_for(f"a Slack message: {title}", lambda: [m for m in slack() if m[0] == title])
+    check("Nothing was deleted" in msgs[0][1], f"it says what was done: {msgs[0][1]!r}")
+    wait_for("an email attempt for it", lambda: email_attempts() > emails_before)
+    check(not any(t.startswith("ClickHouse backup broken") for t, _ in slack()), "no 'completed' message for it")
+
+
+def test_interrupted_full():
+    print("== 5. a new month's full, interrupted, then completed")
+    (chain, backups), = chains(MEASUREMENTS, "obs_web").items()
+    full = f"prod/ooni/obs_web/{chain}/{min(backups)}/"
+    old, previous = months_ago(2), months_ago(1)
+    # obs_web's chains as at the start of a month: the full of two months ago and of last month
+    for c in (old, previous):
+        copy_prefix(MEASUREMENTS, full, f"prod/ooni/obs_web/{c}/{c}/")
+    delete_prefix(MEASUREMENTS, f"prod/ooni/obs_web/{chain}/")
+
+    run = subprocess.Popen(COMPOSE + ["exec", "-T", "runner", "python3", "/role/ooni_clickhouse_backup.py",
+                                      "--config", "/work/slow.json", "run", "--table", "obs_web"],
+                           stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
+    wait_for("the new full's first files", lambda: set(chains(MEASUREMENTS, "obs_web")) - {old, previous}, 60)
+    subprocess.run(COMPOSE + ["restart", "clickhouse"], check=True, capture_output=True)
+    out, _ = run.communicate(timeout=180)
+    check(run.returncode == 1, "the interrupted run exits 1")
+    c = chains(MEASUREMENTS, "obs_web")
+    new = sorted(set(c) - {old, previous})
+    check(new and not any(c[new[0]].values()), f"the new chain {new[0]} has no .backup")
+    check(all(c[x][x] for x in (old, previous)), f"both older chains, {old} and {previous}, are still there")
+
+    wait_for("ClickHouse after the restart", lambda: ch("SELECT 1") == "1")
+    r = backup("backup", "run", "--table", "obs_web")
+    check(r.returncode == 0, "the next run succeeds")
+    check("removed incomplete chain" in r.stderr, "it removed the interrupted full")
+    c = chains(MEASUREMENTS, "obs_web")
+    check(previous in c and old not in c and len(c) == 2 and all(all(b.values()) for b in c.values()),
+          f"it took the full and pruned {old}: chains {sorted(c)}")
+
+
+def main():
+    setup()
+    emails_before = email_attempts()
+    test_first_run()
+    test_completed_alert(emails_before)
+    test_incremental_and_restore()
+    test_failure_alert(emails_before)
+    test_interrupted_full()
+    print("PASS")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/ansible/roles/clickhouse_backup/tests/test_schedule.py b/ansible/roles/clickhouse_backup/tests/test_schedule.py
new file mode 100644
index 00000000..b7ea3146
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tests/test_schedule.py
@@ -0,0 +1,120 @@
+"""When ooni_clickhouse_backup.py starts a full backup and what it prunes, over months of daily or weekly runs,
+some of them failing. ClickHouse and the bucket are stubbed; test_e2e.py runs the real thing.
+    python3 -m unittest test_schedule"""
+import importlib.util
+import json
+import os
+import unittest
+from datetime import datetime, timedelta, timezone
+
+spec = importlib.util.spec_from_file_location(
+    "backup", os.path.join(os.path.dirname(__file__), "..", "files", "ooni_clickhouse_backup.py")
+)
+backup = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(backup)
+
+START = datetime(2026, 10, 23, 2, 30, tzinfo=timezone.utc)  # a Friday
+
+
+class FakeStore:
+    """The bucket: {chain: {"backups": {backup: complete}, "keep": [labels]}}"""
+
+    def __init__(self):
+        self.c = {}
+
+    def table_prefix(self, db, table):
+        return f"{db}/{table}/"
+
+    def ch_s3(self, key):
+        return f"'{key}'"
+
+    def chains(self, db, table):
+        return {k: {"backups": dict(e["backups"]), "keep": list(e["keep"])} for k, e in self.c.items()}
+
+    def delete_prefix(self, prefix):
+        parts = prefix.rstrip("/").split("/")
+        if parts[-2] in self.c:  # one backup of a chain
+            self.c[parts[-2]]["backups"].pop(parts[-1], None)
+        else:
+            self.c.pop(parts[-1], None)
+        return 1
+
+    def put_empty(self, key):
+        pass
+
+
+class FakeClickHouse:
+    """Takes a backup, or fails one part-way: files uploaded, no .backup"""
+
+    def __init__(self, store):
+        self.store, self.fail = store, False
+
+    def query(self, sql, settings=None, timeout=600):
+        chain, name = sql.split("TO S3('")[1].split("'")[0].split("/")[-2:]
+        self.store.c.setdefault(chain, {"backups": {}, "keep": []})["backups"][name] = not self.fail
+        if self.fail:
+            raise RuntimeError("BACKUP_FAILED")
+        return json.dumps({"id": "op"})
+
+
+def runs(every_days, until, failing=(), jitter=True):
+    """Back up a table at 02:30 UTC daily or on Sundays from START. Yields (day, full, complete chains)."""
+    job = backup.BackupJob.__new__(backup.BackupJob)
+    job.db, job.name, job.keep_chains, job.settings = "ooni", "t", 2, {}
+    job.store = FakeStore()
+    job.ch = FakeClickHouse(job.store)
+    job.metrics = backup.Metrics()
+    job.wait = lambda op, what: {"num_files": 1, "uncompressed_size": 1}
+    t, n = START, 0
+    while t < until:
+        # a table's backup starts a few minutes earlier or later each time
+        now = t + timedelta(minutes=9 if n % 2 else -7)
+        backup.utcnow = lambda now=now: now
+        job.ch.fail = now.date().isoformat() in failing
+        try:
+            full = job.backup_table("obs_web", "ReplicatedMergeTree", False, "")
+        except RuntimeError:
+            full = None
+        complete = sorted(c for c, e in job.store.c.items() if any(e["backups"].values()))
+        yield now.date().isoformat(), full, complete
+        n += 1
+        if every_days == 7:
+            t += timedelta(days=(6 - t.weekday()) % 7 or 7)  # the next Sunday
+        else:
+            t += timedelta(days=1)
+
+
+class TestSchedule(unittest.TestCase):
+    def fulls(self, every_days):
+        return [day for day, full, _ in runs(every_days, datetime(2027, 4, 1, tzinfo=timezone.utc)) if full]
+
+    def test_weekly_fulls_on_the_first_sunday_of_each_month(self):
+        self.assertEqual(
+            self.fulls(7), ["2026-10-23", "2026-11-01", "2026-12-06", "2027-01-03", "2027-02-07", "2027-03-07"]
+        )
+
+    def test_daily_fulls_on_the_first_of_each_month(self):
+        self.assertEqual(
+            self.fulls(1), ["2026-10-23", "2026-11-01", "2026-12-01", "2027-01-01", "2027-02-01", "2027-03-01"]
+        )
+
+    def test_two_complete_chains_kept(self):
+        *_, (day, _, complete) = runs(7, datetime(2027, 4, 1, tzinfo=timezone.utc))
+        self.assertEqual([c[:8] for c in complete], ["20270207", "20270307"])
+
+    def test_failed_fulls_prune_nothing_and_are_retried(self):
+        failing = {"2026-12-06", "2026-12-13", "2027-02-07", "2027-02-14", "2027-02-21", "2027-02-28", "2027-03-07"}
+        before = []
+        for day, full, complete in runs(7, datetime(2027, 5, 1, tzinfo=timezone.utc), failing):
+            month = day[:7].replace("-", "")
+            if not any(c.startswith(month) for c in complete):
+                # no complete full this month yet: every complete chain from before is still there
+                self.assertTrue(set(before) <= set(complete), f"{day}: deleted {set(before) - set(complete)}")
+            if day in failing:
+                self.assertIsNone(full, day)
+            before = complete
+        self.assertEqual([c[:8] for c in complete], ["20270314", "20270404"])
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/ansible/roles/clickhouse_backup/tests/vars.yml b/ansible/roles/clickhouse_backup/tests/vars.yml
new file mode 100644
index 00000000..0db73f9f
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tests/vars.yml
@@ -0,0 +1,34 @@
+# Role variables for the test environment in compose.yml, with data3's two
+# jobs: the measurement tables and the private tables, each in its own bucket.
+clickhouse_admin_user: admin
+clickhouse_admin_password: admin
+clickhouse_backup_clickhouse_url: http://clickhouse:8123/
+clickhouse_backup_s3_endpoint: http://minio:9000
+clickhouse_backup_s3_access_key_id: testkey
+clickhouse_backup_s3_secret_access_key: testsecret123
+
+test_jobs:
+  - name: measurements
+    s3_bucket: ooni-clickhouse-backup
+    include_tables: []
+    exclude_tables: [msmt_feedback, labeling_frames]
+  - name: private
+    s3_bucket: ooni-clickhouse-backup-private
+    include_tables: [msmt_feedback]
+    exclude_tables: []
+
+# One script config per entry, in work/<name>.json
+test_configs:
+  - name: backup  # as deployed
+    bandwidth: 0
+    jobs: "{{ test_jobs }}"
+  - name: slow  # ~1 MB/s, so a full backup can be interrupted
+    bandwidth: 3000000
+    jobs: "{{ test_jobs }}"
+  - name: broken  # its bucket doesn't exist, so every backup fails
+    bandwidth: 0
+    jobs:
+      - name: broken
+        s3_bucket: no-such-bucket
+        include_tables: [labeling_frames]
+        exclude_tables: []

From 7857e1c2490d8dbf9d62f10491954e7e748b09ce Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 10:03:31 +0000
Subject: [PATCH 07/32] clickhouse_backup tests: use Debian's python3-boto3,
 not pip

The script now runs on Debian bookworm with its python3 and
python3-boto3, the packages the role installs on the ClickHouse hosts,
instead of a Python image with boto3 from pip. The workflow installs
ansible-core and python3-boto3 with apt for the scripts around it.
---
 .github/workflows/test_clickhouse_backup.yml            | 5 +----
 ansible/roles/clickhouse_backup/tests/README.md         | 2 +-
 ansible/roles/clickhouse_backup/tests/run.sh            | 2 +-
 ansible/roles/clickhouse_backup/tests/runner.Dockerfile | 9 ++++++---
 4 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/.github/workflows/test_clickhouse_backup.yml b/.github/workflows/test_clickhouse_backup.yml
index 04a73ed6..f660b978 100644
--- a/.github/workflows/test_clickhouse_backup.yml
+++ b/.github/workflows/test_clickhouse_backup.yml
@@ -22,8 +22,5 @@ jobs:
     timeout-minutes: 30
     steps:
       - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.11"
-      - run: pip install ansible-core==2.17.7 boto3==1.26.27
+      - run: sudo apt-get update && sudo apt-get install -y --no-install-recommends ansible-core python3-boto3
       - run: ansible/roles/clickhouse_backup/tests/run.sh
diff --git a/ansible/roles/clickhouse_backup/tests/README.md b/ansible/roles/clickhouse_backup/tests/README.md
index d2a97e6d..6a5c9fc1 100644
--- a/ansible/roles/clickhouse_backup/tests/README.md
+++ b/ansible/roles/clickhouse_backup/tests/README.md
@@ -10,7 +10,7 @@
   bucket stubbed.
 - `test_e2e.py`: the real thing, in the containers of `compose.yml`:
   ClickHouse at production's version, MinIO standing in for R2, the script
-  in a Debian bookworm Python with bookworm's boto3, node_exporter,
+  on Debian bookworm with its python3-boto3, as on the hosts, node_exporter,
   Prometheus with the repo's alert rules, Alertmanager with the repo's
   routes, and a stand-in for Slack. Full and incremental backups, a
   restore, a failed run, an interrupted full, the month's new full and
diff --git a/ansible/roles/clickhouse_backup/tests/run.sh b/ansible/roles/clickhouse_backup/tests/run.sh
index 3f15659c..f2491da8 100755
--- a/ansible/roles/clickhouse_backup/tests/run.sh
+++ b/ansible/roles/clickhouse_backup/tests/run.sh
@@ -1,7 +1,7 @@
 #!/bin/bash
 # Tests the clickhouse_backup role: renders its configs from the role's templates, checks the backup alerts
 # (promtool) and their Alertmanager routes (amtool), runs the schedule tests, then test_e2e.py against the
-# containers in compose.yml. Needs docker compose, python3 with boto3, and ansible-core. With rootless podman,
+# containers in compose.yml. Needs docker compose, python3-boto3 and ansible-core. With rootless podman,
 # point DOCKER_HOST at its socket. KEEP=1 leaves the containers running.
 set -euo pipefail
 cd "$(dirname "$0")"
diff --git a/ansible/roles/clickhouse_backup/tests/runner.Dockerfile b/ansible/roles/clickhouse_backup/tests/runner.Dockerfile
index 48ba2c22..3abc14ab 100644
--- a/ansible/roles/clickhouse_backup/tests/runner.Dockerfile
+++ b/ansible/roles/clickhouse_backup/tests/runner.Dockerfile
@@ -1,3 +1,6 @@
-# Debian bookworm's python3 and python3-boto3, as on the ClickHouse hosts
-FROM docker.io/library/python:3.11-slim-bookworm
-RUN pip install --no-cache-dir boto3==1.26.27 && mkdir -p /run/lock
+# Debian bookworm with its python3 and python3-boto3, which the role installs on the ClickHouse hosts
+FROM docker.io/library/debian:bookworm-slim
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends python3 python3-boto3 \
+    && rm -rf /var/lib/apt/lists/* \
+    && mkdir -p /run/lock

From 294ef5449f3d60f1aaf93addea9af8b8f3c098cf Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 10:39:10 +0000
Subject: [PATCH 08/32] clickhouse_backup: cap backups at 50 MB/s until the
 real rate is measured

max_backup_bandwidth was 120 MB/s, on the strength of 24.8 uploading at
about a third of it in tests against MinIO. If production doesn't show
the same, that would take most of the nodes' 1 Gbit/s link. Cap it at
50 MB/s, about 31 hours for a full backup of everything, and raise it
if the first backups show the third: about 4 days for a full, longer
than ClickHouseBackupStale's 3. The comments and the runbook say how to
check the rate.
---
 ansible/roles/clickhouse_backup/defaults/main.yml      | 10 ++++++----
 .../clickhouse_backup/files/ooni_clickhouse_backup.py  |  3 ++-
 docs/Runbooks.md                                       | 10 +++++++---
 3 files changed, 15 insertions(+), 8 deletions(-)

diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml
index c2165e59..e8a7b461 100644
--- a/ansible/roles/clickhouse_backup/defaults/main.yml
+++ b/ansible/roles/clickhouse_backup/defaults/main.yml
@@ -46,10 +46,12 @@ clickhouse_backup_jobs:
 # Complete chains kept per table, besides chains pinned with --keep: the
 # current one and the one before, so a month or two of restore points
 clickhouse_backup_keep_chains: 2
-# Bytes/s passed to ClickHouse as max_backup_bandwidth. On 24.8 actual
-# throughput measured at about a third of this, so 120 MB/s gives ~40 MB/s,
-# which leaves room on the 1 Gbit/s link for replication.
-clickhouse_backup_max_bandwidth: 120000000
+# Bytes/s passed to ClickHouse as max_backup_bandwidth: at most 50 MB/s, 40%
+# of the 1 Gbit/s link, leaving the rest for replication and queries.
+# Against MinIO in testing, 24.8 uploaded at about a third of this value; check
+# the real rate on the first backups (ooni_clickhouse_backup_last_size_bytes /
+# ooni_clickhouse_backup_last_duration_seconds) and raise it if they're too slow.
+clickhouse_backup_max_bandwidth: 50000000
 
 # fnmatch patterns, for every job. Materialized views' .inner tables are
 # always skipped: they're backed up with their view.
diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
index 69a5dd70..31fb628b 100644
--- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
+++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
@@ -369,7 +369,8 @@ def __init__(self, conf: dict, metrics: Metrics = None):
         bw = int(conf.get("max_backup_bandwidth", 0))
         if bw:
             # A query setting, not a BACKUP setting: 24.8 rejects it inside
-            # BACKUP ... SETTINGS. Measured throughput is ~1/3 of this value.
+            # BACKUP ... SETTINGS. Against MinIO in testing, the throughput
+            # was about a third of this value.
             self.settings["max_backup_bandwidth"] = str(bw)
 
     def tables(self, only: list) -> tuple:
diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index 02b8a82f..fe04034e 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -209,9 +209,13 @@ When a data1-only table becomes replicated, remove it from
   so one backup file can reach 1.25 TiB.
 - Two complete chains are kept per table (`clickhouse_backup_keep_chains`),
   so a month or two of restore points.
-- Bandwidth is capped with `clickhouse_backup_max_bandwidth`. On 24.8 the
-  actual rate is about a third of the cap: 120 MB/s gives ~40 MB/s. A new
-  full backup of everything (~5.6 TB) takes about 39 hours at that rate.
+- Bandwidth is capped with `clickhouse_backup_max_bandwidth`, at 50 MB/s. A
+  new full backup of everything (~5.6 TB) takes about 31 hours at that rate.
+  Against MinIO in testing, 24.8 uploaded at only about a third of the cap,
+  which would make it about 4 days, longer than `ClickHouseBackupStale`'s 3:
+  check the rate of the first backups
+  (`ooni_clickhouse_backup_last_size_bytes / ooni_clickhouse_backup_last_duration_seconds`)
+  and raise the cap if they're that slow.
 
 Layout in the bucket:
 

From caf6d7d1d04a17575d56210ef8ee37464e43e4dc Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 10:39:10 +0000
Subject: [PATCH 09/32] clickhouse_backup: stop excluding counters_test_list,
 which is detached

counters_test_list is detached permanently, so it's no longer in
system.tables, which the script takes its tables from.
---
 ansible/roles/clickhouse_backup/defaults/main.yml | 1 -
 1 file changed, 1 deletion(-)

diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml
index e8a7b461..7f0c2989 100644
--- a/ansible/roles/clickhouse_backup/defaults/main.yml
+++ b/ansible/roles/clickhouse_backup/defaults/main.yml
@@ -58,7 +58,6 @@ clickhouse_backup_max_bandwidth: 50000000
 clickhouse_backup_exclude_tables:
   - "*_tmp"               # updater staging tables
   - msmt_feedback_old
-  - counters_test_list    # unused materialized view, to be dropped
   - fastpath_tuning       # experiment on data2
 
 # Results for Prometheus, read by node_exporter's textfile collector

From 69bc63b043b79dda60c165818b988525176b3040 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 10:47:47 +0000
Subject: [PATCH 10/32] clickhouse_backup: say which RocksDB tables are and
 aren't backed up

The runbook listed fingerprints_* and session_expunge among the tables
whose rows each backup exports, but neither is backed up: the
fingerprints tables are only on data1 and data2 and the hourly updater
rebuilds them, and session_expunge is empty on both. Only test_groups,
on data3, is exported.
---
 docs/Runbooks.md | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index fe04034e..c019a72b 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -200,9 +200,12 @@ When a data1-only table becomes replicated, remove it from
   table starts a new chain with its first backup of each calendar month
   (UTC), which is the first Sunday of the month if the timer runs weekly.
   Once a new chain is complete, older chains are deleted unless pinned.
-- EmbeddedRocksDB and Join tables (`fingerprints_*`, `session_expunge`,
-  `test_groups`): `BACKUP` saves only their schema, so their rows are also
-  exported to `data.native.zst` inside each backup.
+- EmbeddedRocksDB and Join tables (`test_groups`): `BACKUP` saves only their
+  schema, so their rows are also exported to `data.native.zst` inside each
+  backup.
+- Not backed up: `fingerprints_dns` and `fingerprints_http`, which are only on
+  data1 and data2 and which the hourly updater rebuilds, and `session_expunge`,
+  which is empty and no longer used.
 - R2 rejects multipart uploads whose parts (all but the last) differ in
   size, so `clickhouse_backup_s3_settings` fixes ClickHouse's part size at
   128 MiB (`s3_upload_part_size_multiply_factor: 1`). R2 allows 10,000 parts,

From 2447141010a53a49cb016cf29a5a3c23e0175925 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 10:47:47 +0000
Subject: [PATCH 11/32] clickhouse_backup: stop excluding fastpath_tuning,
 which is dropped

The experiment table on data2 is gone, and data2 isn't backed up anyway.
---
 ansible/roles/clickhouse_backup/defaults/main.yml | 1 -
 1 file changed, 1 deletion(-)

diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml
index 7f0c2989..ccd578c9 100644
--- a/ansible/roles/clickhouse_backup/defaults/main.yml
+++ b/ansible/roles/clickhouse_backup/defaults/main.yml
@@ -58,7 +58,6 @@ clickhouse_backup_max_bandwidth: 50000000
 clickhouse_backup_exclude_tables:
   - "*_tmp"               # updater staging tables
   - msmt_feedback_old
-  - fastpath_tuning       # experiment on data2
 
 # Results for Prometheus, read by node_exporter's textfile collector
 # (node_exporter_textfile_dir in the prometheus_node_exporter role)

From 113451d19c01e87e5aafa1993971484e4f9272ba Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 11:04:35 +0000
Subject: [PATCH 12/32] clickhouse_backup: say why list prints instead of
 logging

The listing is the command's output, for stdout and pipes; log is for
what run and release report, to stderr and the journal.
---
 .../roles/clickhouse_backup/files/ooni_clickhouse_backup.py    | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
index 31fb628b..48372969 100644
--- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
+++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
@@ -559,6 +559,9 @@ def run(self, only: list, force_full: bool, keep: str) -> int:
         return 0
 
     def list(self, only: list) -> int:
+        # The listing is the command's output, so it goes to stdout with
+        # print(), to be read or piped (e.g. the latest: URL into a RESTORE);
+        # log is for what run and release report, to stderr and the journal.
         tables = only or sorted(
             n for n in self.store.subdirs(f"{self.store.prefix}/{self.db}/") if n
         )

From 7cab3dc158c8bdb92bc582d7b01fda8346fe18fb Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 11:08:37 +0000
Subject: [PATCH 13/32] clickhouse_backup: back up weekly, on Sundays

A full backup of everything takes more than a day, so run weekly
instead of daily for now: incrementals every Sunday at 02:30 UTC, each
on top of the one before, and a new full on the first Sunday of each
month. The timer's first run after a deploy is then the next Sunday;
the runbook says how to start the first full on deploy day.
---
 .../roles/clickhouse_backup/defaults/main.yml  |  8 ++++----
 docs/Runbooks.md                               | 18 +++++++++---------
 2 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml
index ccd578c9..4a59d13f 100644
--- a/ansible/roles/clickhouse_backup/defaults/main.yml
+++ b/ansible/roles/clickhouse_backup/defaults/main.yml
@@ -7,10 +7,10 @@
 
 # false stops and disables the timer, e.g. while upgrading the cluster
 clickhouse_backup_enabled: true
-# Daily, at a quiet time (UTC). A run backs up every job's tables, one job
-# after the other; a run still going (a new full backup of obs_web takes
-# more than a day) delays the next one instead of overlapping it.
-clickhouse_backup_on_calendar: "*-*-* 02:30:00"
+# Weekly, on Sundays at a quiet time (UTC): a full backup of everything takes
+# more than a day. A run backs up every job's tables, one job after the other;
+# a run still going delays the next one instead of overlapping it.
+clickhouse_backup_on_calendar: "Sun *-*-* 02:30:00"
 
 clickhouse_backup_database: ooni
 clickhouse_backup_clickhouse_url: "http://127.0.0.1:8123/"
diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index c019a72b..461528cb 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -195,10 +195,10 @@ When a data1-only table becomes replicated, remove it from
 `clickhouse_backup_data1_tables` and data3 picks it up.
 
 - `ooni-clickhouse-backup.timer` runs `/usr/local/bin/ooni-clickhouse-backup run`
-  daily at 02:30 UTC.
-- Each table has its own chain: a full backup, then daily incrementals. A
-  table starts a new chain with its first backup of each calendar month
-  (UTC), which is the first Sunday of the month if the timer runs weekly.
+  weekly, on Sundays at 02:30 UTC (`clickhouse_backup_on_calendar`).
+- Each table has its own chain: a full backup, then weekly incrementals, each
+  on top of the one before. A table starts a new chain with its first backup
+  of each calendar month (UTC): the first Sunday of the month.
   Once a new chain is complete, older chains are deleted unless pinned.
 - EmbeddedRocksDB and Join tables (`test_groups`): `BACKUP` saves only their
   schema, so their rows are also exported to `data.native.zst` inside each
@@ -215,8 +215,7 @@ When a data1-only table becomes replicated, remove it from
 - Bandwidth is capped with `clickhouse_backup_max_bandwidth`, at 50 MB/s. A
   new full backup of everything (~5.6 TB) takes about 31 hours at that rate.
   Against MinIO in testing, 24.8 uploaded at only about a third of the cap,
-  which would make it about 4 days, longer than `ClickHouseBackupStale`'s 3:
-  check the rate of the first backups
+  which would make it about 4 days: check the rate of the first backups
   (`ooni_clickhouse_backup_last_size_bytes / ooni_clickhouse_backup_last_duration_seconds`)
   and raise the cap if they're that slow.
 
@@ -248,9 +247,10 @@ without a restart.
    ./play -i inventory deploy-clickhouse.yml -l data1.htz-fsn.prod.ooni.nu,data3.htz-fsn.prod.ooni.nu -t clickhouse_backup --diff
    ```
    and the alert rules with `deploy-monitoring.yml`.
-4. The timer takes the first (full) backup at its next 02:30 UTC. A full
-   backup of everything takes about 39 hours, so to have it run over a
-   weekend, deploy on Friday after 02:30 UTC. To start one now instead:
+4. The timer takes the first (full) backup at its next run, on Sunday at
+   02:30 UTC. A full backup of everything takes about 31 hours, or longer if
+   the upload rate is below the cap. To start it on deploy day instead, e.g. a
+   Friday so that it runs over the weekend:
    ```bash
    sudo systemctl start --no-block ooni-clickhouse-backup.service
    sudo journalctl -f --identifier ooni-clickhouse-backup

From f1305c615acfcf26e0d31b61a4212b23e2b857eb Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 11:08:37 +0000
Subject: [PATCH 14/32] clickhouse_backup: allow a week between backups before
 ClickHouseBackupStale

With weekly runs a table's last backup is normally up to a week old, so
ClickHouseBackupStale fired after 3 days would fire every week. It now
fires after 8 days, and not while the table's job is still running: on
the first Sunday of a month the fulls take more than a day, and the
largest tables come last. The promtool tests cover both.
---
 .../clickhouse_backup/tests/alert_rules_test.yml      | 11 ++++++++---
 ansible/roles/prometheus/files/alert_rules.yml        |  8 ++++++--
 docs/Runbooks.md                                      |  3 ++-
 3 files changed, 16 insertions(+), 6 deletions(-)

diff --git a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
index ce753a0f..85e47052 100644
--- a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
+++ b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
@@ -10,12 +10,17 @@ tests:
       - {series: 'ooni_clickhouse_backup_last_run_failed_tables{instance="data3", backup="measurements"}', values: '0x180'}
       - {series: 'ooni_clickhouse_backup_last_run_tables{instance="data3", backup="measurements"}', values: '14x180'}
       - {series: 'ooni_clickhouse_backup_last_run_full_backups{instance="data3", backup="measurements"}', values: '1x180'}
-      # private finished at the same time with a table failed: fastpath, last backed up 4 days ago
+      # private finished at the same time with a table failed: fastpath, last backed up 9 days ago.
+      # Test time starts at 0 (1970-01-01), so a timestamp 9 days before it is -9 * 86400.
       - {series: 'ooni_clickhouse_backup_last_run_finished_timestamp_seconds{instance="data3", backup="private"}', values: '0x180'}
       - {series: 'ooni_clickhouse_backup_last_run_duration_seconds{instance="data3", backup="private"}', values: '60x180'}
       - {series: 'ooni_clickhouse_backup_last_run_failed_tables{instance="data3", backup="private"}', values: '1x180'}
       - {series: 'ooni_clickhouse_backup_last_attempt_failed{instance="data3", backup="private", table="fastpath"}', values: '1x180'}
-      - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data3", backup="private", table="fastpath"}', values: '-345600x180'}
+      - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data3", backup="private", table="fastpath"}', values: '-777600x180'}
+      - {series: 'ooni_clickhouse_backup_running{instance="data3", backup="private"}', values: '0x180'}
+      # measurements' obs_web was also last backed up 9 days ago, but its job is still running
+      - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data3", backup="measurements", table="obs_web"}', values: '-777600x180'}
+      - {series: 'ooni_clickhouse_backup_running{instance="data3", backup="measurements"}', values: '1x180'}
     alert_rule_test:
       - eval_time: 10m
         alertname: ClickHouseBackupCompleted
@@ -40,5 +45,5 @@ tests:
         exp_alerts:
           - exp_labels: {instance: data3, backup: private, table: fastpath}
             exp_annotations:
-              summary: 'No ClickHouse backup of fastpath (private) for 4d 0h 10m 0s'
+              summary: 'No ClickHouse backup of fastpath (private) for 9d 0h 10m 0s'
               description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on data3.'
diff --git a/ansible/roles/prometheus/files/alert_rules.yml b/ansible/roles/prometheus/files/alert_rules.yml
index 66eb2e3c..54ed39e3 100644
--- a/ansible/roles/prometheus/files/alert_rules.yml
+++ b/ansible/roles/prometheus/files/alert_rules.yml
@@ -168,9 +168,13 @@ groups:
       summary: 'ClickHouse backup {{ $labels.backup }} on {{ $labels.instance }} completed in {{ $value | humanizeDuration }}'
       description: 'Tables backed up: {{ with printf `ooni_clickhouse_backup_last_run_tables{instance="%s",backup="%s"}` $labels.instance $labels.backup | query }}{{ . | first | value }}{{ end }}, in full: {{ with printf `ooni_clickhouse_backup_last_run_full_backups{instance="%s",backup="%s"}` $labels.instance $labels.backup | query }}{{ . | first | value }}{{ end }}. Nothing to do.'
 
-  # Backups run daily; a new full backup of obs_web can take a day and a half
+  # Backups run weekly, so a table's last one is normally up to a week old. Not
+  # while its job runs: on the first Sunday of a month the full backups take
+  # more than a day, and the largest tables come last.
   - alert: ClickHouseBackupStale
-    expr: time() - ooni_clickhouse_backup_last_success_timestamp_seconds > 3 * 86400
+    expr: |
+      time() - ooni_clickhouse_backup_last_success_timestamp_seconds > 8 * 86400
+        and on (instance, backup) ooni_clickhouse_backup_running == 0
     annotations:
       summary: 'No ClickHouse backup of {{ $labels.table }} ({{ $labels.backup }}) for {{ $value | humanizeDuration }}'
       description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on {{ $labels.instance }}.'
diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index 461528cb..61e0cf4f 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -266,7 +266,8 @@ Alerts, in `alert_rules.yml`:
   deleted, and the next run retries it.
 - `ClickHouseBackupCompleted`: a job's run backed up every table. One Slack
   message per run in `#ooni-bots`, no email.
-- `ClickHouseBackupStale`: a table hasn't been backed up for 3 days.
+- `ClickHouseBackupStale`: a table hasn't been backed up for 8 days (backups
+  are weekly), and its job isn't running.
 - `ClickHouseBackupMissing`: data1 or data3 hasn't written results for a day.
 - `ClickHouseBackupHostLoadHigh`: data3's 5-minute load is over half its
   CPUs for 15 minutes.

From 83fe34e9e245c63491b60294996324be4b54922d Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 12:00:53 +0000
Subject: [PATCH 15/32] clickhouse_backup: hold back ClickHouseBackupStale only
 for a recent run on the host

The alert was held back only while the table's own job ran. On data3 the
jobs run one after the other, so on the first Sunday of a month private
waited more than a day behind measurements' full backups and passed 8
days without one, while nothing was wrong. And a run killed without
reaching its finally left running at 1, which held the alert back for
good.

The script now also exports, per job, when its current run started
(ooni_clickhouse_backup_running_since_timestamp_seconds, 0 when idle).
The alert is held back while any job on the host is in a run that
started in the last 5 days. The promtool tests cover a 7- and a 9-day
old table with nothing running, a job waiting behind another job's run,
and a stuck run; test_job.py the new metric and older state files.
---
 .../files/ooni_clickhouse_backup.py           |  8 +++
 .../roles/clickhouse_backup/tests/README.md   |  1 +
 .../tests/alert_rules_test.yml                | 38 ++++++++---
 ansible/roles/clickhouse_backup/tests/run.sh  |  2 +-
 .../roles/clickhouse_backup/tests/test_job.py | 67 +++++++++++++++++++
 .../roles/prometheus/files/alert_rules.yml    |  8 ++-
 docs/Runbooks.md                              |  2 +-
 7 files changed, 110 insertions(+), 16 deletions(-)
 create mode 100644 ansible/roles/clickhouse_backup/tests/test_job.py

diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
index 48372969..3ad742e4 100644
--- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
+++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
@@ -279,6 +279,7 @@ def __init__(self, metrics_file: str = "", state_file: str = ""):
             with open(state_file) as f:
                 self.state = json.load(f)
         self.state.setdefault("runs", {})
+        self.state.setdefault("running_since", {})
 
     def table_result(self, job: str, table: str, ok: bool, st: dict = None, duration_s: float = 0) -> None:
         entry = self.state["tables"].setdefault(f"{job}/{table}", {"job": job, "table": table})
@@ -294,6 +295,7 @@ def table_result(self, job: str, table: str, ok: bool, st: dict = None, duration
 
     def job_running(self, job: str, running: bool) -> None:
         self.state["running"][job] = 1 if running else 0
+        self.state["running_since"][job] = time.time() if running else 0
         self.save()
 
     def run_result(self, job: str, duration_s: float, tables: int, fulls: int, failed: int) -> None:
@@ -311,6 +313,12 @@ def render(self) -> str:
         ]
         for job, v in sorted(self.state["running"].items()):
             lines.append(f'{p}running{{backup="{label_value(job)}"}} {v}')
+        lines += [
+            f"# HELP {p}running_since_timestamp_seconds When the job's current run started, 0 if none",
+            f"# TYPE {p}running_since_timestamp_seconds gauge",
+        ]
+        for job, v in sorted(self.state["running_since"].items()):
+            lines.append(f'{p}running_since_timestamp_seconds{{backup="{label_value(job)}"}} {v}')
         for name, field, help_ in self.RUN_GAUGES:
             lines += [f"# HELP {p}{name} {help_}", f"# TYPE {p}{name} gauge"]
             for job, run in sorted(self.state["runs"].items()):
diff --git a/ansible/roles/clickhouse_backup/tests/README.md b/ansible/roles/clickhouse_backup/tests/README.md
index 6a5c9fc1..4d548799 100644
--- a/ansible/roles/clickhouse_backup/tests/README.md
+++ b/ansible/roles/clickhouse_backup/tests/README.md
@@ -8,6 +8,7 @@
 - `test_schedule.py`: which runs start a full backup and what gets pruned,
   over months of daily and weekly runs with failures, ClickHouse and the
   bucket stubbed.
+- `test_job.py`: how a job runs and what it reports, ClickHouse stubbed.
 - `test_e2e.py`: the real thing, in the containers of `compose.yml`:
   ClickHouse at production's version, MinIO standing in for R2, the script
   on Debian bookworm with its python3-boto3, as on the hosts, node_exporter,
diff --git a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
index 85e47052..323c12b0 100644
--- a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
+++ b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
@@ -1,7 +1,9 @@
-# promtool unit tests for the ClickHouse backup alerts in alert_rules.yml
+# promtool unit tests for the ClickHouse backup alerts in alert_rules.yml. Test time starts at 0
+# (1970-01-01), so a timestamp N days before the test starts is -N * 86400.
 rule_files: [alert_rules.yml]
 evaluation_interval: 1m
 tests:
+  # Completed and Failed
   - interval: 1m
     input_series:
       # measurements finished at t=0 after 30 min, all 14 tables backed up, 1 in full
@@ -10,17 +12,11 @@ tests:
       - {series: 'ooni_clickhouse_backup_last_run_failed_tables{instance="data3", backup="measurements"}', values: '0x180'}
       - {series: 'ooni_clickhouse_backup_last_run_tables{instance="data3", backup="measurements"}', values: '14x180'}
       - {series: 'ooni_clickhouse_backup_last_run_full_backups{instance="data3", backup="measurements"}', values: '1x180'}
-      # private finished at the same time with a table failed: fastpath, last backed up 9 days ago.
-      # Test time starts at 0 (1970-01-01), so a timestamp 9 days before it is -9 * 86400.
+      # private finished at the same time, with fastpath failed
       - {series: 'ooni_clickhouse_backup_last_run_finished_timestamp_seconds{instance="data3", backup="private"}', values: '0x180'}
       - {series: 'ooni_clickhouse_backup_last_run_duration_seconds{instance="data3", backup="private"}', values: '60x180'}
       - {series: 'ooni_clickhouse_backup_last_run_failed_tables{instance="data3", backup="private"}', values: '1x180'}
       - {series: 'ooni_clickhouse_backup_last_attempt_failed{instance="data3", backup="private", table="fastpath"}', values: '1x180'}
-      - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data3", backup="private", table="fastpath"}', values: '-777600x180'}
-      - {series: 'ooni_clickhouse_backup_running{instance="data3", backup="private"}', values: '0x180'}
-      # measurements' obs_web was also last backed up 9 days ago, but its job is still running
-      - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data3", backup="measurements", table="obs_web"}', values: '-777600x180'}
-      - {series: 'ooni_clickhouse_backup_running{instance="data3", backup="measurements"}', values: '1x180'}
     alert_rule_test:
       - eval_time: 10m
         alertname: ClickHouseBackupCompleted
@@ -40,10 +36,30 @@ tests:
             exp_annotations:
               summary: 'ClickHouse backup of fastpath (private) failed on data3'
               description: 'Nothing was deleted: the older backups of fastpath are kept. The next run retries it. See `journalctl --identifier ooni-clickhouse-backup` on data3.'
+
+  # Stale: after 8 days, unless a job on the same host is in a run started in the last 5 days
+  - interval: 1m
+    input_series:
+      # data1: nothing running; one table backed up 9 days ago, one 7 days ago
+      - {series: 'ooni_clickhouse_backup_running_since_timestamp_seconds{instance="data1", backup="data1"}', values: '0x30'}
+      - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data1", backup="data1", table="nine_days"}', values: '-777600x30'}
+      - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data1", backup="data1", table="seven_days"}', values: '-604800x30'}
+      # data3: measurements' monthly full started 2 days ago; private, waiting behind it, last ran 9 days ago
+      - {series: 'ooni_clickhouse_backup_running_since_timestamp_seconds{instance="data3", backup="measurements"}', values: '-172800x30'}
+      - {series: 'ooni_clickhouse_backup_running_since_timestamp_seconds{instance="data3", backup="private"}', values: '0x30'}
+      - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="data3", backup="private", table="msmt_feedback"}', values: '-777600x30'}
+      # stuck: a run that "started" 6 days ago, after the script was killed; its table is 9 days old
+      - {series: 'ooni_clickhouse_backup_running_since_timestamp_seconds{instance="stuck", backup="measurements"}', values: '-518400x30'}
+      - {series: 'ooni_clickhouse_backup_last_success_timestamp_seconds{instance="stuck", backup="measurements", table="fastpath"}', values: '-777600x30'}
+    alert_rule_test:
       - eval_time: 10m
         alertname: ClickHouseBackupStale
         exp_alerts:
-          - exp_labels: {instance: data3, backup: private, table: fastpath}
+          - exp_labels: {instance: data1, backup: data1, table: nine_days}
+            exp_annotations:
+              summary: 'No ClickHouse backup of nine_days (data1) for 9d 0h 10m 0s'
+              description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on data1.'
+          - exp_labels: {instance: stuck, backup: measurements, table: fastpath}
             exp_annotations:
-              summary: 'No ClickHouse backup of fastpath (private) for 9d 0h 10m 0s'
-              description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on data3.'
+              summary: 'No ClickHouse backup of fastpath (measurements) for 9d 0h 10m 0s'
+              description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on stuck.'
diff --git a/ansible/roles/clickhouse_backup/tests/run.sh b/ansible/roles/clickhouse_backup/tests/run.sh
index f2491da8..51b2295a 100755
--- a/ansible/roles/clickhouse_backup/tests/run.sh
+++ b/ansible/roles/clickhouse_backup/tests/run.sh
@@ -26,7 +26,7 @@ route "alertname=ClickHouseBackupCompleted instance=data3 backup=measurements" s
 route "alertname=ClickHouseBackupFailed instance=data3 backup=measurements table=fastpath" team-all
 
 echo "== schedule"
-python3 -m unittest test_schedule
+python3 -m unittest test_schedule test_job
 
 echo "== end to end"
 [ -n "${KEEP:-}" ] || trap 'compose down -v --remove-orphans >/dev/null 2>&1' EXIT
diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py
new file mode 100644
index 00000000..76526d0a
--- /dev/null
+++ b/ansible/roles/clickhouse_backup/tests/test_job.py
@@ -0,0 +1,67 @@
+"""How a job runs and what it reports, with ClickHouse stubbed; test_e2e.py runs the real thing.
+    python3 -m unittest test_job"""
+import importlib.util
+import json
+import os
+import tempfile
+import unittest
+import urllib.error
+
+spec = importlib.util.spec_from_file_location(
+    "backup", os.path.join(os.path.dirname(__file__), "..", "files", "ooni_clickhouse_backup.py")
+)
+backup = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(backup)
+
+
+class FlakyClickHouse:
+    """Fails the first `failures` calls with a connection error, then answers"""
+
+    def __init__(self, failures=0, status="BACKUP_CREATED"):
+        self.failures, self.status, self.calls = failures, status, 0
+
+    def query(self, sql, settings=None, timeout=600):
+        self.calls += 1
+        if self.calls <= self.failures:
+            raise urllib.error.URLError("Connection refused")
+        return ""
+
+    def rows(self, sql, **kw):
+        self.query(sql)
+        if "system.backups WHERE id" in sql:
+            return [{"status": self.status, "error": "", "num_files": 1, "total_size": 1, "uncompressed_size": 1}]
+        return []
+
+
+def job(name="measurements", include=(), exclude=(), metrics=None, ch=None):
+    j = backup.BackupJob.__new__(backup.BackupJob)
+    j.name, j.include, j.exclude = name, list(include), list(exclude)
+    j.metrics = metrics or backup.Metrics()
+    j.ch = ch or FlakyClickHouse()
+    j.poll_s, j.clickhouse_wait_s = 0, 0
+    return j
+
+
+class TestRunningSince(unittest.TestCase):
+    def setUp(self):
+        d = tempfile.mkdtemp()
+        self.metrics = backup.Metrics(os.path.join(d, "out.prom"), os.path.join(d, "state.json"))
+
+    def test_running_since_is_set_while_a_job_runs(self):
+        self.metrics.job_running("measurements", True)
+        self.assertGreater(self.metrics.state["running_since"]["measurements"], 0)
+        with open(self.metrics.metrics_file) as f:
+            self.assertIn('ooni_clickhouse_backup_running_since_timestamp_seconds{backup="measurements"}', f.read())
+        self.metrics.job_running("measurements", False)
+        self.assertEqual(self.metrics.state["running_since"]["measurements"], 0)
+
+    def test_old_state_files_still_load(self):
+        with open(self.metrics.state_file, "w") as f:
+            json.dump({"tables": {}, "running": {"measurements": 0}}, f)
+        m = backup.Metrics(self.metrics.metrics_file, self.metrics.state_file)
+        m.job_running("measurements", True)
+        self.assertIn("measurements", m.state["running_since"])
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/ansible/roles/prometheus/files/alert_rules.yml b/ansible/roles/prometheus/files/alert_rules.yml
index 54ed39e3..a0b86b2b 100644
--- a/ansible/roles/prometheus/files/alert_rules.yml
+++ b/ansible/roles/prometheus/files/alert_rules.yml
@@ -169,12 +169,14 @@ groups:
       description: 'Tables backed up: {{ with printf `ooni_clickhouse_backup_last_run_tables{instance="%s",backup="%s"}` $labels.instance $labels.backup | query }}{{ . | first | value }}{{ end }}, in full: {{ with printf `ooni_clickhouse_backup_last_run_full_backups{instance="%s",backup="%s"}` $labels.instance $labels.backup | query }}{{ . | first | value }}{{ end }}. Nothing to do.'
 
   # Backups run weekly, so a table's last one is normally up to a week old. Not
-  # while its job runs: on the first Sunday of a month the full backups take
-  # more than a day, and the largest tables come last.
+  # while any job on the host is in a run that started in the last 5 days: on
+  # the first Sunday of a month the full backups take more than a day, and the
+  # jobs run one after the other. A run older than that is stuck (e.g. the
+  # script was killed and left running at 1), and no longer holds the alert back.
   - alert: ClickHouseBackupStale
     expr: |
       time() - ooni_clickhouse_backup_last_success_timestamp_seconds > 8 * 86400
-        and on (instance, backup) ooni_clickhouse_backup_running == 0
+        unless on (instance) (time() - (ooni_clickhouse_backup_running_since_timestamp_seconds != 0) < 5 * 86400)
     annotations:
       summary: 'No ClickHouse backup of {{ $labels.table }} ({{ $labels.backup }}) for {{ $value | humanizeDuration }}'
       description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on {{ $labels.instance }}.'
diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index 61e0cf4f..307a8048 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -267,7 +267,7 @@ Alerts, in `alert_rules.yml`:
 - `ClickHouseBackupCompleted`: a job's run backed up every table. One Slack
   message per run in `#ooni-bots`, no email.
 - `ClickHouseBackupStale`: a table hasn't been backed up for 8 days (backups
-  are weekly), and its job isn't running.
+  are weekly), and no job on its host is in a run started in the last 5 days.
 - `ClickHouseBackupMissing`: data1 or data3 hasn't written results for a day.
 - `ClickHouseBackupHostLoadHigh`: data3's 5-minute load is over half its
   CPUs for 15 minutes.

From 6ea1f2f7dda39a455575df4bf794c5f37bc408e5 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 12:00:53 +0000
Subject: [PATCH 16/32] clickhouse_backup: run --table runs only the jobs that
 back up those tables

The other jobs ran with no tables, recorded a completed run of 0 tables
(ClickHouseBackupCompleted posted it to Slack) and overwrote their real
weekly results.
---
 .../files/ooni_clickhouse_backup.py                  | 12 ++++++++++--
 ansible/roles/clickhouse_backup/tests/test_job.py    | 12 ++++++++++++
 2 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
index 3ad742e4..52f8b852 100644
--- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
+++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
@@ -602,6 +602,13 @@ def release(self, label: str) -> int:
         return 0
 
 
+def jobs_for_tables(jobs: list, tables: list) -> tuple:
+    """(the jobs that back up one of tables, the tables no job backs up)"""
+    owns = lambda j, t: j.includes(t) and not j.excludes(t)
+    orphans = [t for t in tables if not any(owns(j, t) for j in jobs)]
+    return [j for j in jobs if any(owns(j, t) for t in tables)], orphans
+
+
 def job_confs(conf: dict, names: list) -> list:
     """Each job's config: the top-level keys, overridden by the job's own.
 
@@ -655,8 +662,9 @@ def main() -> int:
             job.list(args.table)
         return 0
 
-    if args.command == "run":
-        orphans = [t for t in args.table if not any(j.includes(t) and not j.excludes(t) for j in jobs)]
+    if args.command == "run" and args.table:
+        # only these tables' jobs: the others would run with no tables, and report that
+        jobs, orphans = jobs_for_tables(jobs, args.table)
         if orphans:
             ap.error(f"no job backs up: {', '.join(orphans)}")
 
diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py
index 76526d0a..63424a66 100644
--- a/ansible/roles/clickhouse_backup/tests/test_job.py
+++ b/ansible/roles/clickhouse_backup/tests/test_job.py
@@ -63,5 +63,17 @@ def test_old_state_files_still_load(self):
         self.assertIn("measurements", m.state["running_since"])
 
 
+class TestRunTable(unittest.TestCase):
+    def test_only_the_jobs_that_back_up_the_tables(self):
+        jobs = [job("measurements", exclude=["msmt_feedback"]), job("private", include=["msmt_feedback"])]
+        selected, orphans = backup.jobs_for_tables(jobs, ["msmt_feedback"])
+        self.assertEqual([j.name for j in selected], ["private"])
+        self.assertEqual(orphans, [])
+
+    def test_tables_no_job_backs_up(self):
+        jobs = [job("private", include=["msmt_feedback"])]
+        self.assertEqual(backup.jobs_for_tables(jobs, ["fastpath"]), ([], ["fastpath"]))
+
+
 if __name__ == "__main__":
     unittest.main()

From 61ff892a7863a1734d0906eecf6dc7220dfbf8ff Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 12:00:54 +0000
Subject: [PATCH 17/32] clickhouse_backup: don't let a brief ClickHouse outage
 abort a run

At boot, Persistent=true catches up a missed Sunday before ClickHouse
accepts queries. The checks before the table loop then failed outside
any try: the run died with a traceback, skipped the later jobs and
wrote no metrics, and nothing alerted until ClickHouseBackupStale 8
days later. A run now waits up to 10 minutes for ClickHouse
(clickhouse_wait_seconds). If it still can't start, the job's tables
are marked failed, so ClickHouseBackupFailed says so, and the next job
runs.

One failed status poll also gave up on a backup that can run for days,
while the BACKUP carried on server-side. Polls are now retried for up to
10 minutes.
---
 .../files/ooni_clickhouse_backup.py           | 63 +++++++++++++++----
 .../roles/clickhouse_backup/tests/test_job.py | 29 +++++++++
 2 files changed, 80 insertions(+), 12 deletions(-)

diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
index 52f8b852..a045594a 100644
--- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
+++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
@@ -298,6 +298,14 @@ def job_running(self, job: str, running: bool) -> None:
         self.state["running_since"][job] = time.time() if running else 0
         self.save()
 
+    def job_failed(self, job: str) -> None:
+        """The job couldn't start: its tables' last attempts failed"""
+        for entry in self.state["tables"].values():
+            if entry["job"] == job:
+                entry["last_attempt"] = time.time()
+                entry["last_failed"] = 1
+        self.save()
+
     def run_result(self, job: str, duration_s: float, tables: int, fulls: int, failed: int) -> None:
         self.state["runs"][job] = {
             "finished": time.time(), "duration": round(duration_s, 1),
@@ -371,6 +379,7 @@ def __init__(self, conf: dict, metrics: Metrics = None):
         self.include = conf.get("include_tables", [])
         self.exclude = conf.get("exclude_tables", [])
         self.poll_s = float(conf.get("poll_interval_seconds", 30))
+        self.clickhouse_wait_s = float(conf.get("clickhouse_wait_seconds", 600))
         # Query settings for the uploads, e.g. R2 needs every part of a
         # multipart upload but the last to be the same size
         self.settings = {k: str(v) for k, v in conf.get("s3_settings", {}).items()}
@@ -435,16 +444,41 @@ def running_backups(self) -> list:
         )
         return [r for r in rows if self.store.ch_base in r["name"]]
 
+    def wait_for_clickhouse(self) -> None:
+        """At boot a missed run starts before ClickHouse accepts queries"""
+        t0 = time.monotonic()
+        while True:
+            try:
+                self.ch.query("SELECT 1", timeout=30)
+                return
+            except (OSError, RuntimeError) as e:
+                if time.monotonic() - t0 >= self.clickhouse_wait_s:
+                    raise RuntimeError(f"ClickHouse not answering after {self.clickhouse_wait_s:.0f}s: {e}")
+                log.info(f"waiting for ClickHouse: {e}")
+                time.sleep(10)
+
     def wait(self, op_id: str, what: str) -> dict:
         t0 = time.monotonic()
         last_log = t0
+        failing_since = None
         while True:
-            rows = self.ch.rows(
-                f"""
-                SELECT status, error, num_files, total_size, uncompressed_size
-                FROM system.backups WHERE id = {sql_str(op_id)}
-                """
-            )
+            # A backup can run for days: retry a failed poll for a while
+            # rather than give up on a BACKUP that is still running
+            try:
+                rows = self.ch.rows(
+                    f"""
+                    SELECT status, error, num_files, total_size, uncompressed_size
+                    FROM system.backups WHERE id = {sql_str(op_id)}
+                    """
+                )
+                failing_since = None
+            except (OSError, RuntimeError) as e:
+                failing_since = failing_since or time.monotonic()
+                if time.monotonic() - failing_since > 600:
+                    raise
+                log.warning(f"{what}: polling failed, retrying: {e}")
+                time.sleep(self.poll_s)
+                continue
             if not rows:
                 raise RuntimeError(f"{what}: gone from system.backups (server restart?)")
             st = rows[0]
@@ -537,14 +571,19 @@ def backup_table(self, table: str, engine: str, force_full: bool, keep: str) ->
         return full
 
     def run(self, only: list, force_full: bool, keep: str) -> int:
-        running = self.running_backups()
-        if running:
-            for r in running:
-                log.error(f"backup {r['id']} is still running on the server")
-            log.error("not starting: wait for it to finish, or for the server to drop it")
+        try:
+            self.wait_for_clickhouse()
+            running = self.running_backups()
+            if running:
+                for r in running:
+                    log.error(f"backup {r['id']} is still running on the server")
+                raise RuntimeError("not starting: wait for it to finish, or for the server to drop it")
+            tables, failed = self.tables(only)
+        except Exception as e:
+            log.error(f"{self.name}: could not start: {e}")
+            self.metrics.job_failed(self.name)
             return 1
 
-        tables, failed = self.tables(only)
         t0, done, fulls = time.monotonic(), 0, 0
         self.metrics.job_running(self.name, True)
         try:
diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py
index 63424a66..daec61c5 100644
--- a/ansible/roles/clickhouse_backup/tests/test_job.py
+++ b/ansible/roles/clickhouse_backup/tests/test_job.py
@@ -6,6 +6,7 @@
 import tempfile
 import unittest
 import urllib.error
+from unittest import mock
 
 spec = importlib.util.spec_from_file_location(
     "backup", os.path.join(os.path.dirname(__file__), "..", "files", "ooni_clickhouse_backup.py")
@@ -75,5 +76,33 @@ def test_tables_no_job_backs_up(self):
         self.assertEqual(backup.jobs_for_tables(jobs, ["fastpath"]), ([], ["fastpath"]))
 
 
+class TestClickHouseErrors(unittest.TestCase):
+    def setUp(self):
+        d = tempfile.mkdtemp()
+        self.metrics = backup.Metrics(os.path.join(d, "out.prom"), os.path.join(d, "state.json"))
+
+    def test_waits_for_clickhouse_at_start(self):
+        j = job(metrics=self.metrics, ch=FlakyClickHouse(failures=2))
+        j.clickhouse_wait_s = 60
+        with mock.patch.object(backup.time, "sleep"):
+            j.wait_for_clickhouse()
+        self.assertEqual(j.ch.calls, 3)
+
+    def test_a_job_that_cant_start_marks_its_tables_failed(self):
+        self.metrics.table_result("measurements", "fastpath", True, {"total_size": 1, "num_files": 1})
+        self.metrics.table_result("private", "msmt_feedback", True, {"total_size": 1, "num_files": 1})
+        j = job(metrics=self.metrics, ch=FlakyClickHouse(failures=1000))
+        self.assertEqual(j.run([], False, ""), 1)
+        tables = self.metrics.state["tables"]
+        self.assertEqual(tables["measurements/fastpath"]["last_failed"], 1)
+        self.assertEqual(tables["private/msmt_feedback"]["last_failed"], 0, "another job's table")
+        self.assertNotIn("measurements", self.metrics.state["runs"], "no completed run to report")
+
+    def test_a_failed_poll_is_retried(self):
+        j = job(ch=FlakyClickHouse(failures=2))
+        self.assertEqual(j.wait("op", "obs_web")["status"], "BACKUP_CREATED")
+        self.assertEqual(j.ch.calls, 3)
+
+
 if __name__ == "__main__":
     unittest.main()

From 2e2fa2efe849b871d05801e2f944287e7996713d Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 12:01:06 +0000
Subject: [PATCH 18/32] clickhouse_backup: wait 8 days before
 ClickHouseBackupMissing

A host has no backup results until its first run, which with weekly
runs can be up to a week after the role is deployed: the 26 h wait
alerted every 3 hours until then.
---
 .../clickhouse_backup/tests/alert_rules_test.yml    | 13 +++++++++++++
 ansible/roles/prometheus/files/alert_rules.yml      |  7 ++++---
 docs/Runbooks.md                                    |  3 ++-
 3 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
index 323c12b0..08e03fe5 100644
--- a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
+++ b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
@@ -63,3 +63,16 @@ tests:
             exp_annotations:
               summary: 'No ClickHouse backup of fastpath (measurements) for 9d 0h 10m 0s'
               description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on stuck.'
+
+  # Missing: data1 has results, data3 never wrote any; it fires after 8 days
+  - interval: 1h
+    input_series:
+      - {series: 'ooni_clickhouse_backup_running{instance="data1.htz-fsn.prod.ooni.nu", backup="data1"}', values: '0x240'}
+    alert_rule_test:
+      - {eval_time: 7d, alertname: ClickHouseBackupMissing, exp_alerts: []}
+      - eval_time: 9d
+        alertname: ClickHouseBackupMissing
+        exp_alerts:
+          - exp_labels: {instance: data3.htz-fsn.prod.ooni.nu}
+            exp_annotations:
+              summary: 'No ClickHouse backup results from data3.htz-fsn.prod.ooni.nu for over a week'
diff --git a/ansible/roles/prometheus/files/alert_rules.yml b/ansible/roles/prometheus/files/alert_rules.yml
index a0b86b2b..aae32954 100644
--- a/ansible/roles/prometheus/files/alert_rules.yml
+++ b/ansible/roles/prometheus/files/alert_rules.yml
@@ -181,12 +181,13 @@ groups:
       summary: 'No ClickHouse backup of {{ $labels.table }} ({{ $labels.backup }}) for {{ $value | humanizeDuration }}'
       description: 'Check `systemctl status ooni-clickhouse-backup.timer` and the journal on {{ $labels.instance }}.'
 
-  # The job hasn't written any result since its host was set up, or the file is gone
+  # The job hasn't written any result since its host was set up, or the file is
+  # gone. Backups run weekly, so a new host has none until its first Sunday.
   - alert: ClickHouseBackupMissing
     expr: absent(ooni_clickhouse_backup_running{instance="data3.htz-fsn.prod.ooni.nu"}) or absent(ooni_clickhouse_backup_running{instance="data1.htz-fsn.prod.ooni.nu"})
-    for: 26h
+    for: 8d
     annotations:
-      summary: 'No ClickHouse backup results from data1 or data3 for over a day'
+      summary: 'No ClickHouse backup results from {{ $labels.instance }} for over a week'
 
   # data3 takes the backups. Above half its CPUs (load per CPU > 0.5), the
   # backup or merges are crowding the node; lower clickhouse_backup_max_bandwidth
diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index 307a8048..0fc49adc 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -268,7 +268,8 @@ Alerts, in `alert_rules.yml`:
   message per run in `#ooni-bots`, no email.
 - `ClickHouseBackupStale`: a table hasn't been backed up for 8 days (backups
   are weekly), and no job on its host is in a run started in the last 5 days.
-- `ClickHouseBackupMissing`: data1 or data3 hasn't written results for a day.
+- `ClickHouseBackupMissing`: data1 or data3 hasn't written any results for 8
+  days, e.g. since the role was deployed and before its first Sunday run.
 - `ClickHouseBackupHostLoadHigh`: data3's 5-minute load is over half its
   CPUs for 15 minutes.
 

From 79660f27a9f5b8a2648c4bde180564e67c164984 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 12:01:19 +0000
Subject: [PATCH 19/32] clickhouse_backup runbook: start the post-upgrade full
 as a service, restore test_groups

Before upgrading, step 4 re-enabled the timer first, which with
Persistent=true may start a run for a missed Sunday, then ran a
multi-day run --full in a shell. Run it with systemd-run while the
timer is still disabled, and re-enable the timer afterwards. Note that
ClickHouse waits for running backups when it stops
(shutdown_wait_backups_and_restores, TimeoutStopSec=infinity), so a
restart during a full hangs until it is done.

The restore example loaded fingerprints_dns, which isn't backed up;
test_groups is the table it's for.
---
 docs/Runbooks.md | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)

diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index 0fc49adc..d2735a9a 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -296,17 +296,28 @@ before that step, so:
 
 1. Disable the timer with `clickhouse_backup_enabled: false` (or
    `sudo systemctl stop ooni-clickhouse-backup.timer`) and check that
-   `system.backups` shows nothing running.
-2. Take a pinned backup:
+   `system.backups` shows nothing running. ClickHouse waits for running
+   backups when it stops (`shutdown_wait_backups_and_restores`, and the
+   unit's `TimeoutStopSec=infinity`), so restarting a node during a full
+   backup hangs until the backup is done, a day or more.
+2. Take a pinned backup (incrementals, minutes):
    ```bash
    sudo ooni-clickhouse-backup run --keep pre-25.8.29.51
    ```
 3. Upgrade all three nodes. Don't run backups while nodes are on different
    versions.
-4. Re-enable the timer. Once the whole upgrade is confirmed good, start
-   new chains on the new version and unpin the old ones:
+4. Once the whole upgrade is confirmed good, and with the timer still
+   disabled (once enabled, it may start a run straight away for a missed
+   Sunday), start new chains on the new version. That takes a day or more,
+   so run it as a service rather than in a shell:
+   ```bash
+   sudo systemd-run --unit=ooni-clickhouse-backup-full -p TimeoutStartSec=infinity \
+     -p SyslogIdentifier=ooni-clickhouse-backup /usr/local/bin/ooni-clickhouse-backup run --full
+   sudo journalctl -f --identifier ooni-clickhouse-backup
+   ```
+   When the journal shows `all tables backed up` for every job, unpin the
+   old chains and re-enable the timer (`clickhouse_backup_enabled: true`):
    ```bash
-   sudo ooni-clickhouse-backup run --full
    sudo ooni-clickhouse-backup release pre-25.8.29.51
    ```
 
@@ -329,9 +340,9 @@ each table. Restoring the latest incremental restores the whole chain.
    ```sql
    RESTORE TABLE ooni.obs_web FROM S3('https://<account id>.r2.cloudflarestorage.com/ooni-clickhouse-backup/prod/ooni/obs_web/<chain>/<backup>')
    ```
-3. For EmbeddedRocksDB and Join tables, load the rows too:
+3. For EmbeddedRocksDB and Join tables (`test_groups`), load the rows too:
    ```sql
-   INSERT INTO ooni.fingerprints_dns
+   INSERT INTO ooni.test_groups
    SELECT * FROM s3('<backup URL>/data.native.zst', 'Native')
    ```
 4. On the other replicas, create replicated tables with the same

From a16960b19ea4331e8eff47bbea671239b9ba4954 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 12:01:31 +0000
Subject: [PATCH 20/32] prometheus: make TextfileJam's exclusion match the
 backup file's path, test the backup host alert

node_exporter's file label is the file's full path, so the anchored
"ooni_clickhouse_backup.*" never matched (TextfileJam matches nothing
today anyway: it uses node_exporter's pre-0.16 metric names). The
promtool tests now cover the exclusion, and
ClickHouseBackupHostLoadHigh, whose description no longer assumes a
backup is the cause.
---
 .../tests/alert_rules_test.yml                | 43 ++++++++++++++++++-
 .../roles/prometheus/files/alert_rules.yml    |  4 +-
 2 files changed, 43 insertions(+), 4 deletions(-)

diff --git a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
index 08e03fe5..3ff312dd 100644
--- a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
+++ b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
@@ -1,5 +1,6 @@
-# promtool unit tests for the ClickHouse backup alerts in alert_rules.yml. Test time starts at 0
-# (1970-01-01), so a timestamp N days before the test starts is -N * 86400.
+# promtool unit tests for the ClickHouse backup alerts in alert_rules.yml, and for the backup file's
+# exclusion from TextfileJam. Test time starts at 0 (1970-01-01), so a timestamp N days before the test
+# starts is -N * 86400.
 rule_files: [alert_rules.yml]
 evaluation_interval: 1m
 tests:
@@ -76,3 +77,41 @@ tests:
           - exp_labels: {instance: data3.htz-fsn.prod.ooni.nu}
             exp_annotations:
               summary: 'No ClickHouse backup results from data3.htz-fsn.prod.ooni.nu for over a week'
+
+  # HostLoadHigh: data3's 8 CPUs with a load of 5
+  - interval: 1m
+    input_series:
+      - {series: 'node_load5{instance="data3.htz-fsn.prod.ooni.nu"}', values: '5x60'}
+      - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="0", mode="idle"}', values: '0+1x60'}
+      - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="1", mode="idle"}', values: '0+1x60'}
+      - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="2", mode="idle"}', values: '0+1x60'}
+      - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="3", mode="idle"}', values: '0+1x60'}
+      - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="4", mode="idle"}', values: '0+1x60'}
+      - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="5", mode="idle"}', values: '0+1x60'}
+      - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="6", mode="idle"}', values: '0+1x60'}
+      - {series: 'node_cpu_seconds_total{instance="data3.htz-fsn.prod.ooni.nu", cpu="7", mode="idle"}', values: '0+1x60'}
+    alert_rule_test:
+      - {eval_time: 10m, alertname: ClickHouseBackupHostLoadHigh, exp_alerts: []}
+      - eval_time: 20m
+        alertname: ClickHouseBackupHostLoadHigh
+        exp_alerts:
+          - exp_labels: {instance: data3.htz-fsn.prod.ooni.nu}
+            exp_annotations:
+              summary: 'data3 load is 62.5% of its CPUs'
+              description: 'If a backup is running (ooni_clickhouse_backup_running on data3), lower clickhouse_backup_max_bandwidth; otherwise check merges and queries.'
+
+  # TextfileJam leaves out the backup's file, which changes only once a week. node_exporter's file label
+  # is the file's full path.
+  - interval: 1m
+    input_series:
+      - {series: 'node_time{instance="data3", job="node"}', values: '0+60x240'}
+      - {series: 'node_textfile_mtime{instance="data3", job="node", file="/var/lib/node_exporter/textfile_collector/ooni_clickhouse_backup.prom"}', values: '0x240'}
+      - {series: 'node_textfile_mtime{instance="data3", job="node", file="/var/lib/node_exporter/textfile_collector/other.prom"}', values: '0x240'}
+    alert_rule_test:
+      - eval_time: 3h
+        alertname: TextfileJam
+        exp_alerts:
+          - exp_labels: {instance: data3, job: node, file: /var/lib/node_exporter/textfile_collector/other.prom}
+            exp_annotations:
+              summary: 'Stale `textfile` at data3'
+              description: '`/var/lib/node_exporter/textfile_collector/other.prom` is 3h old.'
diff --git a/ansible/roles/prometheus/files/alert_rules.yml b/ansible/roles/prometheus/files/alert_rules.yml
index aae32954..1149f04f 100644
--- a/ansible/roles/prometheus/files/alert_rules.yml
+++ b/ansible/roles/prometheus/files/alert_rules.yml
@@ -102,7 +102,7 @@ groups:
   # all the textfile are updated at least hourly, except the ClickHouse
   # backup results, which change after each backup (see ClickHouseBackupStale)
   - alert: TextfileJam
-    expr: (node_textfile_mtime{file!~"ooni_clickhouse_backup.*"} - ON(instance,job) GROUP_LEFT() node_time) / -3600 > 2
+    expr: (node_textfile_mtime{file!~".*/ooni_clickhouse_backup\\.prom"} - ON(instance,job) GROUP_LEFT() node_time) / -3600 > 2
     annotations:
       summary: 'Stale `textfile` at {{ $labels.instance }}'
       description: '`{{ $labels.file }}` is {{ printf "%.0f" $value }}h old.'
@@ -197,5 +197,5 @@ groups:
     for: 15m
     annotations:
       summary: 'data3 load is {{ $value | humanizePercentage }} of its CPUs'
-      description: 'Backup running: check ooni_clickhouse_backup_running on data3.'
+      description: 'If a backup is running (ooni_clickhouse_backup_running on data3), lower clickhouse_backup_max_bandwidth; otherwise check merges and queries.'
 ...

From d45700ed4f2b77b8ed90ad7144b55f993a3ca354 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 13:59:08 +0000
Subject: [PATCH 21/32] clickhouse_backup: "*" selects every table, an empty
 include_tables none

An empty include_tables meant every table. data1's job includes only
clickhouse_backup_data1_tables, which is to be emptied as its tables
become replicated; data1 would then have backed up every table,
private ones included, into the measurements bucket, under the same
prefixes as data3, whose in-progress full it could also have removed
as incomplete. "*" now selects every table and an empty list none, with
a warning; data3's measurements job and the role's default job say
["*"].
---
 ansible/host_vars/data3.htz-fsn.prod.ooni.nu  |  2 +-
 .../roles/clickhouse_backup/defaults/main.yml |  7 +++---
 .../files/ooni_clickhouse_backup.py           | 13 ++++++----
 ansible/roles/clickhouse_backup/tests/run.sh  |  4 ++-
 .../roles/clickhouse_backup/tests/test_e2e.py | 25 +++++++++++--------
 .../roles/clickhouse_backup/tests/test_job.py | 15 ++++++++++-
 .../roles/clickhouse_backup/tests/vars.yml    |  2 +-
 7 files changed, 46 insertions(+), 22 deletions(-)

diff --git a/ansible/host_vars/data3.htz-fsn.prod.ooni.nu b/ansible/host_vars/data3.htz-fsn.prod.ooni.nu
index 39c522cd..0e4eef14 100644
--- a/ansible/host_vars/data3.htz-fsn.prod.ooni.nu
+++ b/ansible/host_vars/data3.htz-fsn.prod.ooni.nu
@@ -7,7 +7,7 @@ clickhouse_base_path: /data/clickhouse
 clickhouse_backup_jobs:
   - name: measurements
     s3_bucket: ooni-clickhouse-backup
-    include_tables: []
+    include_tables: ["*"]
     exclude_tables: "{{ clickhouse_backup_private_tables + clickhouse_backup_data1_tables }}"
   - name: private
     s3_bucket: ooni-clickhouse-backup-private
diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml
index 4a59d13f..5c3637ee 100644
--- a/ansible/roles/clickhouse_backup/defaults/main.yml
+++ b/ansible/roles/clickhouse_backup/defaults/main.yml
@@ -35,12 +35,13 @@ clickhouse_backup_s3_settings:
   s3_min_upload_part_size: 134217728
   s3_max_upload_part_size: 134217728
 
-# One job per bucket. A job backs up the tables matching include_tables
-# (every table if empty) minus the excludes below and its own.
+# One job per bucket. A job backs up the tables matching include_tables ("*"
+# for every table; an empty list backs up none) minus the excludes below and
+# its own.
 clickhouse_backup_jobs:
   - name: measurements
     s3_bucket: ""
-    include_tables: []
+    include_tables: ["*"]
     exclude_tables: []
 
 # Complete chains kept per table, besides chains pinned with --keep: the
diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
index a045594a..0a5cbbb4 100644
--- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
+++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
@@ -375,7 +375,8 @@ def __init__(self, conf: dict, metrics: Metrics = None):
         )
         self.store = Store(conf)
         self.keep_chains = max(1, int(conf.get("keep_chains", 1)))
-        # fnmatch patterns; an empty include list means every table
+        # fnmatch patterns: "*" for every table; an empty list selects none, so
+        # emptying a host's list can't make it back up every table
         self.include = conf.get("include_tables", [])
         self.exclude = conf.get("exclude_tables", [])
         self.poll_s = float(conf.get("poll_interval_seconds", 30))
@@ -432,7 +433,7 @@ def tables(self, only: list) -> tuple:
         return out, missing
 
     def includes(self, table: str) -> bool:
-        return not self.include or any(fnmatch.fnmatchcase(table, p) for p in self.include)
+        return any(fnmatch.fnmatchcase(table, p) for p in self.include)
 
     def excludes(self, table: str) -> bool:
         return any(fnmatch.fnmatchcase(table, p) for p in self.exclude)
@@ -579,6 +580,8 @@ def run(self, only: list, force_full: bool, keep: str) -> int:
                     log.error(f"backup {r['id']} is still running on the server")
                 raise RuntimeError("not starting: wait for it to finish, or for the server to drop it")
             tables, failed = self.tables(only)
+            if not tables and not only:
+                log.warning(f"{self.name}: no tables to back up (include_tables: {self.include})")
         except Exception as e:
             log.error(f"{self.name}: could not start: {e}")
             self.metrics.job_failed(self.name)
@@ -651,11 +654,11 @@ def jobs_for_tables(jobs: list, tables: list) -> tuple:
 def job_confs(conf: dict, names: list) -> list:
     """Each job's config: the top-level keys, overridden by the job's own.
 
-    A config without "jobs" is one job named "default". A job's
-    exclude_tables add to the top-level ones."""
+    A config without "jobs" is one job named "default", for every table. A
+    job's exclude_tables add to the top-level ones."""
     common = {k: v for k, v in conf.items() if k != "jobs"}
     out = []
-    for job in conf.get("jobs") or [{"name": "default"}]:
+    for job in conf.get("jobs") or [{"name": "default", "include_tables": ["*"]}]:
         merged = {**common, **job}
         merged["exclude_tables"] = common.get("exclude_tables", []) + job.get("exclude_tables", [])
         if not names or merged["name"] in names:
diff --git a/ansible/roles/clickhouse_backup/tests/run.sh b/ansible/roles/clickhouse_backup/tests/run.sh
index 51b2295a..c45abe8f 100755
--- a/ansible/roles/clickhouse_backup/tests/run.sh
+++ b/ansible/roles/clickhouse_backup/tests/run.sh
@@ -29,6 +29,8 @@ echo "== schedule"
 python3 -m unittest test_schedule test_job
 
 echo "== end to end"
-[ -n "${KEEP:-}" ] || trap 'compose down -v --remove-orphans >/dev/null 2>&1' EXIT
+# the containers' logs if it fails, then remove them, unless KEEP is set
+trap 'rc=$?; [ $rc = 0 ] || compose logs --no-color --tail=200; [ -n "${KEEP:-}" ] || compose down -v --remove-orphans >/dev/null 2>&1' EXIT
+compose down -v --remove-orphans >/dev/null 2>&1  # left from a run with KEEP=1
 compose up -d --build --quiet-pull
 python3 test_e2e.py
diff --git a/ansible/roles/clickhouse_backup/tests/test_e2e.py b/ansible/roles/clickhouse_backup/tests/test_e2e.py
index a43750dd..fe44746f 100644
--- a/ansible/roles/clickhouse_backup/tests/test_e2e.py
+++ b/ansible/roles/clickhouse_backup/tests/test_e2e.py
@@ -4,7 +4,7 @@
   2. ClickHouseBackupCompleted reaches Slack once per job, as a notice, without email
   3. a second run takes incrementals, and a restore from them returns the same rows
   4. a run whose bucket is missing fails, and ClickHouseBackupFailed reaches Slack and email
-  5. a new month's full interrupted by a ClickHouse restart deletes nothing; the next run
+  5. a new month's full interrupted by killing ClickHouse deletes nothing; the next run
      removes what it left, takes the full and prunes the oldest chain
 Prints each check; exits 1 at the first failure."""
 import json
@@ -28,9 +28,11 @@
 )
 
 
-def check(ok, what):
+def check(ok, what, output=""):
     print(f"  {'ok  ' if ok else 'FAIL'} {what}")
     if not ok:
+        if output:
+            print("  the script said:\n" + output[-3000:])
         sys.exit(1)
 
 
@@ -139,7 +141,7 @@ def setup():
 def test_first_run():
     print("== 1. first run: a full backup of every table")
     r = backup("backup", "run")
-    check(r.returncode == 0, f"the run succeeds{'' if r.returncode == 0 else ': ' + r.stderr[-2000:]}")
+    check(r.returncode == 0, "the run succeeds", r.stderr)
     for bucket, table in [(MEASUREMENTS, "fastpath"), (MEASUREMENTS, "obs_web"),
                           (MEASUREMENTS, "fingerprints_dns"), (PRIVATE, "msmt_feedback")]:
         c = chains(bucket, table)
@@ -166,7 +168,7 @@ def test_incremental_and_restore():
     print("== 3. second run: incrementals, and a restore")
     ch("INSERT INTO ooni.fastpath SELECT toString(number), 'https://example.org/new', 'f' FROM numbers(200000, 1000)")
     r = backup("backup", "run")
-    check(r.returncode == 0, "the run succeeds")
+    check(r.returncode == 0, "the run succeeds", r.stderr)
     (chain, backups), = chains(MEASUREMENTS, "fastpath").items()
     check(len(backups) == 2 and all(backups.values()), f"fastpath: a full and an incremental in chain {chain}")
     latest = f"http://minio:9000/{MEASUREMENTS}/prod/ooni/fastpath/{chain}/{max(backups)}"
@@ -180,7 +182,7 @@ def test_incremental_and_restore():
 def test_failure_alert(emails_before):
     print("== 4. a run that fails: ClickHouseBackupFailed")
     r = backup("broken", "run")
-    check(r.returncode == 1, "the run exits 1")
+    check(r.returncode == 1, "the run exits 1", r.stderr)
     title = f"[FIRING] ClickHouse backup of labeling_frames (broken) failed on {DATA3}"
     msgs = wait_for(f"a Slack message: {title}", lambda: [m for m in slack() if m[0] == title])
     check("Nothing was deleted" in msgs[0][1], f"it says what was done: {msgs[0][1]!r}")
@@ -202,17 +204,20 @@ def test_interrupted_full():
                                       "--config", "/work/slow.json", "run", "--table", "obs_web"],
                            stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
     wait_for("the new full's first files", lambda: set(chains(MEASUREMENTS, "obs_web")) - {old, previous}, 60)
-    subprocess.run(COMPOSE + ["restart", "clickhouse"], check=True, capture_output=True)
-    out, _ = run.communicate(timeout=180)
-    check(run.returncode == 1, "the interrupted run exits 1")
+    # killed, not stopped: ClickHouse waits for running backups when it stops
+    # (shutdown_wait_backups_and_restores), so a clean stop would finish this one
+    subprocess.run(COMPOSE + ["kill", "clickhouse"], check=True, capture_output=True)
+    subprocess.run(COMPOSE + ["start", "clickhouse"], check=True, capture_output=True)
+    out, _ = run.communicate(timeout=900)
+    check(run.returncode == 1, "the interrupted run exits 1", out)
     c = chains(MEASUREMENTS, "obs_web")
     new = sorted(set(c) - {old, previous})
     check(new and not any(c[new[0]].values()), f"the new chain {new[0]} has no .backup")
     check(all(c[x][x] for x in (old, previous)), f"both older chains, {old} and {previous}, are still there")
 
-    wait_for("ClickHouse after the restart", lambda: ch("SELECT 1") == "1")
+    wait_for("ClickHouse after it was killed", lambda: ch("SELECT 1") == "1")
     r = backup("backup", "run", "--table", "obs_web")
-    check(r.returncode == 0, "the next run succeeds")
+    check(r.returncode == 0, "the next run succeeds", r.stderr)
     check("removed incomplete chain" in r.stderr, "it removed the interrupted full")
     c = chains(MEASUREMENTS, "obs_web")
     check(previous in c and old not in c and len(c) == 2 and all(all(b.values()) for b in c.values()),
diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py
index daec61c5..15c888f6 100644
--- a/ansible/roles/clickhouse_backup/tests/test_job.py
+++ b/ansible/roles/clickhouse_backup/tests/test_job.py
@@ -66,7 +66,7 @@ def test_old_state_files_still_load(self):
 
 class TestRunTable(unittest.TestCase):
     def test_only_the_jobs_that_back_up_the_tables(self):
-        jobs = [job("measurements", exclude=["msmt_feedback"]), job("private", include=["msmt_feedback"])]
+        jobs = [job("measurements", include=["*"], exclude=["msmt_feedback"]), job("private", include=["msmt_feedback"])]
         selected, orphans = backup.jobs_for_tables(jobs, ["msmt_feedback"])
         self.assertEqual([j.name for j in selected], ["private"])
         self.assertEqual(orphans, [])
@@ -76,6 +76,19 @@ def test_tables_no_job_backs_up(self):
         self.assertEqual(backup.jobs_for_tables(jobs, ["fastpath"]), ([], ["fastpath"]))
 
 
+class TestIncludeTables(unittest.TestCase):
+    def test_star_is_every_table(self):
+        self.assertTrue(job(include=["*"]).includes("fastpath"))
+
+    def test_an_empty_list_is_no_table(self):
+        # once data1's list of its own tables is emptied, data1 mustn't back up everything
+        self.assertFalse(job(include=[]).includes("msmt_feedback"))
+
+    def test_a_config_without_jobs_backs_up_every_table(self):
+        [conf] = backup.job_confs({"exclude_tables": ["*_tmp"]}, [])
+        self.assertEqual(conf["include_tables"], ["*"])
+
+
 class TestClickHouseErrors(unittest.TestCase):
     def setUp(self):
         d = tempfile.mkdtemp()
diff --git a/ansible/roles/clickhouse_backup/tests/vars.yml b/ansible/roles/clickhouse_backup/tests/vars.yml
index 0db73f9f..06835714 100644
--- a/ansible/roles/clickhouse_backup/tests/vars.yml
+++ b/ansible/roles/clickhouse_backup/tests/vars.yml
@@ -10,7 +10,7 @@ clickhouse_backup_s3_secret_access_key: testsecret123
 test_jobs:
   - name: measurements
     s3_bucket: ooni-clickhouse-backup
-    include_tables: []
+    include_tables: ["*"]
     exclude_tables: [msmt_feedback, labeling_frames]
   - name: private
     s3_bucket: ooni-clickhouse-backup-private

From d2a98d89e8742fc9d35ad344a49dcb3076dd2ac0 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 13:59:08 +0000
Subject: [PATCH 22/32] clickhouse_backup: forget the results of tables and
 jobs no longer backed up

The state file kept every table and job it had ever seen, so a table
dropped, or moved to another job (as data1's tables will be once
replicated), kept its old results and alerted forever:
ClickHouseBackupStale after 8 days, or ClickHouseBackupFailed. After a
run without --table, a job now forgets the tables it no longer selects,
and every run forgets the jobs no longer in the config.
---
 .../files/ooni_clickhouse_backup.py           | 25 +++++++++++++++
 .../roles/clickhouse_backup/tests/test_job.py | 32 +++++++++++++++++++
 2 files changed, 57 insertions(+)

diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
index 0a5cbbb4..1106d232 100644
--- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
+++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
@@ -306,6 +306,26 @@ def job_failed(self, job: str) -> None:
                 entry["last_failed"] = 1
         self.save()
 
+    def forget_tables(self, job: str, tables: set) -> None:
+        """Drop the job's other tables, e.g. dropped or moved to another job,
+        so that their old results don't alert forever"""
+        for key, entry in list(self.state["tables"].items()):
+            if entry["job"] == job and entry["table"] not in tables:
+                log.info(f"{job}: {entry['table']}: no longer backed up by this job, forgetting its results")
+                del self.state["tables"][key]
+        self.save()
+
+    def forget_jobs(self, jobs: set) -> None:
+        """Drop the results of jobs that are no longer configured"""
+        for key, entry in list(self.state["tables"].items()):
+            if entry["job"] not in jobs:
+                del self.state["tables"][key]
+        for part in ("runs", "running", "running_since"):
+            for job in set(self.state[part]) - jobs:
+                log.info(f"{job}: no longer configured, forgetting its results")
+                del self.state[part][job]
+        self.save()
+
     def run_result(self, job: str, duration_s: float, tables: int, fulls: int, failed: int) -> None:
         self.state["runs"][job] = {
             "finished": time.time(), "duration": round(duration_s, 1),
@@ -601,6 +621,8 @@ def run(self, only: list, force_full: bool, keep: str) -> int:
         finally:
             self.metrics.job_running(self.name, False)
         self.metrics.run_result(self.name, time.monotonic() - t0, done, fulls, len(failed))
+        if not only:
+            self.metrics.forget_tables(self.name, {table for table, _ in tables})
 
         if failed:
             log.error(f"{self.name}: failed tables: {', '.join(failed)}")
@@ -698,6 +720,7 @@ def main() -> int:
         conf = json.load(f)
     metrics = Metrics(conf.get("metrics_file", ""), conf.get("state_file", ""))
     jobs = [BackupJob(j, metrics) for j in job_confs(conf, args.job)]
+    configured = {j["name"] for j in job_confs(conf, [])}
 
     if args.command == "list":
         for job in jobs:
@@ -727,6 +750,8 @@ def main() -> int:
                 rc |= job.run(args.table, args.full, args.keep)
             else:
                 rc |= job.release(args.label)
+        if args.command == "run":
+            metrics.forget_jobs(configured)
         return rc
 
 
diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py
index 15c888f6..874aa1fe 100644
--- a/ansible/roles/clickhouse_backup/tests/test_job.py
+++ b/ansible/roles/clickhouse_backup/tests/test_job.py
@@ -89,6 +89,38 @@ def test_a_config_without_jobs_backs_up_every_table(self):
         self.assertEqual(conf["include_tables"], ["*"])
 
 
+class TestForget(unittest.TestCase):
+    def setUp(self):
+        d = tempfile.mkdtemp()
+        self.metrics = backup.Metrics(os.path.join(d, "out.prom"), os.path.join(d, "state.json"))
+        for job_, table in [("data1", "labeling_frames"), ("data1", "changepoint_label"), ("private", "msmt_feedback")]:
+            self.metrics.table_result(job_, table, True, {"total_size": 1, "num_files": 1})
+        self.metrics.job_running("old_job", False)
+
+    def tables(self):
+        return sorted(self.metrics.state["tables"])
+
+    def test_a_table_the_job_no_longer_selects(self):
+        # labeling_frames was replicated, and moved from data1's job to data3's
+        self.metrics.forget_tables("data1", {"changepoint_label"})
+        self.assertEqual(self.tables(), ["data1/changepoint_label", "private/msmt_feedback"])
+
+    def test_a_job_no_longer_configured(self):
+        self.metrics.forget_jobs({"data1"})
+        self.assertEqual(self.tables(), ["data1/changepoint_label", "data1/labeling_frames"])
+        self.assertNotIn("old_job", self.metrics.state["running"])
+
+    def test_a_run_with_table_forgets_nothing(self):
+        j = job("data1", include=["*"], metrics=self.metrics)
+        j.running_backups = lambda: []
+        j.tables = lambda only: ([("changepoint_label", "MergeTree")], [])
+        j.backup_table = lambda *a: False
+        j.run(["changepoint_label"], False, "")
+        self.assertIn("data1/labeling_frames", self.tables())
+        j.run([], False, "")
+        self.assertNotIn("data1/labeling_frames", self.tables())
+
+
 class TestClickHouseErrors(unittest.TestCase):
     def setUp(self):
         d = tempfile.mkdtemp()

From 862b236d0a93958033ef46001bd09cba8e4561c0 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 13:59:09 +0000
Subject: [PATCH 23/32] clickhouse_backup: match only this bucket's running
 backups, check deletes, fsync state

- running_backups() matched the endpoint by substring, so the
  measurements bucket also matched the private bucket's backups; match
  <bucket>/<prefix>/.
- delete_objects reports keys it didn't delete in a 200 response, which
  were ignored; raise on them.
- Fsync the state and metrics files before they replace the old ones,
  and start from no results if the state file is unreadable instead of
  failing every command.
- Pass the upload settings to the EmbeddedRocksDB/Join export too: R2
  needs its multipart parts to be the same size.
---
 .../files/ooni_clickhouse_backup.py           | 22 ++++++++++---
 .../roles/clickhouse_backup/tests/test_job.py | 31 +++++++++++++++++++
 2 files changed, 49 insertions(+), 4 deletions(-)

diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
index 1106d232..7f11ea7c 100644
--- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
+++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
@@ -206,7 +206,12 @@ def delete_prefix(self, prefix: str) -> int:
         for page in pager.paginate(Bucket=self.bucket, Prefix=prefix):
             keys = [{"Key": o["Key"]} for o in page.get("Contents", [])]
             if keys:
-                self.s3.delete_objects(Bucket=self.bucket, Delete={"Objects": keys, "Quiet": True})
+                resp = self.s3.delete_objects(Bucket=self.bucket, Delete={"Objects": keys, "Quiet": True})
+                # a 200 response can still list keys it didn't delete
+                errors = resp.get("Errors", [])
+                if errors:
+                    e = errors[0]
+                    raise RuntimeError(f"{len(errors)} objects not deleted under {prefix}, e.g. {e.get('Key')}: {e.get('Code')} {e.get('Message')}")
                 n += len(keys)
         return n
 
@@ -233,6 +238,8 @@ def write_atomically(path: str, text: str) -> None:
     tmp = f"{path}.{os.getpid()}.tmp"
     with open(tmp, "w") as f:
         f.write(text)
+        f.flush()
+        os.fsync(f.fileno())  # or a power loss can leave an empty file behind
     os.chmod(tmp, 0o644)
     os.replace(tmp, path)
 
@@ -276,8 +283,12 @@ def __init__(self, metrics_file: str = "", state_file: str = ""):
         self.state_file = state_file
         self.state = {"tables": {}, "running": {}}
         if state_file and os.path.exists(state_file):
-            with open(state_file) as f:
-                self.state = json.load(f)
+            try:
+                with open(state_file) as f:
+                    self.state = json.load(f)
+            except ValueError as e:
+                # the results are rebuilt as tables are backed up again
+                log.warning(f"{state_file} is unreadable ({e}), starting from no results")
         self.state.setdefault("runs", {})
         self.state.setdefault("running_since", {})
 
@@ -463,7 +474,9 @@ def running_backups(self) -> list:
         rows = self.ch.rows(
             "SELECT id, name FROM system.backups WHERE status = 'CREATING_BACKUP'"
         )
-        return [r for r in rows if self.store.ch_base in r["name"]]
+        # this job's bucket and prefix only: one bucket's name can begin another's
+        ours = f"{self.store.ch_base}/{self.store.prefix}/"
+        return [r for r in rows if ours in r["name"]]
 
     def wait_for_clickhouse(self) -> None:
         """At boot a missed run starts before ClickHouse accepts queries"""
@@ -564,6 +577,7 @@ def backup_table(self, table: str, engine: str, force_full: bool, keep: str) ->
                 INSERT INTO FUNCTION s3({self.store.ch_s3(key + "/data.native.zst")}, 'Native')
                 SELECT * FROM `{self.db}`.`{table}`
                 """,
+                settings=self.settings,  # R2 needs equal upload parts here too
                 timeout=3600,
             )
 
diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py
index 874aa1fe..942a4607 100644
--- a/ansible/roles/clickhouse_backup/tests/test_job.py
+++ b/ansible/roles/clickhouse_backup/tests/test_job.py
@@ -4,6 +4,7 @@
 import json
 import os
 import tempfile
+import types
 import unittest
 import urllib.error
 from unittest import mock
@@ -121,6 +122,36 @@ def test_a_run_with_table_forgets_nothing(self):
         self.assertNotIn("data1/labeling_frames", self.tables())
 
 
+class TestStoreAndState(unittest.TestCase):
+    def test_only_this_buckets_running_backups(self):
+        j = job()
+        j.store = types.SimpleNamespace(ch_base="https://r2.example/ooni-clickhouse-backup", prefix="prod")
+        j.ch = types.SimpleNamespace(rows=lambda sql: [
+            {"id": "1", "name": "S3('https://r2.example/ooni-clickhouse-backup-private/prod/ooni/msmt_feedback/a/b')"},
+            {"id": "2", "name": "S3('https://r2.example/ooni-clickhouse-backup/prod/ooni/fastpath/a/b')"},
+        ])
+        self.assertEqual([r["id"] for r in j.running_backups()], ["2"])
+
+    def test_objects_not_deleted_are_an_error(self):
+        store = backup.Store.__new__(backup.Store)
+        store.bucket, store.prefix = "b", "prod"
+        store.s3 = types.SimpleNamespace(
+            get_paginator=lambda name: types.SimpleNamespace(paginate=lambda **kw: [{"Contents": [{"Key": "prod/ooni/t/c/x"}]}]),
+            delete_objects=lambda **kw: {"Errors": [{"Key": "prod/ooni/t/c/x", "Code": "AccessDenied", "Message": "no"}]},
+        )
+        with self.assertRaisesRegex(RuntimeError, "1 objects not deleted"):
+            store.delete_prefix("prod/ooni/t/c/")
+
+    def test_an_unreadable_state_file_starts_empty(self):
+        d = tempfile.mkdtemp()
+        state = os.path.join(d, "state.json")
+        with open(state, "w") as f:
+            f.write("")  # e.g. truncated by a power loss
+        m = backup.Metrics(os.path.join(d, "out.prom"), state)
+        m.job_running("measurements", True)
+        self.assertEqual(m.state["running"], {"measurements": 1})
+
+
 class TestClickHouseErrors(unittest.TestCase):
     def setUp(self):
         d = tempfile.mkdtemp()

From c29e2b5124e45942402120a26fa7dfd523e361b2 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 13:59:09 +0000
Subject: [PATCH 24/32] clickhouse_backup tests: interrupt the full
 deterministically, show output and logs on failure

ClickHouse waits for running backups when it stops
(shutdown_wait_backups_and_restores), so the interrupted-full scenario
only worked because compose restart killed it after 10 s; kill it
instead. Failing run checks now show what the script said, run.sh
prints the containers' logs when the test fails and removes containers
left by KEEP=1 before starting, and the unit tests silence the
script's log.
---
 ansible/roles/clickhouse_backup/tests/test_job.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py
index 942a4607..1eeacbd1 100644
--- a/ansible/roles/clickhouse_backup/tests/test_job.py
+++ b/ansible/roles/clickhouse_backup/tests/test_job.py
@@ -14,6 +14,7 @@
 )
 backup = importlib.util.module_from_spec(spec)
 spec.loader.exec_module(backup)
+backup.log.disabled = True  # the tests check results, not log lines
 
 
 class FlakyClickHouse:

From d5e76116c4707605b1432849251e256dbf0380a1 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 14:36:40 +0000
Subject: [PATCH 25/32] clickhouse_backup tests: restore the full and each
 incremental, the RocksDB rows, and the new full after pruning

Only the newest incremental was restored. Two more runs now give
fastpath a full and two incrementals, and each of the three restores
fastpath as it was at that backup (200,000, 201,000 and 202,000 rows,
by count and a checksum of every row). fingerprints_dns, an
EmbeddedRocksDB table, is restored as the runbook says: RESTORE its
schema, then load data.native.zst. After the interrupted full and the
pruning, the new full restores obs_web. The run takes about 9 minutes.
---
 .../roles/clickhouse_backup/tests/README.md   | 10 ++--
 .../roles/clickhouse_backup/tests/test_e2e.py | 50 ++++++++++++++-----
 2 files changed, 44 insertions(+), 16 deletions(-)

diff --git a/ansible/roles/clickhouse_backup/tests/README.md b/ansible/roles/clickhouse_backup/tests/README.md
index 4d548799..77eaac1f 100644
--- a/ansible/roles/clickhouse_backup/tests/README.md
+++ b/ansible/roles/clickhouse_backup/tests/README.md
@@ -1,6 +1,6 @@
 # Tests for the clickhouse_backup role
 
-`./run.sh` runs them all, in about seven minutes:
+`./run.sh` runs them all, in about nine minutes:
 
 - `alert_rules_test.yml`: promtool unit tests of the backup alerts in
   `roles/prometheus/files/alert_rules.yml`, then amtool checks that
@@ -13,9 +13,11 @@
   ClickHouse at production's version, MinIO standing in for R2, the script
   on Debian bookworm with its python3-boto3, as on the hosts, node_exporter,
   Prometheus with the repo's alert rules, Alertmanager with the repo's
-  routes, and a stand-in for Slack. Full and incremental backups, a
-  restore, a failed run, an interrupted full, the month's new full and
-  pruning, and the Slack messages and emails each one causes.
+  routes, and a stand-in for Slack. Full and incremental backups, and
+  restoring the full and each incremental of a chain, and an
+  EmbeddedRocksDB table with its exported rows; a failed run; an
+  interrupted full, the month's new full, pruning and restoring what's
+  left; and the Slack messages and emails each one causes.
 
 The configs are rendered from the role's templates into `work/` by
 `render.yml`, with the variables in `vars.yml`.
diff --git a/ansible/roles/clickhouse_backup/tests/test_e2e.py b/ansible/roles/clickhouse_backup/tests/test_e2e.py
index fe44746f..f4c59636 100644
--- a/ansible/roles/clickhouse_backup/tests/test_e2e.py
+++ b/ansible/roles/clickhouse_backup/tests/test_e2e.py
@@ -2,10 +2,11 @@
 In order:
   1. a first run takes a full backup of every table, each job to its own bucket
   2. ClickHouseBackupCompleted reaches Slack once per job, as a notice, without email
-  3. a second run takes incrementals, and a restore from them returns the same rows
+  3. two more runs take incrementals; restoring the full and each incremental returns the rows of
+     its moment, and an EmbeddedRocksDB table comes back with its exported rows
   4. a run whose bucket is missing fails, and ClickHouseBackupFailed reaches Slack and email
   5. a new month's full interrupted by killing ClickHouse deletes nothing; the next run
-     removes what it left, takes the full and prunes the oldest chain
+     removes what it left, takes the full and prunes the oldest chain, and the full restores
 Prints each check; exits 1 at the first failure."""
 import json
 import os
@@ -164,19 +165,41 @@ def test_completed_alert(emails_before):
     check(email_attempts() == emails_before, "no email sent")
 
 
+def digest(table):
+    """row count and a checksum of every row"""
+    return ch(f"SELECT count(), sum(cityHash64(*)) FROM {table}")
+
+
+def backup_url(bucket, table, chain, name):
+    return f"http://minio:9000/{bucket}/prod/ooni/{table}/{chain}/{name}"
+
+
 def test_incremental_and_restore():
-    print("== 3. second run: incrementals, and a restore")
-    ch("INSERT INTO ooni.fastpath SELECT toString(number), 'https://example.org/new', 'f' FROM numbers(200000, 1000)")
-    r = backup("backup", "run")
-    check(r.returncode == 0, "the run succeeds", r.stderr)
+    print("== 3. incrementals, and restoring each backup of a chain")
+    # fastpath as at the first run's full, then after each of two more runs
+    states = [digest("ooni.fastpath")]
+    for first in (200000, 201000):
+        ch(f"INSERT INTO ooni.fastpath SELECT toString(number), 'https://example.org/new', 'f' FROM numbers({first}, 1000)")
+        r = backup("backup", "run")
+        check(r.returncode == 0, "the run succeeds", r.stderr)
+        states.append(digest("ooni.fastpath"))
     (chain, backups), = chains(MEASUREMENTS, "fastpath").items()
-    check(len(backups) == 2 and all(backups.values()), f"fastpath: a full and an incremental in chain {chain}")
-    latest = f"http://minio:9000/{MEASUREMENTS}/prod/ooni/fastpath/{chain}/{max(backups)}"
+    names = sorted(backups)
+    check(len(names) == 3 and all(backups.values()), f"fastpath: a full and two incrementals in chain {chain}")
     ch("CREATE DATABASE restored")
-    ch(f"RESTORE TABLE ooni.fastpath AS restored.fastpath FROM S3('{latest}')")
-    digest = "SELECT count(), sum(cityHash64(measurement_uid, input, anomaly)) FROM {}"
-    check(ch(digest.format("restored.fastpath")) == ch(digest.format("ooni.fastpath")),
-          "the restored fastpath has the same rows (201000)")
+    for i, (name, state) in enumerate(zip(names, states)):
+        what = "the full" if i == 0 else f"incremental {i}"
+        ch(f"RESTORE TABLE ooni.fastpath AS restored.fastpath_{i} FROM S3('{backup_url(MEASUREMENTS, 'fastpath', chain, name)}')")
+        check(digest(f"restored.fastpath_{i}") == state, f"{what} restores fastpath as it was then ({state.split()[0]} rows)")
+
+    # EmbeddedRocksDB: BACKUP keeps only the schema, the rows are in data.native.zst (the runbook's steps 2 and 3)
+    c = chains(MEASUREMENTS, "fingerprints_dns")
+    chain = max(ch_ for ch_, b in c.items() if any(b.values()))
+    url = backup_url(MEASUREMENTS, "fingerprints_dns", chain, max(n for n, ok in c[chain].items() if ok))
+    ch(f"RESTORE TABLE ooni.fingerprints_dns AS restored.fingerprints_dns FROM S3('{url}')")
+    ch(f"INSERT INTO restored.fingerprints_dns SELECT * FROM s3('{url}/data.native.zst', 'Native')")
+    check(digest("restored.fingerprints_dns") == digest("ooni.fingerprints_dns"),
+          "fingerprints_dns (EmbeddedRocksDB) restores with its exported rows")
 
 
 def test_failure_alert(emails_before):
@@ -222,6 +245,9 @@ def test_interrupted_full():
     c = chains(MEASUREMENTS, "obs_web")
     check(previous in c and old not in c and len(c) == 2 and all(all(b.values()) for b in c.values()),
           f"it took the full and pruned {old}: chains {sorted(c)}")
+    new = max(c)
+    ch(f"RESTORE TABLE ooni.obs_web AS restored.obs_web FROM S3('{backup_url(MEASUREMENTS, 'obs_web', new, new)}')")
+    check(digest("restored.obs_web") == digest("ooni.obs_web"), f"the new full {new} restores obs_web")
 
 
 def main():

From 18cd22621b10e3bc56ce24bb3ded05b78d7aa1b1 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 15:08:47 +0000
Subject: [PATCH 26/32] clickhouse_backup: report the tables on the cluster
 that no backup covers

A table created on data1 or data2 only, like data1's own tables were,
was in no backup and nothing said so: data3 doesn't have it, and data1's
job lists its tables. After each run, data3 now lists the tables on
every node (clickhouse_backup_check_cluster: oonidata_cluster) and
reports those that no job here backs up (from a table on this host),
that aren't in clickhouse_backup_data1_tables (backed up by data1) and
that aren't excluded, as ooni_clickhouse_backup_table_not_backed_up.
ClickHouseTableNotBackedUp names the table and its nodes and says how to
fix it.

clickhouse_backup_exclude_tables becomes the list of tables that are
never backed up, each with its reason: fingerprints_dns and
fingerprints_http (the hourly updater rebuilds them) and session_expunge
(empty, unused) join it. That changes no backup: they're not on data3,
and data1's job lists its tables. The end-to-end test's EmbeddedRocksDB
table is now test_groups, as in production.
---
 ansible/host_vars/data3.htz-fsn.prod.ooni.nu  |  5 ++
 .../roles/clickhouse_backup/defaults/main.yml | 12 ++++
 .../files/ooni_clickhouse_backup.py           | 55 +++++++++++++++++++
 .../templates/clickhouse-backup.json.j2       |  2 +
 .../tests/alert_rules_test.yml                | 13 +++++
 .../roles/clickhouse_backup/tests/test_e2e.py | 22 ++++----
 .../roles/clickhouse_backup/tests/test_job.py | 26 +++++++++
 .../roles/prometheus/files/alert_rules.yml    |  8 +++
 docs/Runbooks.md                              | 13 ++++-
 9 files changed, 144 insertions(+), 12 deletions(-)

diff --git a/ansible/host_vars/data3.htz-fsn.prod.ooni.nu b/ansible/host_vars/data3.htz-fsn.prod.ooni.nu
index 0e4eef14..34959203 100644
--- a/ansible/host_vars/data3.htz-fsn.prod.ooni.nu
+++ b/ansible/host_vars/data3.htz-fsn.prod.ooni.nu
@@ -13,3 +13,8 @@ clickhouse_backup_jobs:
     s3_bucket: ooni-clickhouse-backup-private
     include_tables: "{{ clickhouse_backup_private_tables }}"
     exclude_tables: []
+
+# After each run, report the tables on data1-3 that no backup covers;
+# data1's own tables are backed up by data1
+clickhouse_backup_check_cluster: oonidata_cluster
+clickhouse_backup_backed_up_elsewhere: "{{ clickhouse_backup_data1_tables }}"
diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml
index 5c3637ee..6ae23389 100644
--- a/ansible/roles/clickhouse_backup/defaults/main.yml
+++ b/ansible/roles/clickhouse_backup/defaults/main.yml
@@ -56,9 +56,21 @@ clickhouse_backup_max_bandwidth: 50000000
 
 # fnmatch patterns, for every job. Materialized views' .inner tables are
 # always skipped: they're backed up with their view.
+# Tables that are never backed up, for every job, and why
 clickhouse_backup_exclude_tables:
   - "*_tmp"               # updater staging tables
   - msmt_feedback_old
+  - fingerprints_dns      # the hourly fingerprints updater rebuilds them
+  - fingerprints_http
+  - session_expunge       # empty, no longer used (backend #972)
+
+# The cluster whose tables are checked after each run: any table on any of its
+# nodes that no job here backs up, that isn't excluded above and that isn't in
+# clickhouse_backup_backed_up_elsewhere is reported (ClickHouseTableNotBackedUp).
+# Set on one host, the one taking most backups; empty checks nothing.
+clickhouse_backup_check_cluster: ""
+# fnmatch patterns of the tables another host backs up
+clickhouse_backup_backed_up_elsewhere: []
 
 # Results for Prometheus, read by node_exporter's textfile collector
 # (node_exporter_textfile_dir in the prometheus_node_exporter role)
diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
index 7f11ea7c..e743fb5c 100644
--- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
+++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
@@ -291,6 +291,7 @@ def __init__(self, metrics_file: str = "", state_file: str = ""):
                 log.warning(f"{state_file} is unreadable ({e}), starting from no results")
         self.state.setdefault("runs", {})
         self.state.setdefault("running_since", {})
+        self.state.setdefault("not_backed_up", {})
 
     def table_result(self, job: str, table: str, ok: bool, st: dict = None, duration_s: float = 0) -> None:
         entry = self.state["tables"].setdefault(f"{job}/{table}", {"job": job, "table": table})
@@ -337,6 +338,11 @@ def forget_jobs(self, jobs: set) -> None:
                 del self.state[part][job]
         self.save()
 
+    def not_backed_up(self, tables: dict) -> None:
+        """{table: [hosts]} of the cluster's tables that no backup covers"""
+        self.state["not_backed_up"] = tables
+        self.save()
+
     def run_result(self, job: str, duration_s: float, tables: int, fulls: int, failed: int) -> None:
         self.state["runs"][job] = {
             "finished": time.time(), "duration": round(duration_s, 1),
@@ -362,6 +368,13 @@ def render(self) -> str:
             lines += [f"# HELP {p}{name} {help_}", f"# TYPE {p}{name} gauge"]
             for job, run in sorted(self.state["runs"].items()):
                 lines.append(f'{p}{name}{{backup="{label_value(job)}"}} {run[field]}')
+        lines += [
+            f"# HELP {p}table_not_backed_up 1 for a table on the cluster that no backup job covers or excludes",
+            f"# TYPE {p}table_not_backed_up gauge",
+        ]
+        for table, hosts in sorted(self.state["not_backed_up"].items()):
+            labels = f'table="{label_value(table)}",hosts="{label_value(",".join(sorted(hosts)))}"'
+            lines.append(f"{p}table_not_backed_up{{{labels}}} 1")
         for name, field, help_ in self.GAUGES:
             lines += [f"# HELP {p}{name} {help_}", f"# TYPE {p}{name} gauge"]
             for entry in sorted(self.state["tables"].values(), key=lambda e: (e["job"], e["table"])):
@@ -680,6 +693,47 @@ def release(self, label: str) -> int:
         return 0
 
 
+def not_backed_up(tables: dict, here: str, jobs: list, excluded: list, elsewhere: list) -> dict:
+    """The tables of {table: [hosts]} that no job here backs up and that
+    aren't excluded (never backed up) or backed up elsewhere (by another
+    host). A job only backs up the tables that exist on this host."""
+    match = lambda t, patterns: any(fnmatch.fnmatchcase(t, p) for p in patterns)
+    return {
+        t: hosts for t, hosts in tables.items()
+        if not (here in hosts and any(j.includes(t) and not j.excludes(t) for j in jobs))
+        and not match(t, excluded) and not match(t, elsewhere)
+    }
+
+
+def check_cluster(conf: dict, metrics: "Metrics") -> None:
+    """Report the tables on any node of the cluster that no backup covers,
+    e.g. a new table created on one node only"""
+    cluster = conf.get("check_cluster", "")
+    if not cluster:
+        return
+    jobs = [BackupJob(j, metrics) for j in job_confs(conf, [])]
+    try:
+        here = jobs[0].ch.query("SELECT hostName()").strip()
+        rows = jobs[0].ch.rows(
+            f"""
+            SELECT name, groupUniqArray(hostName()) AS hosts
+            FROM clusterAllReplicas({sql_str(cluster)}, system.tables)
+            WHERE database = {sql_str(jobs[0].db)} AND NOT is_temporary AND NOT startsWith(name, '.inner')
+            GROUP BY name
+            """
+        )
+    except Exception as e:
+        log.error(f"could not list the tables of cluster {cluster}: {e}")
+        return
+    missing = not_backed_up(
+        {r["name"]: r["hosts"] for r in rows}, here, jobs,
+        conf.get("exclude_tables", []), conf.get("backed_up_elsewhere", []),
+    )
+    for table, hosts in sorted(missing.items()):
+        log.error(f"{table} (on {', '.join(sorted(hosts))}) is in no backup job and not excluded")
+    metrics.not_backed_up(missing)
+
+
 def jobs_for_tables(jobs: list, tables: list) -> tuple:
     """(the jobs that back up one of tables, the tables no job backs up)"""
     owns = lambda j, t: j.includes(t) and not j.excludes(t)
@@ -766,6 +820,7 @@ def main() -> int:
                 rc |= job.release(args.label)
         if args.command == "run":
             metrics.forget_jobs(configured)
+            check_cluster(conf, metrics)
         return rc
 
 
diff --git a/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2 b/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2
index bc591a0e..01c7b3ca 100644
--- a/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2
+++ b/ansible/roles/clickhouse_backup/templates/clickhouse-backup.json.j2
@@ -12,6 +12,8 @@
   "keep_chains": clickhouse_backup_keep_chains,
   "max_backup_bandwidth": clickhouse_backup_max_bandwidth,
   "exclude_tables": clickhouse_backup_exclude_tables,
+  "check_cluster": clickhouse_backup_check_cluster,
+  "backed_up_elsewhere": clickhouse_backup_backed_up_elsewhere,
   "metrics_file": clickhouse_backup_metrics_file,
   "state_file": clickhouse_backup_state_file,
   "jobs": clickhouse_backup_jobs,
diff --git a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
index 3ff312dd..72d1759a 100644
--- a/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
+++ b/ansible/roles/clickhouse_backup/tests/alert_rules_test.yml
@@ -115,3 +115,16 @@ tests:
             exp_annotations:
               summary: 'Stale `textfile` at data3'
               description: '`/var/lib/node_exporter/textfile_collector/other.prom` is 3h old.'
+
+  # TableNotBackedUp: a table no backup job covers
+  - interval: 1m
+    input_series:
+      - {series: 'ooni_clickhouse_backup_table_not_backed_up{instance="data3", table="made_on_data1", hosts="data1.htz-fsn.prod.ooni.nu"}', values: '1x30'}
+    alert_rule_test:
+      - eval_time: 5m
+        alertname: ClickHouseTableNotBackedUp
+        exp_alerts:
+          - exp_labels: {instance: data3, table: made_on_data1, hosts: data1.htz-fsn.prod.ooni.nu}
+            exp_annotations:
+              summary: 'ClickHouse table made_on_data1 is not backed up (on data1.htz-fsn.prod.ooni.nu)'
+              description: 'No backup job covers it and it is not excluded. Add it to a job (clickhouse_backup_data1_tables if only data1 holds its data, clickhouse_backup_private_tables if private), or to clickhouse_backup_exclude_tables if it needs no backup. See the runbook, "ClickHouse backup".'
diff --git a/ansible/roles/clickhouse_backup/tests/test_e2e.py b/ansible/roles/clickhouse_backup/tests/test_e2e.py
index f4c59636..f2dd11b5 100644
--- a/ansible/roles/clickhouse_backup/tests/test_e2e.py
+++ b/ansible/roles/clickhouse_backup/tests/test_e2e.py
@@ -128,8 +128,8 @@ def setup():
         # ~20 MB that doesn't compress: takes ~20 s to back up with work/slow.json
         "CREATE TABLE ooni.obs_web (measurement_uid String, blob String) ENGINE = MergeTree ORDER BY measurement_uid",
         "INSERT INTO ooni.obs_web SELECT toString(number), randomString(4096) FROM numbers(5000)",
-        "CREATE TABLE ooni.fingerprints_dns (name String, pattern String) ENGINE = EmbeddedRocksDB PRIMARY KEY name",
-        "INSERT INTO ooni.fingerprints_dns VALUES ('a', '1.2.3.4'), ('b', '5.6.7.8')",
+        "CREATE TABLE ooni.test_groups (name String, pattern String) ENGINE = EmbeddedRocksDB PRIMARY KEY name",
+        "INSERT INTO ooni.test_groups VALUES ('a', '1.2.3.4'), ('b', '5.6.7.8')",
         "CREATE TABLE ooni.msmt_feedback (measurement_uid String, comment String) ENGINE = MergeTree ORDER BY measurement_uid",
         "INSERT INTO ooni.msmt_feedback VALUES ('m1', 'blocked?')",
         "CREATE TABLE ooni.labeling_frames (measurement_uid String) ENGINE = MergeTree ORDER BY measurement_uid",
@@ -144,11 +144,11 @@ def test_first_run():
     r = backup("backup", "run")
     check(r.returncode == 0, "the run succeeds", r.stderr)
     for bucket, table in [(MEASUREMENTS, "fastpath"), (MEASUREMENTS, "obs_web"),
-                          (MEASUREMENTS, "fingerprints_dns"), (PRIVATE, "msmt_feedback")]:
+                          (MEASUREMENTS, "test_groups"), (PRIVATE, "msmt_feedback")]:
         c = chains(bucket, table)
         check(len(c) == 1 and all(all(b.values()) for b in c.values()), f"{table}: one complete chain in {bucket}")
-    check(any(k.endswith("/data.native.zst") for k in keys(MEASUREMENTS, "prod/ooni/fingerprints_dns/")),
-          "fingerprints_dns (EmbeddedRocksDB): its rows are exported too")
+    check(any(k.endswith("/data.native.zst") for k in keys(MEASUREMENTS, "prod/ooni/test_groups/")),
+          "test_groups (EmbeddedRocksDB): its rows are exported too")
     check(not keys(MEASUREMENTS, "prod/ooni/msmt_feedback/"), "msmt_feedback isn't in the measurements bucket")
     check(not keys(MEASUREMENTS, "prod/ooni/asnmeta_tmp/"), "asnmeta_tmp (excluded) isn't backed up")
     check(not keys(MEASUREMENTS, "prod/ooni/labeling_frames/"), "labeling_frames (data1's) isn't backed up")
@@ -193,13 +193,13 @@ def test_incremental_and_restore():
         check(digest(f"restored.fastpath_{i}") == state, f"{what} restores fastpath as it was then ({state.split()[0]} rows)")
 
     # EmbeddedRocksDB: BACKUP keeps only the schema, the rows are in data.native.zst (the runbook's steps 2 and 3)
-    c = chains(MEASUREMENTS, "fingerprints_dns")
+    c = chains(MEASUREMENTS, "test_groups")
     chain = max(ch_ for ch_, b in c.items() if any(b.values()))
-    url = backup_url(MEASUREMENTS, "fingerprints_dns", chain, max(n for n, ok in c[chain].items() if ok))
-    ch(f"RESTORE TABLE ooni.fingerprints_dns AS restored.fingerprints_dns FROM S3('{url}')")
-    ch(f"INSERT INTO restored.fingerprints_dns SELECT * FROM s3('{url}/data.native.zst', 'Native')")
-    check(digest("restored.fingerprints_dns") == digest("ooni.fingerprints_dns"),
-          "fingerprints_dns (EmbeddedRocksDB) restores with its exported rows")
+    url = backup_url(MEASUREMENTS, "test_groups", chain, max(n for n, ok in c[chain].items() if ok))
+    ch(f"RESTORE TABLE ooni.test_groups AS restored.test_groups FROM S3('{url}')")
+    ch(f"INSERT INTO restored.test_groups SELECT * FROM s3('{url}/data.native.zst', 'Native')")
+    check(digest("restored.test_groups") == digest("ooni.test_groups"),
+          "test_groups (EmbeddedRocksDB) restores with its exported rows")
 
 
 def test_failure_alert(emails_before):
diff --git a/ansible/roles/clickhouse_backup/tests/test_job.py b/ansible/roles/clickhouse_backup/tests/test_job.py
index 1eeacbd1..51417fc1 100644
--- a/ansible/roles/clickhouse_backup/tests/test_job.py
+++ b/ansible/roles/clickhouse_backup/tests/test_job.py
@@ -153,6 +153,32 @@ def test_an_unreadable_state_file_starts_empty(self):
         self.assertEqual(m.state["running"], {"measurements": 1})
 
 
+class TestNotBackedUp(unittest.TestCase):
+    def test_tables_no_job_covers(self):
+        jobs = [job("measurements", include=["*"], exclude=["msmt_feedback", "labeling_frames", "*_tmp"]),
+                job("private", include=["msmt_feedback"])]
+        tables = {
+            "fastpath": ["data1", "data2", "data3"],         # replicated: measurements
+            "msmt_feedback": ["data1", "data2", "data3"],    # private
+            "labeling_frames": ["data1", "data2", "data3"],  # data1 backs it up
+            "asnmeta_tmp": ["data1"],                         # excluded
+            "fingerprints_dns": ["data1", "data2"],           # excluded: rebuilt hourly
+            "made_on_data1": ["data1"],                       # nothing: data3 doesn't have it
+        }
+        missing = backup.not_backed_up(tables, "data3", jobs, ["*_tmp", "fingerprints_*"], ["labeling_frames"])
+        self.assertEqual(missing, {"made_on_data1": ["data1"]})
+
+    def test_the_metric_names_the_table_and_its_hosts(self):
+        d = tempfile.mkdtemp()
+        m = backup.Metrics(os.path.join(d, "out.prom"), os.path.join(d, "state.json"))
+        m.not_backed_up({"made_on_data1": ["data2", "data1"]})
+        with open(m.metrics_file) as f:
+            self.assertIn('ooni_clickhouse_backup_table_not_backed_up{table="made_on_data1",hosts="data1,data2"} 1', f.read())
+        m.not_backed_up({})
+        with open(m.metrics_file) as f:
+            self.assertNotIn("made_on_data1", f.read())
+
+
 class TestClickHouseErrors(unittest.TestCase):
     def setUp(self):
         d = tempfile.mkdtemp()
diff --git a/ansible/roles/prometheus/files/alert_rules.yml b/ansible/roles/prometheus/files/alert_rules.yml
index 1149f04f..a2f33105 100644
--- a/ansible/roles/prometheus/files/alert_rules.yml
+++ b/ansible/roles/prometheus/files/alert_rules.yml
@@ -156,6 +156,14 @@ groups:
       summary: 'ClickHouse backup of {{ $labels.table }} ({{ $labels.backup }}) failed on {{ $labels.instance }}'
       description: 'Nothing was deleted: the older backups of {{ $labels.table }} are kept. The next run retries it. See `journalctl --identifier ooni-clickhouse-backup` on {{ $labels.instance }}.'
 
+  # A table on the cluster that no backup job covers, and that isn't excluded:
+  # e.g. a new table created on one node only (data3 checks after each run)
+  - alert: ClickHouseTableNotBackedUp
+    expr: ooni_clickhouse_backup_table_not_backed_up == 1
+    annotations:
+      summary: 'ClickHouse table {{ $labels.table }} is not backed up (on {{ $labels.hosts }})'
+      description: 'No backup job covers it and it is not excluded. Add it to a job (clickhouse_backup_data1_tables if only data1 holds its data, clickhouse_backup_private_tables if private), or to clickhouse_backup_exclude_tables if it needs no backup. See the runbook, "ClickHouse backup".'
+
   # A job's run finished with every table backed up. One Slack message per
   # job run, no email and no "resolved" message: Alertmanager routes it to
   # slack-notice. It clears an hour after the run.
diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index d2735a9a..d4e88919 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -205,7 +205,12 @@ When a data1-only table becomes replicated, remove it from
   backup.
 - Not backed up: `fingerprints_dns` and `fingerprints_http`, which are only on
   data1 and data2 and which the hourly updater rebuilds, and `session_expunge`,
-  which is empty and no longer used.
+  which is empty and no longer used. They're in
+  `clickhouse_backup_exclude_tables`, with the other tables that are never
+  backed up.
+- After each run, data3 lists the tables on every node of the cluster
+  (`clickhouse_backup_check_cluster`) and reports any that no job backs up and
+  that aren't excluded, e.g. a new table created on data1 only.
 - R2 rejects multipart uploads whose parts (all but the last) differ in
   size, so `clickhouse_backup_s3_settings` fixes ClickHouse's part size at
   128 MiB (`s3_upload_part_size_multiply_factor: 1`). R2 allows 10,000 parts,
@@ -264,6 +269,12 @@ Alerts, in `alert_rules.yml`:
 
 - `ClickHouseBackupFailed`: a table's last attempt failed. Nothing is
   deleted, and the next run retries it.
+- `ClickHouseTableNotBackedUp`: a table on the cluster is in no backup job and
+  not excluded. Add it to a job: replicated tables are backed up by data3's
+  `measurements` job unless they're private (`clickhouse_backup_private_tables`),
+  and a table whose data only data1 holds goes in
+  `clickhouse_backup_data1_tables`. Or, if it needs no backup, add it to
+  `clickhouse_backup_exclude_tables`.
 - `ClickHouseBackupCompleted`: a job's run backed up every table. One Slack
   message per run in `#ooni-bots`, no email.
 - `ClickHouseBackupStale`: a table hasn't been backed up for 8 days (backups

From 70ba6e706470ba5a2b7baa54d7e9e4f1dd6d363c Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Thu, 22 Oct 2026 15:08:47 +0000
Subject: [PATCH 27/32] clickhouse_backup tests: a second ClickHouse node, and
 a table on it that no backup covers

The test environment gets a second ClickHouse node, data1, with data3 in
one test cluster. A new step creates a table on data1 only and checks
that ClickHouseTableNotBackedUp names it in Slack, and no other table
(labeling_frames is data1's, asnmeta_tmp is excluded). It comes after
the Completed step, whose check that nothing went to email it would
otherwise break.
---
 .../roles/clickhouse_backup/tests/README.md   |  6 ++--
 .../clickhouse_backup/tests/clickhouse.xml    |  9 ++++++
 .../roles/clickhouse_backup/tests/compose.yml | 16 ++++++++++
 .../roles/clickhouse_backup/tests/render.yml  |  1 +
 .../roles/clickhouse_backup/tests/test_e2e.py | 29 +++++++++++++++----
 .../roles/clickhouse_backup/tests/vars.yml    |  6 +++-
 6 files changed, 58 insertions(+), 9 deletions(-)

diff --git a/ansible/roles/clickhouse_backup/tests/README.md b/ansible/roles/clickhouse_backup/tests/README.md
index 77eaac1f..cf2e9b50 100644
--- a/ansible/roles/clickhouse_backup/tests/README.md
+++ b/ansible/roles/clickhouse_backup/tests/README.md
@@ -10,12 +10,14 @@
   bucket stubbed.
 - `test_job.py`: how a job runs and what it reports, ClickHouse stubbed.
 - `test_e2e.py`: the real thing, in the containers of `compose.yml`:
-  ClickHouse at production's version, MinIO standing in for R2, the script
+  two ClickHouse nodes at production's version (`data3`, where the backups
+  run, and `data1`, in one test cluster), MinIO standing in for R2, the script
   on Debian bookworm with its python3-boto3, as on the hosts, node_exporter,
   Prometheus with the repo's alert rules, Alertmanager with the repo's
   routes, and a stand-in for Slack. Full and incremental backups, and
   restoring the full and each incremental of a chain, and an
-  EmbeddedRocksDB table with its exported rows; a failed run; an
+  EmbeddedRocksDB table with its exported rows; a table created on `data1`
+  only, which ClickHouseTableNotBackedUp must name; a failed run; an
   interrupted full, the month's new full, pruning and restoring what's
   left; and the Slack messages and emails each one causes.
 
diff --git a/ansible/roles/clickhouse_backup/tests/clickhouse.xml b/ansible/roles/clickhouse_backup/tests/clickhouse.xml
index 918706bd..4410ded5 100644
--- a/ansible/roles/clickhouse_backup/tests/clickhouse.xml
+++ b/ansible/roles/clickhouse_backup/tests/clickhouse.xml
@@ -4,4 +4,13 @@
     <asynchronous_metric_log remove="1"/>
     <trace_log remove="1"/>
     <max_server_memory_usage_to_ram_ratio>0.5</max_server_memory_usage_to_ram_ratio>
+    <!-- The two test nodes, for the check of tables no backup covers -->
+    <remote_servers>
+        <test_cluster>
+            <shard>
+                <replica><host>clickhouse</host><port>9000</port><user>admin</user><password>admin</password></replica>
+                <replica><host>clickhouse2</host><port>9000</port><user>admin</user><password>admin</password></replica>
+            </shard>
+        </test_cluster>
+    </remote_servers>
 </clickhouse>
diff --git a/ansible/roles/clickhouse_backup/tests/compose.yml b/ansible/roles/clickhouse_backup/tests/compose.yml
index e90a9881..0f8eea78 100644
--- a/ansible/roles/clickhouse_backup/tests/compose.yml
+++ b/ansible/roles/clickhouse_backup/tests/compose.yml
@@ -12,8 +12,10 @@ services:
       MINIO_ROOT_PASSWORD: testsecret123
     ports: ["127.0.0.1:19910:9000"]
 
+  # data3, the node the backups run against
   clickhouse:
     image: docker.io/clickhouse/clickhouse-server:24.8.6.70
+    hostname: data3
     environment:
       CLICKHOUSE_USER: admin
       CLICKHOUSE_PASSWORD: admin
@@ -25,6 +27,20 @@ services:
     mem_limit: 2g
     ports: ["127.0.0.1:19911:8123"]
 
+  # another node of the cluster, where a table nobody backs up gets created
+  clickhouse2:
+    image: docker.io/clickhouse/clickhouse-server:24.8.6.70
+    hostname: data1
+    environment:
+      CLICKHOUSE_USER: admin
+      CLICKHOUSE_PASSWORD: admin
+    volumes:
+      - ./clickhouse.xml:/etc/clickhouse-server/config.d/test.xml:ro,z
+    ulimits:
+      nofile: 262144
+    mem_limit: 1g
+    ports: ["127.0.0.1:19912:8123"]
+
   runner:
     build: {context: ., dockerfile: runner.Dockerfile}
     command: sleep infinity
diff --git a/ansible/roles/clickhouse_backup/tests/render.yml b/ansible/roles/clickhouse_backup/tests/render.yml
index a42ae1ef..62c84e3a 100644
--- a/ansible/roles/clickhouse_backup/tests/render.yml
+++ b/ansible/roles/clickhouse_backup/tests/render.yml
@@ -14,6 +14,7 @@
       vars:
         clickhouse_backup_jobs: "{{ item.jobs }}"
         clickhouse_backup_max_bandwidth: "{{ item.bandwidth }}"
+        clickhouse_backup_check_cluster: "{{ item.check_cluster | default('') }}"
       loop: "{{ test_configs }}"
 
     - name: Render ClickHouse's bucket credentials, for every bucket of every config
diff --git a/ansible/roles/clickhouse_backup/tests/test_e2e.py b/ansible/roles/clickhouse_backup/tests/test_e2e.py
index f2dd11b5..eb70e3ec 100644
--- a/ansible/roles/clickhouse_backup/tests/test_e2e.py
+++ b/ansible/roles/clickhouse_backup/tests/test_e2e.py
@@ -4,8 +4,9 @@
   2. ClickHouseBackupCompleted reaches Slack once per job, as a notice, without email
   3. two more runs take incrementals; restoring the full and each incremental returns the rows of
      its moment, and an EmbeddedRocksDB table comes back with its exported rows
-  4. a run whose bucket is missing fails, and ClickHouseBackupFailed reaches Slack and email
-  5. a new month's full interrupted by killing ClickHouse deletes nothing; the next run
+  4. a table created on the cluster's other node only: ClickHouseTableNotBackedUp names it
+  5. a run whose bucket is missing fails, and ClickHouseBackupFailed reaches Slack and email
+  6. a new month's full interrupted by killing ClickHouse deletes nothing; the next run
      removes what it left, takes the full and prunes the oldest chain, and the full restores
 Prints each check; exits 1 at the first failure."""
 import json
@@ -42,8 +43,9 @@ def http(url, data=None):
         return r.read().decode()
 
 
-def ch(sql):
-    return http("http://127.0.0.1:19911/?user=admin&password=admin", sql.encode()).strip()
+def ch(sql, port=19911):
+    """a query on data3, or with port=19912 on the cluster's other node, data1"""
+    return http(f"http://127.0.0.1:{port}/?user=admin&password=admin", sql.encode()).strip()
 
 
 def backup(config, *args):
@@ -113,6 +115,7 @@ def months_ago(n):
 def setup():
     print("== waiting for the services")
     wait_for("ClickHouse", lambda: ch("SELECT 1") == "1")
+    wait_for("ClickHouse's other node", lambda: ch("SELECT 1", port=19912) == "1")
     wait_for("MinIO", lambda: http("http://127.0.0.1:19910/minio/health/live") == "")
     wait_for("Prometheus", lambda: "Ready" in http("http://127.0.0.1:19990/-/ready"))
     wait_for("Alertmanager", lambda: "OK" in http("http://127.0.0.1:19993/-/ready"))
@@ -137,6 +140,7 @@ def setup():
         "CREATE TABLE ooni.asnmeta_tmp (asn UInt32) ENGINE = MergeTree ORDER BY asn",  # excluded: *_tmp
     ]:
         ch(sql)
+    ch("CREATE DATABASE ooni", port=19912)
 
 
 def test_first_run():
@@ -165,6 +169,7 @@ def test_completed_alert(emails_before):
     check(email_attempts() == emails_before, "no email sent")
 
 
+
 def digest(table):
     """row count and a checksum of every row"""
     return ch(f"SELECT count(), sum(cityHash64(*)) FROM {table}")
@@ -176,6 +181,8 @@ def backup_url(bucket, table, chain, name):
 
 def test_incremental_and_restore():
     print("== 3. incrementals, and restoring each backup of a chain")
+    # for step 4: a table created on data1 alone, which no job backs up
+    ch("CREATE TABLE ooni.made_on_data1 (x UInt8) ENGINE = MergeTree ORDER BY x", port=19912)
     # fastpath as at the first run's full, then after each of two more runs
     states = [digest("ooni.fastpath")]
     for first in (200000, 201000):
@@ -202,8 +209,17 @@ def test_incremental_and_restore():
           "test_groups (EmbeddedRocksDB) restores with its exported rows")
 
 
+def test_not_backed_up_alert():
+    print("== 4. a table no backup covers: ClickHouseTableNotBackedUp")
+    title = "[FIRING] ClickHouse table made_on_data1 is not backed up (on data1)"
+    msgs = wait_for(f"a Slack message: {title}", lambda: [m for m in slack() if m[0] == title])
+    check("Add it to a job" in msgs[0][1], f"it says what to do: {msgs[0][1][:90]!r}...")
+    others = [t for t, _ in slack() if "is not backed up" in t and t != title]
+    check(not others, f"no other table is reported (labeling_frames is data1's, asnmeta_tmp excluded): {others}")
+
+
 def test_failure_alert(emails_before):
-    print("== 4. a run that fails: ClickHouseBackupFailed")
+    print("== 5. a run that fails: ClickHouseBackupFailed")
     r = backup("broken", "run")
     check(r.returncode == 1, "the run exits 1", r.stderr)
     title = f"[FIRING] ClickHouse backup of labeling_frames (broken) failed on {DATA3}"
@@ -214,7 +230,7 @@ def test_failure_alert(emails_before):
 
 
 def test_interrupted_full():
-    print("== 5. a new month's full, interrupted, then completed")
+    print("== 6. a new month's full, interrupted, then completed")
     (chain, backups), = chains(MEASUREMENTS, "obs_web").items()
     full = f"prod/ooni/obs_web/{chain}/{min(backups)}/"
     old, previous = months_ago(2), months_ago(1)
@@ -256,6 +272,7 @@ def main():
     test_first_run()
     test_completed_alert(emails_before)
     test_incremental_and_restore()
+    test_not_backed_up_alert()
     test_failure_alert(emails_before)
     test_interrupted_full()
     print("PASS")
diff --git a/ansible/roles/clickhouse_backup/tests/vars.yml b/ansible/roles/clickhouse_backup/tests/vars.yml
index 06835714..33957689 100644
--- a/ansible/roles/clickhouse_backup/tests/vars.yml
+++ b/ansible/roles/clickhouse_backup/tests/vars.yml
@@ -17,11 +17,15 @@ test_jobs:
     include_tables: [msmt_feedback]
     exclude_tables: []
 
+# data1 backs up its own labeling_frames
+clickhouse_backup_backed_up_elsewhere: [labeling_frames]
+
 # One script config per entry, in work/<name>.json
 test_configs:
-  - name: backup  # as deployed
+  - name: backup  # as deployed on data3
     bandwidth: 0
     jobs: "{{ test_jobs }}"
+    check_cluster: test_cluster
   - name: slow  # ~1 MB/s, so a full backup can be interrupted
     bandwidth: 3000000
     jobs: "{{ test_jobs }}"

From f041162af49b3138cfee19f59c865e54d28e4287 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Fri, 23 Oct 2026 10:07:28 +0000
Subject: [PATCH 28/32] Runbook: restore a view's TO table before the view

While a materialized view exists without its target table, every insert
into the view's source table fails: for counters_asn_test_list, fastpath's
inserts. Tested on 24.8. It matters once ooni/devops#521's follow-up gives
counters_asn_test_list an explicit target table.
---
 docs/Runbooks.md | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index d4e88919..aa0fa228 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -345,7 +345,9 @@ each table. Restoring the latest incremental restores the whole chain.
    CREATE DATABASE IF NOT EXISTS ooni
    ```
 2. Restore tables before the materialized views that read from them, on
-   one node. A replicated table that still exists makes `RESTORE` fail with
+   one node, and a view's `TO` table before the view: while a view exists
+   without its table, every insert into the view's source fails (fastpath's,
+   for `counters_asn_test_list`). A replicated table that still exists makes `RESTORE` fail with
    `REPLICA_ALREADY_EXISTS`: drop it (`DROP TABLE ... SYNC`) or restore on
    another server.
    ```sql

From 0cdf727a13a54b7d3e23dae77dcd52b5758dad56 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Tue, 27 Oct 2026 18:47:31 +0000
Subject: [PATCH 29/32] Fix node_exporter.service.j2 whitespace

---
 .../prometheus_node_exporter/templates/node_exporter.service.j2 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ansible/roles/prometheus_node_exporter/templates/node_exporter.service.j2 b/ansible/roles/prometheus_node_exporter/templates/node_exporter.service.j2
index 31a822b0..29800e7d 100644
--- a/ansible/roles/prometheus_node_exporter/templates/node_exporter.service.j2
+++ b/ansible/roles/prometheus_node_exporter/templates/node_exporter.service.j2
@@ -4,7 +4,7 @@ Description=NodeExporter
 [Service]
 TimeoutStartSec=0
 User=node_exporter
-ExecStart={{ node_exporter_bin_path }} --web.listen-address={{ node_exporter_host }}:{{ node_exporter_port }} {{ node_exporter_options }}{% if node_exporter_textfile_dir %} --collector.textfile.directory={{ node_exporter_textfile_dir }}{% endif %}
+ExecStart={{ node_exporter_bin_path }} --web.listen-address={{ node_exporter_host }}:{{ node_exporter_port }} {{ node_exporter_options }}{{ (' --collector.textfile.directory=' ~ node_exporter_textfile_dir) if node_exporter_textfile_dir else '' }}
 Restart={{ node_exporter_restart }}
 
 [Install]

From 201d8b93d9836e91fd4d75f93e2b75ba282e1b23 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Wed, 28 Oct 2026 07:30:27 +0000
Subject: [PATCH 30/32] clickhouse_backup: one part in flight per file, cap at
 100 MB/s

With the default of 20 in-flight parts per file, disk seek on data3
caused so much degredation uploads timed out and nothing finished.
Set s3_max_inflight_parts_for_one_file to 1, and raise
max_backup_bandwidth to 100 MB/s.
---
 ansible/roles/clickhouse_backup/defaults/main.yml | 11 +++++------
 .../files/ooni_clickhouse_backup.py               |  3 +--
 docs/Runbooks.md                                  | 15 ++++++---------
 3 files changed, 12 insertions(+), 17 deletions(-)

diff --git a/ansible/roles/clickhouse_backup/defaults/main.yml b/ansible/roles/clickhouse_backup/defaults/main.yml
index 6ae23389..f33e38ef 100644
--- a/ansible/roles/clickhouse_backup/defaults/main.yml
+++ b/ansible/roles/clickhouse_backup/defaults/main.yml
@@ -30,10 +30,13 @@ clickhouse_backup_s3_secret_access_key: ""
 # whose parts (all but the last) differ in size, and ClickHouse grows the
 # part size as an upload goes on unless the factor is 1. R2 also allows at
 # most 10,000 parts, so with 128 MiB parts one file can reach 1.25 TiB.
+# One part in flight per file: with the default of 20, the parts in flight
+# shared the bandwidth cap so thinly that their uploads timed out.
 clickhouse_backup_s3_settings:
   s3_upload_part_size_multiply_factor: 1
   s3_min_upload_part_size: 134217728
   s3_max_upload_part_size: 134217728
+  s3_max_inflight_parts_for_one_file: 1
 
 # One job per bucket. A job backs up the tables matching include_tables ("*"
 # for every table; an empty list backs up none) minus the excludes below and
@@ -47,12 +50,8 @@ clickhouse_backup_jobs:
 # Complete chains kept per table, besides chains pinned with --keep: the
 # current one and the one before, so a month or two of restore points
 clickhouse_backup_keep_chains: 2
-# Bytes/s passed to ClickHouse as max_backup_bandwidth: at most 50 MB/s, 40%
-# of the 1 Gbit/s link, leaving the rest for replication and queries.
-# Against MinIO in testing, 24.8 uploaded at about a third of this value; check
-# the real rate on the first backups (ooni_clickhouse_backup_last_size_bytes /
-# ooni_clickhouse_backup_last_duration_seconds) and raise it if they're too slow.
-clickhouse_backup_max_bandwidth: 50000000
+# Bytes/s passed to ClickHouse as max_backup_bandwidth: 100 MB/s
+clickhouse_backup_max_bandwidth: 100000000
 
 # fnmatch patterns, for every job. Materialized views' .inner tables are
 # always skipped: they're backed up with their view.
diff --git a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
index e743fb5c..dd00dd23 100644
--- a/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
+++ b/ansible/roles/clickhouse_backup/files/ooni_clickhouse_backup.py
@@ -431,8 +431,7 @@ def __init__(self, conf: dict, metrics: Metrics = None):
         bw = int(conf.get("max_backup_bandwidth", 0))
         if bw:
             # A query setting, not a BACKUP setting: 24.8 rejects it inside
-            # BACKUP ... SETTINGS. Against MinIO in testing, the throughput
-            # was about a third of this value.
+            # BACKUP ... SETTINGS.
             self.settings["max_backup_bandwidth"] = str(bw)
 
     def tables(self, only: list) -> tuple:
diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index aa0fa228..f4b44bdb 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -217,12 +217,10 @@ When a data1-only table becomes replicated, remove it from
   so one backup file can reach 1.25 TiB.
 - Two complete chains are kept per table (`clickhouse_backup_keep_chains`),
   so a month or two of restore points.
-- Bandwidth is capped with `clickhouse_backup_max_bandwidth`, at 50 MB/s. A
-  new full backup of everything (~5.6 TB) takes about 31 hours at that rate.
-  Against MinIO in testing, 24.8 uploaded at only about a third of the cap,
-  which would make it about 4 days: check the rate of the first backups
-  (`ooni_clickhouse_backup_last_size_bytes / ooni_clickhouse_backup_last_duration_seconds`)
-  and raise the cap if they're that slow.
+- Bandwidth is capped with `clickhouse_backup_max_bandwidth`, at 100 MB/s.
+- One part uploads at a time per file (`s3_max_inflight_parts_for_one_file: 1`
+  in `clickhouse_backup_s3_settings`). With ClickHouse's default of 20, the
+  parts in flight shared the cap so thinly that their uploads timed out.
 
 Layout in the bucket:
 
@@ -253,9 +251,8 @@ without a restart.
    ```
    and the alert rules with `deploy-monitoring.yml`.
 4. The timer takes the first (full) backup at its next run, on Sunday at
-   02:30 UTC. A full backup of everything takes about 31 hours, or longer if
-   the upload rate is below the cap. To start it on deploy day instead, e.g. a
-   Friday so that it runs over the weekend:
+   02:30 UTC. To start it on deploy day instead, e.g. a Friday so that it runs
+   over the weekend:
    ```bash
    sudo systemctl start --no-block ooni-clickhouse-backup.service
    sudo journalctl -f --identifier ooni-clickhouse-backup

From 8a724692df272d80aa743c17631b59932ccb0be1 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Wed, 28 Oct 2026 10:51:02 +0100
Subject: [PATCH 31/32] fix incorrect comment

---
 docs/Runbooks.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index f4b44bdb..9a073617 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -187,8 +187,8 @@ Each host runs the jobs set in its host_vars, one after the other:
 | data3 | `private` | `ooni-clickhouse-backup-private` | `clickhouse_backup_private_tables`: `msmt_feedback`, `faulty_measurements`, `url_priorities` |
 | data1 | `data1` | `ooni-clickhouse-backup` | `clickhouse_backup_data1_tables`: plain MergeTree tables whose data only data1 holds |
 
-data3 takes the replicated tables because it isn't behind clickhouseproxy
-and doesn't run Airflow; replicated tables hold the same data on every node.
+data3 runs the backup for the replicated tables because it isn't on a critical
+path like data1 (fastpath, airflow) or data2 (oonimeasurements).
 Private tables have their own bucket because R2 tokens grant access per
 bucket, and researchers may be given read access to the measurement tables.
 When a data1-only table becomes replicated, remove it from

From 8d2de3719279c5544d0a546b9cb86eebd148b5a3 Mon Sep 17 00:00:00 2001
From: Aaron Gibson <aagbsn@extc.org>
Date: Wed, 28 Oct 2026 10:59:14 +0100
Subject: [PATCH 32/32] add how to reload config; SYSTEM CONFIG RELOAD did not
 work

---
 docs/Runbooks.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/Runbooks.md b/docs/Runbooks.md
index 9a073617..7cff702a 100644
--- a/docs/Runbooks.md
+++ b/docs/Runbooks.md
@@ -233,7 +233,7 @@ Never put S3 keys in `BACKUP`/`RESTORE`/`s3()` queries: an incremental
 backup stores its `base_backup` argument, keys included, in plain text in
 its `.backup` file. ClickHouse gets the keys from
 `/etc/clickhouse-server/config.d/ooni_backup_s3.xml`, which it reloads
-without a restart.
+without a restart using systemctl reload clickhouse-server.
 
 ### Setting up
 1. In the Cloudflare dashboard, create the R2 buckets `ooni-clickhouse-backup`