Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
"""Connectors for datasets published by the Scooby Team."""

# Keep this a regular package, not a namespace package. pkgutil.walk_packages finds the
# connector packages below it only through this __init__.py.
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# scooby-team/swiss-grid-anomaly-cot

Weekly windows of the Swiss electricity system, 2020-2025, with explained load anomalies. Every
label is computed by rules from public data; no label comes from a model.

## Records

One Record per ISO week (312 weeks), with two sources of hourly signals:

- **Swissgrid, Swiss control block** (MW): end-user consumption, total consumption, production,
and the net import from Austria, Germany, France and Italy. The connector sums Swissgrid's
quarter-hourly energies (kWh) into hourly mean power.
- **MeteoSwiss SwissMetNet** (12 stations): air temperature (°C) and global radiation (W/m²), each
a mean weighted by the 2023 BFS population of the cantons around each station.

A week has 168 hours, or 167 / 169 in the weeks of a clock change. Everything is stored in UTC.

## Tasks

| task | type | target |
| --- | --- | --- |
| cause | `ClassificationTask` | cause of the week's largest anomaly, or `none` (9 labels) |
| days | `TemporalLocalizationTask` | every anomalous day of the week, possibly none |
| exchange | `AnswerTask` | net importer or exporter, and through which borders |

Each task carries a rationale written by fixed templates from the computed facts. A check rejects
any number in the text that is not one of the facts, which the record keeps as a JSON annotation.

**The prompts name no holiday.** They give the week's dates and describe the series. Most anomalous
days are holidays, and rules that read only the holiday calendar, with no load data, reach a cause
accuracy of 0.98 and a day F1 of 0.81 on the 2025 test weeks: with the calendar in the prompt, a
model could answer without reading the series. The holidays stay in each record as
`public_holiday` annotations, and the rationales name them.

## How the labels are computed

1. A daily linear baseline (weekday type × month, heating and cooling degree days, lagged heating
degree days, radiation, trend), fitted on the ordinary days of 2020-2023 only, gives the
expected end-user consumption. Its MAPE is 2.3 % on the ordinary days of the 2025 test year.
2. A day is anomalous when its robust z-score is at least 3 in absolute value.
3. Ordered rules give the cause: `covid_lockdown`, `christmas_newyear` (22 Dec - 6 Jan),
`public_holiday` (a holiday for at least 20 % of the population), `bridge_day`,
`summer_holidays` (14 Jul - 15 Aug). Holidays come from `holidays` 0.108 per canton.

No day of 2020-2025 lies above expectation at this threshold, so `cold_snap`, `heatwave` and
`unexplained` are in the label set but never occur.

## Splits

208 train weeks (2020-2023), 52 validation (2024) and 52 test (2025). The baseline is fitted on
train days only and never sees a validation or test day.

## Source quirks the connector handles

- Swissgrid writes Swiss civil time. In 2020-2024 each label is the **end** of its quarter hour;
in 2025 it is the **start**. The convention is detected from the first label of each file.
- On the fall-back day the 02:xx labels appear twice. 2020 writes the fall-back instant as 03:00
CEST, 2021-2024 as 02:00 CET. Every label is checked against the wall-clock reading of its
instant, so a missing, extra or shuffled row raises.
- MeteoSwiss writes UTC, labelled at the end of each hour.

## Limitations

- **License.** Swissgrid states no license for these files, so the card says `NONE`. MeteoSwiss
data is CC BY 4.0.
- **Few causes in the test year.** 2025 holds 3 of the 9 causes (`none`, `public_holiday`,
`christmas_newyear`), and 19 anomalous days in 8 weeks, so scores on the days task are noisy.
- **Numbers in the rationales.** Some quoted numbers (the expected load, z-scores, the weather
effect) come from the baseline and cannot be read exactly from the signals.
- **Revisions.** Swissgrid re-published 2023 in March 2026, and MeteoSwiss rewrites the station
files once a year. A revised file fails its SHA-256 pin, so the build stops instead of silently
changing the labels.
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
"""Swiss Grid Anomaly-CoT: weekly Swiss power and weather windows with rule-based anomaly explanations."""

from timenet_connectors.datasets.scooby_team.swiss_grid_anomaly_cot.connector import (
CONNECTOR as CONNECTOR,
SwissGridAnomalyCotConnector as SwissGridAnomalyCotConnector,
)
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""The annotation pipeline, copied from the swissgrid_cot project by its sync script."""
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
"""Flag anomalous days from the baseline residual with a robust z-score.

rel_d = (load_d - expected_d) / expected_d
z_d = (rel_d - median_fit) / (1.4826 * MAD_fit)

The median and MAD come from the baseline's own fit days (train years, ordinary days only), so the
scale is that of normal days and nothing is learned from validation or test.

Threshold |z| >= 3.0. Chosen on train years (see reports/phase3/threshold_table.md): it flags about
25 days a year, 91 % of public holidays (share >= 0.5) and 0.6 % of ordinary days.
"""

from dataclasses import dataclass

import numpy as np
import pandas as pd

from .baseline import Baseline, fit_days


Z_THRESHOLD = 3.0


@dataclass(frozen=True)
class ResidualScale:
median: float
robust_sd: float


def residual_scale(d: pd.DataFrame, rel: pd.Series) -> ResidualScale:
r = rel[fit_days(d)]
med = float(r.median())
return ResidualScale(median=med, robust_sd=float(1.4826 * (r - med).abs().median()))


def score_days(d: pd.DataFrame, baseline: Baseline) -> tuple[pd.DataFrame, ResidualScale]:
out = d.copy()
out["expected_mw"] = baseline.predict(d)
out["rel_dev"] = (out.load_mw - out.expected_mw) / out.expected_mw
scale = residual_scale(out, out.rel_dev)
out["z"] = (out.rel_dev - scale.median) / scale.robust_sd
out["is_anomaly"] = out.z.abs() >= Z_THRESHOLD
out.loc[out.z.isna(), "is_anomaly"] = False
return out, scale


def threshold_table(scored: pd.DataFrame, thresholds=(2.0, 2.5, 3.0, 3.5, 4.0)) -> pd.DataFrame:
rows = []
train = scored[scored.split == "train"]
for t in thresholds:
a = train.z.abs() >= t
rows.append({
"threshold": t,
"flagged_days_per_year": round(float(a.mean() * 365.25), 1),
"holiday_recall": round(float(a[train.is_holiday].mean()), 3),
"ordinary_day_flag_rate": round(float(a[~train.special_day].mean()), 4),
"christmas_period_recall": round(float(a[train.christmas_period].mean()), 3),
}) # fmt: skip
return pd.DataFrame(rows)


def events(scored: pd.DataFrame) -> pd.DataFrame:
"""Merge consecutive anomalous days of the same sign into events."""
a = scored[scored.is_anomaly].copy()
if a.empty:
return pd.DataFrame(columns=["start", "end", "n_days", "sign", "peak_z"])
sign = np.sign(a.z)
new = (a.index.to_series().diff() != pd.Timedelta("1D")) | (sign != sign.shift())
a["event_id"] = new.cumsum().to_numpy()
return a.groupby("event_id").agg(
start=("z", lambda s: s.index.min()),
end=("z", lambda s: s.index.max()),
n_days=("z", "size"),
sign=("z", lambda s: int(np.sign(s.iloc[0]))),
peak_z=("z", lambda s: float(s.loc[s.abs().idxmax()])),
)
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
"""Assemble the aligned hourly national table from parsed sources. No file or network access."""

import pandas as pd

from . import calendar_ch, meteoswiss, swissgrid
from .timefields import field, normalize


TZ = swissgrid.TZ


def hourly_dataset(quarters: pd.DataFrame, stations: dict[str, pd.DataFrame]) -> pd.DataFrame:
"""Join Swissgrid hourly power, national weather and the holiday calendar.

Args:
quarters: The Swissgrid quarter hours of every year, indexed by UTC start, in kWh.
stations: The hourly frame of each SwissMetNet station, indexed by UTC start.

Returns:
One row per UTC hour with power in MW, weather, local calendar fields and holiday flags.

Raises:
SwissgridFormatError: If the yearly files overlap.
"""
if quarters.index.has_duplicates:
raise swissgrid.SwissgridFormatError("the yearly files overlap")
hourly = swissgrid.to_hourly_mw(quarters.sort_index())
population = meteoswiss.canton_population()
weather = meteoswiss.national_series(
stations, meteoswiss.station_weights(population), pd.DatetimeIndex(hourly.index)
)
df = hourly.join(weather)
local = pd.DatetimeIndex(df.index).tz_convert(TZ)
df["local_date"] = normalize(local.tz_localize(None))
df["local_hour"] = field(local, "hour")
df["dow"] = field(local, "day_of_week")
first, last = df.local_date.min(), df.local_date.max()
cal = calendar_ch.holiday_calendar(str(first.date()), str(last.date()), population)
df = df.join(cal, on="local_date")
df["year"] = df.local_date.dt.year
return df
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
"""Expected daily end-user consumption: a transparent linear model, fitted on train years only.

load_d = profile[daytype(d), month(d)]
+ b_hdd * HDD(T_d) + b_hdd_lag * HDD(T_{d-1}) + b_cdd * CDD(T_d)
+ b_rad * radiation_d + b_trend * years_since_2020

* daytype: Monday, Tuesday-Thursday, Friday, Saturday, Sunday (5 types) x 12 months = 60 levels.
* HDD(T) = max(0, 16 - T), CDD(T) = max(0, T - 20), on the daily mean population-weighted temperature.
Yesterday's HDD captures the thermal inertia of buildings.
* radiation: daily mean global radiation (W/m²). Behind-the-meter PV lowers metered consumption.
* The trend absorbs slow drift (efficiency, PV growth). It is linear in time, so its extrapolation
to the validation and test years is part of what the evaluation measures.

Days excluded from the fit, because they are exactly what the model must NOT learn as normal:
public holidays (population share >= 0.2), bridge days, 22 Dec - 6 Jan (the school holidays of
all cantons) and the 2020 COVID-19 lockdown. Fitted by ordinary least squares (numpy.linalg.lstsq). No other tuning.
"""

from dataclasses import dataclass

import numpy as np
import pandas as pd

from .timefields import field


TRAIN_YEARS = (2020, 2021, 2022, 2023)
VALIDATION_YEARS = (2024,)
TEST_YEARS = (2025,)
HDD_BASE_C = 16.0
CDD_BASE_C = 20.0
PARTIAL_HOLIDAY_SHARE = 0.2

# Swiss Federal Council: "extraordinary situation" from 16 March 2020. Step 1 of the reopening on
# 27 April 2020, step 2 (shops, schools) on 11 May 2020. The lockdown period ends the day before step 2.
LOCKDOWN = (pd.Timestamp("2020-03-16"), pd.Timestamp("2020-05-10"))


def split_of_year(year: int) -> str:
if year in TRAIN_YEARS:
return "train"
if year in VALIDATION_YEARS:
return "validation"
if year in TEST_YEARS:
return "test"
raise ValueError(f"year {year} is in no split")


def daytype(dow: pd.Series) -> pd.Series:
return dow.map({0: "mon", 1: "tue-thu", 2: "tue-thu", 3: "tue-thu", 4: "fri", 5: "sat", 6: "sun"})


CHRISTMAS_START = (12, 22)
"""(month, day) of the first day of the Christmas period."""
CHRISTMAS_END = (1, 6)
"""(month, day) of the last day of the Christmas period, in the next year."""
SUMMER_HOLIDAY_START = (7, 14)
SUMMER_HOLIDAY_END = (8, 15)


def _from(dates: pd.DatetimeIndex, month_day: tuple[int, int]) -> np.ndarray:
month, day = month_day
return (field(dates, "month") == month) & (field(dates, "day") >= day)


def _until(dates: pd.DatetimeIndex, month_day: tuple[int, int]) -> np.ndarray:
month, day = month_day
return (field(dates, "month") == month) & (field(dates, "day") <= day)


def is_christmas_period(dates: pd.DatetimeIndex) -> np.ndarray:
"""Flag 22 December - 6 January, the Christmas school holidays, which every canton has in this window.

Returns:
A boolean array, one value per date.
"""
return _from(dates, CHRISTMAS_START) | _until(dates, CHRISTMAS_END)


def is_summer_holiday_core(dates: pd.DatetimeIndex) -> np.ndarray:
"""Flag 14 July - 15 August, when the summer school holidays of nearly all cantons overlap.

Many firms close in these weeks too (construction-industry holidays). The period is not excluded
from the fit, since it is a month of ordinary days as well; it only serves to attribute a cause.

Returns:
A boolean array, one value per date.
"""
return _from(dates, SUMMER_HOLIDAY_START) | _until(dates, SUMMER_HOLIDAY_END)


def is_lockdown(dates: pd.DatetimeIndex) -> np.ndarray:
return (dates >= LOCKDOWN[0]) & (dates <= LOCKDOWN[1])


def daily_frame(hourly: pd.DataFrame) -> pd.DataFrame:
"""One row per local day with the inputs of the model and the facts used downstream."""
g = hourly.groupby("local_date")
d = g.agg(
load_mw=("load_enduser_mw", "mean"),
load_total_mw=("load_total_mw", "mean"),
production_mw=("production_mw", "mean"),
net_import_mw=("net_import_mw", "mean"),
temp_c=("temp_c", "mean"),
temp_min_c=("temp_c", "min"),
temp_max_c=("temp_c", "max"),
radiation_wm2=("radiation_wm2", "mean"),
dow=("dow", "first"),
holiday_pop_share=("holiday_pop_share", "first"),
holiday_name=("holiday_name", "first"),
is_holiday=("is_holiday", "first"),
is_bridge_day=("is_bridge_day", "first"),
n_hours=("load_enduser_mw", "size"),
)
for b in ("at", "de", "fr", "it"):
d[f"net_import_{b}_mw"] = g[f"net_import_{b}_mw"].mean()
d.index = pd.DatetimeIndex(d.index, name="date")
d["year"] = field(d.index, "year")
d["split"] = d.year.map(split_of_year)
d["daytype"] = daytype(d.dow)
d["month"] = field(d.index, "month")
d["hdd"] = np.maximum(0.0, HDD_BASE_C - d.temp_c)
d["hdd_lag1"] = d.hdd.shift(1)
d["cdd"] = np.maximum(0.0, d.temp_c - CDD_BASE_C)
d["trend_years"] = (d.index - pd.Timestamp("2020-01-01")).days / 365.25
d["christmas_period"] = is_christmas_period(d.index)
d["lockdown"] = is_lockdown(d.index)
d["summer_holiday_core"] = is_summer_holiday_core(d.index)
d["special_day"] = (
(d.holiday_pop_share >= PARTIAL_HOLIDAY_SHARE) | d.is_bridge_day | d.christmas_period | d.lockdown
)
return d


@dataclass(frozen=True)
class Baseline:
columns: tuple[str, ...]
coef: np.ndarray
n_fit_days: int

def design(self, d: pd.DataFrame) -> pd.DataFrame:
levels = pd.get_dummies(d.daytype + "_" + d.month.astype(str).str.zfill(2), dtype=float)
design = pd.concat([levels, d[["hdd", "hdd_lag1", "cdd", "radiation_wm2", "trend_years"]]], axis=1)
return design.reindex(columns=list(self.columns), fill_value=0.0)

def predict(self, d: pd.DataFrame) -> pd.Series:
return pd.Series(self.design(d).to_numpy() @ self.coef, index=d.index, name="expected_mw")

def named_coefficients(self) -> dict[str, float]:
return dict(zip(self.columns, self.coef.round(3).tolist(), strict=True))


def fit_days(d: pd.DataFrame) -> pd.Series:
return (d.split == "train") & ~d.special_day & d[["temp_c", "hdd_lag1", "radiation_wm2"]].notna().all(axis=1)


def fit_baseline(d: pd.DataFrame) -> Baseline:
mask = fit_days(d)
train = d[mask]
levels = sorted((train.daytype + "_" + train.month.astype(str).str.zfill(2)).unique())
columns = (*levels, "hdd", "hdd_lag1", "cdd", "radiation_wm2", "trend_years")
proto = Baseline(columns=columns, coef=np.zeros(len(columns)), n_fit_days=int(mask.sum()))
design = proto.design(train).to_numpy()
coef, *_ = np.linalg.lstsq(design, train.load_mw.to_numpy(), rcond=None)
return Baseline(columns=columns, coef=coef, n_fit_days=int(mask.sum()))
Loading