Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""Renewable Energy Systems datasets."""
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Hill of Towie SCADA Forecasting

`res/hill-of-towie-scada@2.0.0` covers all 21 turbines from January 2016 through April 2026.
It uses annual ten-minute SCADA archives from Zenodo release 2.1.0. Each task predicts six measured
active-power values in kW from the previous 36 ten-minute readings of active power, active wind
speed, and generator RPM. [Source release and CC BY 4.0 attribution](https://zenodo.org/records/22662930)

## Dataset split

All boundaries are UTC, with an inclusive start and exclusive end for physical measurement
intervals. Source timestamps label the **end** of each ten-minute interval, so a reading stamped
at a split boundary belongs to the preceding interval. An hourly task is eligible when its entire
six-hour input and one-hour target fit in one calendar month and one row of the table below. The
same boundaries apply to every turbine. An input needs all three channels, and a target needs
only power. Source gaps and blanks remain nulls in the shared records.

| TimeF split | `evaluation_cohort` | Start UTC | End UTC | Windows | Rationale |
| --- | --- | --- | --- | ---: | --- |
| Train | `historical_train` | 2016-01-01 00:00 | 2024-10-01 00:00 | 1,564,202 | Retains several seasons, the AeroUp rollout, and TuneUp. Ends after the documented late-summer 2024 curtailment period. |
| Validation | `post_tuneup_validation` | 2024-10-01 00:00 | 2025-01-01 00:00 | 45,511 | Tunes on later data following the TuneUp campaign, ahead of the 2025 holdout. |
| Test | `pre_commissioning` | 2025-01-01 00:00 | 2025-10-28 13:00 | 141,619 | Main score for forecasting before documented Dynamic Yaw commissioning. |
| Test | `dynamic_yaw_commissioning` | 2025-10-28 13:00 | 2026-01-07 13:00 | 34,857 | Covers commissioning, tests, and changes to turbine roles. |
| Test | `dynamic_yaw_west_trial` | 2026-01-07 13:00 | 2026-02-03 18:00 | 13,434 | Starts at the formal west trial boundary in the RES configuration. |
| Test | `dynamic_yaw_west_and_east_trial` | 2026-02-03 18:00 | 2026-05-01 00:00 | 42,614 | Starts at the formal east cohort boundary and ends with the archive. |

Use the main test cohort for the headline score and report the three later cohorts separately.
September 2025 lost 44.34% of expected turbine intervals, including a 12.73-day gap across every
turbine. Report a main-test sensitivity score with September omitted to show how that gap affects
coverage. The Dynamic Yaw schedule alternates 50 minutes on and 50 minutes off. The cohort labels
describe dates. Per-turbine exposure, controller state and wake effects require the RES role
definitions and controller telemetry. [Trial configuration](https://github.com/resgroup/hill-of-towie-open-source-analysis/blob/main/scripts/wfc_analysis_2026/wind_up_config/HOT_dynamic_yaw.yaml)
· [Commissioning and curtailment dates](https://github.com/resgroup/hill-of-towie-open-source-analysis/blob/main/scripts/wfc_analysis_2026/wind_up_config/exclusion_periods/HOT_DY.yaml)

## Measurement and operating changes

| When | What changed |
| --- | --- |
| August 2016 | The turbine SCADA schema removed the low-speed power reduction fields. |
| July 2017 | The schema added boost availability. |
| August 1, 2017 and April 10–11, 2018 | The secondary anemometer correction started on T14 and T17 first. Most other turbines followed on April 10, 2018, with T01 and T06 on April 11. The historical audit found a median +11.91 percentage-point step in raw secondary wind speed relative to primary. The active-to-primary ratio was unchanged to two decimal places. |
| September 23–29, 2021 | T13 underwent the first AeroUp retrofit. |
| April 2022 | The schema added actual wind direction, boost activation, and turbine operating state. The three forecast channels span the earlier and later schemas. |
| July 14, 2022–May 26, 2023 | RES rolled out AeroUp to the rest of the fleet. Use the per-turbine first and last work dates to distinguish pre-upgrade, installation, and post-upgrade windows. |
| March 14–August 31, 2024 | TuneUp began on T02, T03, T06, T08, T09, T13, T16, and T20 on March 14. T21 joined on May 2. RES also flags heavy curtailment from August 9 through September 28. |
| October 28, 2025 onward | Dynamic Yaw commissioning began. RES lists turbine role changes and controller configuration tests in October, November, and December. |
| January 7 and February 3, 2026 | The formal west and east trial boundaries were at 13:00 and 18:00 UTC, respectively. Dynamic Yaw changes the relationship between inflow, yaw, and measured power. |

RES records farm-wide pre-campaign tests from October 28 at 13:00 to November 17 at 13:45 UTC.
It records a deliberate north-offset test on T15 and T18 on November 19–20, a change in T06's
role on December 2, and changes in T19's and T21's roles on December 11. These dates are why the
main test stops in October 2025. [RES commissioning log](https://github.com/resgroup/hill-of-towie-open-source-analysis/blob/main/scripts/wfc_analysis_2026/wind_up_config/exclusion_periods/HOT_DY.yaml)

The dataset targets actual delivered power, including valid readings during curtailment and
changes in operation. The connector uses the active wind-speed channel throughout. Use the same
validation period to compare full-history training with a June 2023 onward training run, after
the AeroUp rollout. [SCADA data dictionary](https://zenodo.org/records/22662930/preview/Hill_of_Towie_data_dictionary.md?include_deleted=0)
· [AeroUp dates by turbine](https://zenodo.org/records/22662930/files/Hill_of_Towie_AeroUp_install_dates.csv?download=1)
· [RES campaign summary](https://zenodo.org/records/22662930)

Zenodo release 2.0.0 extended SCADA from August 2024 to April 2026 and added separate fastlog,
LiDAR, and controller telemetry. Release 2.1.0 updated the documentation. The forecasting
records here use the ten-minute SCADA tables. [Source changelog](https://zenodo.org/records/22662930/preview/CHANGELOG.md?include_deleted=0)

## Build and read the forecast windows

From the TimeNet checkout, run `make sync`, then build into a local registry:

```bash
uv run timenet-build build res/hill-of-towie-scada@2.0.0 --no-isolation
```

The source download is approximately 15 GB. The connector reads two CSV tables per month directly
from the pinned archives. In the resulting TimeF release, a record holds one turbine and one
month segment. Each forecast carries an input `scope` for the 36 historical readings and a
target interval for the six observed power values. Apply the input scope before passing values to
a model. Reading a whole shared record would include readings after that task's cutoff.

```python
from timenet import TimeNet
from timenet.types import TimeInterval

with TimeNet().open("res/hill-of-towie-scada@2.0.0") as reader:
task = next(reader.iter_tasks(split="train", batch_size=1))
assert isinstance(task.scope, TimeInterval)
assert task.targets is not None
(target,) = task.targets
assert isinstance(target, TimeInterval) and target.time_series
(power,) = target.time_series
context = {
signal.name: signal.slice(task.scope).to_numpy()
for signal in task.inputs[0].signals
}
observed_power_kw = power.slice(target).to_numpy()
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
"""Hill of Towie SCADA forecasting connector."""

from timenet_connectors.datasets.res.hill_of_towie_scada.connector import (
CONNECTOR as CONNECTOR,
HillOfTowieSCADAConnector as HillOfTowieSCADAConnector,
)
Original file line number Diff line number Diff line change
@@ -0,0 +1,200 @@
"""Convert the Hill of Towie SCADA release into TimeF power forecasting tasks."""

from collections.abc import Iterator
import csv
from dataclasses import dataclass
from datetime import UTC, datetime
from functools import partial
from pathlib import Path
from zipfile import BadZipFile, ZipFile

import numpy as np
from numpy.typing import NDArray
import pyarrow as pa
import pyarrow.csv as arrow_csv

from timenet.connectors import BaseConnector
from timenet.dataset import TimeFDataset
from timenet.errors import TimeNetConnectorError
from timenet.types import ForecastingTask, Split, unix_us
from timenet_connectors.datasets.res.hill_of_towie_scada.windows import (
LAST_MONTH_END,
STEP,
STEP_US,
Turbine,
WindowSeries,
forecasting_tasks,
month_periods,
prepare_windows,
)
from timenet_connectors.download import Artifact, download_files


_FILES_URL = "https://zenodo.org/records/22662930/files"
_METADATA = "Hill_of_Towie_turbine_metadata.csv"
_METADATA_SHA256 = "dd40789c2bf2268da9d77890cc48617266f3ed9d13c5af2be6d6535671f261bc"
_ARCHIVE_SHA256 = {
2016: "800bb3d399e6b79278758bbd95e234e364b062a9ff7643a7f9c045b0db9e9738",
2017: "a49c11def85623b7f0537988debfa499cca15fc6ec971256051597d91224ab1f",
2018: "4d59191bd12ad0e85eb4de44f2f8f7d8e2556452341e89cd194a816debc48761",
2019: "24a9b9bfdf32558d80c518a11954d7230254f5ba384671d3df6ba3e53deb5b26",
2020: "3e55de08be27e6a6e702a99bfcccd57e20abfa69e7c65d5901caf581f128f020",
2021: "2284741792b4bfc7ca08234f49440dad4820bac5e73098011745a3f69e295c19",
2022: "1b0f8657273ddbb8c4f84042a0bcb1c5eb1fb7833849b4c491f2b9f10604700d",
2023: "e65f7acd65664584e5b72e4ad7ccd4c0f0af4ad68f618b8cf0b4e5c1f4a69521",
2024: "b6bb540e3d2fcc2cff8756d96ef4155037b1cc41b284e6249b41054f4e50fbc3",
2025: "cd439de7bf8e297dd8422d5c93eda04785c56e3861be8d2a79f403ac39b873ee",
2026: "4bcfd706769369eacde59ea4f3a45bc3f9c0fb334c67fea75fcf3aa0c424eaea",
}
_POWER = ("wtc_ActPower_mean",)
_TELEMETRY = ("wtc_AcWindSp_mean", "wtc_GenRpm_mean")


@dataclass(frozen=True)
class HillOfTowieFiles:
"""Local annual SCADA archives and their turbine lookup."""

archives: tuple[Path, ...]
metadata: Path


def _read_turbines(path: Path) -> tuple[Turbine, ...]:
"""Read unique station identities and rated capacities."""
turbines: dict[int, Turbine] = {}
with path.open(encoding="utf-8-sig", newline="") as stream:
reader = csv.DictReader(stream)
for row in reader:
try:
turbine = Turbine(int(row["Station ID"]), row["Turbine Name"], int(row["Rated power (kW)"]))
except (KeyError, TypeError, ValueError) as exc:
raise TimeNetConnectorError(f"{path.name}:{reader.line_num}: invalid turbine metadata") from exc
if turbine.station in turbines or not turbine.name or turbine.rated_power_kw <= 0:
raise TimeNetConnectorError(
f"{path.name}:{reader.line_num}: invalid or duplicate station {turbine.station}"
)
turbines[turbine.station] = turbine
if not turbines:
raise TimeNetConnectorError(f"{path.name}: no turbine metadata")
return tuple(turbines[key] for key in sorted(turbines))


@dataclass(frozen=True)
class _Month:
start: datetime
end: datetime

@property
def steps(self) -> int:
return (self.end - self.start) // STEP


def _months(year: int) -> Iterator[_Month]:
"""Yield UTC months in an archive, respecting the release end in April 2026."""
for month in range(1, 13):
start = datetime(year, month, 1, tzinfo=UTC)
if start >= LAST_MONTH_END:
return
end = datetime(year + month // 12, month % 12 + 1, 1, tzinfo=UTC)
yield _Month(start, end)


def _read_table(
archive: ZipFile,
month: _Month,
table: str,
fields: tuple[str, ...],
stations: NDArray[np.int64],
) -> NDArray[np.float64]:
"""Load selected columns into a station-by-time array with source blanks preserved."""
member = f"{table}_{month.start:%Y_%m}.csv"
options = arrow_csv.ConvertOptions(
include_columns=["TimeStamp", "StationId", *fields],
column_types={"TimeStamp": pa.timestamp("us"), "StationId": pa.int64(), **dict.fromkeys(fields, pa.float64())},
)
try:
with archive.open(member) as raw:
data = arrow_csv.read_csv(raw, convert_options=options)
except (KeyError, pa.ArrowInvalid) as exc:
raise TimeNetConnectorError(f"{archive.filename}: unable to read {member}: {exc}") from exc

timestamps = data["TimeStamp"]
station_ids = data["StationId"]
if timestamps.null_count or station_ids.null_count:
raise TimeNetConnectorError(f"{member}: missing timestamp or station ID")
relative_us = timestamps.to_numpy().astype(np.int64) - unix_us(month.start)
# Source labels interval ends. A row at midnight belongs to the previous interval's month.
owned = (relative_us > 0) & (relative_us <= month.steps * STEP_US)
owned_us = relative_us[owned]
if np.any(owned_us % STEP_US):
raise TimeNetConnectorError(f"{member}: timestamp is outside the ten-minute grid")
ids = station_ids.to_numpy()[owned]
if unknown := np.setdiff1d(ids, stations).tolist():
raise TimeNetConnectorError(f"{member}: unknown station IDs {unknown}")
rows = np.searchsorted(stations, ids)
steps = owned_us // STEP_US - 1
if np.unique(rows * month.steps + steps).size != len(steps):
raise TimeNetConnectorError(f"{member}: duplicate measurement")
values = np.full((len(fields), stations.size, month.steps), np.nan)
for field, channel in zip(fields, values, strict=True):
measurements = data[field].to_numpy()[owned]
if np.any(np.isinf(measurements)):
raise TimeNetConnectorError(f"{member}: infinite measurement in {field}")
channel[rows, steps] = measurements
return values


def _month_series(
archive: ZipFile, month: _Month, turbines: tuple[Turbine, ...], stations: NDArray[np.int64]
) -> Iterator[WindowSeries]:
"""Prepare the turbine records and eligible cutoffs for each part of a month."""
power = _read_table(archive, month, "tblSCTurGrid", _POWER, stations)
telemetry = _read_table(archive, month, "tblSCTurbine", _TELEMETRY, stations)
values = np.concatenate((power, telemetry))
for period in month_periods(month.start, month.end):
start = (period.start - month.start) // STEP
stop = (period.end - month.start) // STEP
for index, turbine in enumerate(turbines):
yield prepare_windows(values[:, index, start:stop], turbine, period)


class HillOfTowieSCADAConnector(BaseConnector[HillOfTowieFiles]):
"""Build power forecasting tasks from the pinned 2016 to 2026 annual SCADA archives."""

# Override: the pinned source release has a fixed set of files.
# ruff: ignore[no-self-use]
async def download_async(self, cache_dir: Path) -> HillOfTowieFiles:
"""Cache the Zenodo archives and turbine lookup."""
pins = [(f"{year}.zip", digest) for year, digest in _ARCHIVE_SHA256.items()]
files = await download_files(
[
Artifact(f"{_FILES_URL}/{name}?download=1", cache_dir / name, sha256=digest)
for name, digest in [*pins, (_METADATA, _METADATA_SHA256)]
],
max_concurrency=5,
)
return HillOfTowieFiles(tuple(files[:-1]), files[-1])

def convert(self, raw: HillOfTowieFiles, /) -> TimeFDataset:
"""Register the SCADA records and stream tasks over qualifying windows."""
if len(raw.archives) != len(_ARCHIVE_SHA256):
raise TimeNetConnectorError("Hill of Towie requires all 11 annual archives and turbine metadata")
dataset = TimeFDataset(metadata=self.metadata())
series: list[WindowSeries] = []
try:
turbines = _read_turbines(raw.metadata)
stations = np.array([turbine.station for turbine in turbines], dtype=np.int64)
for year, path in zip(_ARCHIVE_SHA256, raw.archives, strict=True):
with ZipFile(path) as archive:
for month in _months(year):
for entry in _month_series(archive, month, turbines, stations):
dataset.add_record(record=entry.record)
series.append(entry)
except (BadZipFile, OSError, UnicodeError, csv.Error) as exc:
raise TimeNetConnectorError(f"Unable to read Hill of Towie source files: {exc}") from exc
if {entry.period.split for entry in series if entry.cutoffs} != set(Split):
raise TimeNetConnectorError("Hill of Towie SCADA must contain complete windows in all three splits")
dataset.set_task_stream([ForecastingTask], partial(forecasting_tasks, series))
return dataset


CONNECTOR = HillOfTowieSCADAConnector
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# yaml-language-server: $schema=https://docs.timenet.ai/schemas/dataset-card-v1.schema.json
yaml_schema_version: 1
dataset_id: res/hill-of-towie-scada
dataset_version: 2.0.0
name: Hill of Towie SCADA
description: >-
Ten-minute SCADA for 21 Hill of Towie wind turbines from January 2016 through April 2026.
Forecast six observed active-power readings from the preceding 36 readings of active power,
wind speed, and generator speed. Task scopes select history on shared turbine records, and
target spans select only future power. Hourly UTC cutoffs stay within a calendar month and
an evaluation cohort, and every required reading must be available. Training ends September
2024 and validation covers October through December 2024. The 2025 to April 2026 test split
distinguishes pre-commissioning, commissioning, and the two Dynamic Yaw trial periods.
license: CC-BY-4.0
access: open
source_url: https://zenodo.org/records/22662930
citation: >-
Clerc, A., and Lingkan, E. (2026). Hill of Towie wind farm open dataset (Version 2.1.0).
Zenodo. https://doi.org/10.5281/zenodo.22662930. Released by RES on behalf of TRIG.
domains:
- energy
tags:
- wind-energy
- scada
- forecasting
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""Tests for the Hill of Towie SCADA connector."""
Loading