Skip to content

Feature python compatibility - #4251

Open
benderl wants to merge 32 commits into
masterfrom
feature-python-compatibility
Open

benderl wants to merge 32 commits into
masterfrom
feature-python-compatibility

Conversation

@benderl

@benderl benderl commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

UI: openWB/openwb-ui-settings#1178

Summary by Sourcery

Provide a resilient, architecture-aware Python runtime for openWB across supported Raspberry Pi operating systems and Python versions.

New Features:

  • Add Python virtual-environment management with atomic upgrades, dependency validation, fallback handling, cleanup, and a runtime launcher.
  • Expose the running Python version and operating-system version through system configuration and MQTT state.
  • Support Python compatibility across Debian Bullseye/Trixie and Raspberry Pi architectures, including architecture-specific dependency installation and wheel validation.

Bug Fixes:

  • Prevent web access to Python virtual-environment directories.
  • Maintain compatibility with Python versions where the standard telnetlib module is unavailable or deprecated.
  • Improve Bullseye security repository handling and fail clearly when system package installation cannot be completed.

Enhancements:

  • Update services and diagnostic tooling to execute through the managed Python runtime.
  • Modernize and align Python dependency versions and package names.
  • Expand coverage for Python environment lifecycle, telnet behavior, JSON filters, RFID input, and architecture-specific setup.

Build:

  • Extend Python CI to test Bullseye/Python 3.9 and Trixie/Python 3.13 environments, including ARM dependency wheel availability.

CI:

  • Add manual workflow dispatch support and run tests through the managed virtual environment.

Deployment:

  • Install Python virtual-environment prerequisites during setup and coordinate service restarts after runtime changes.
  • Handle migration from legacy system-wide Python dependencies while preserving a verified fallback environment.

Documentation:

  • Document the installed Python runtime, virtual-environment maintenance, and recommended diagnostic script usage.

Tests:

  • Add comprehensive bootstrap tests for environment creation, validation, fallback, cleanup, locking, migration, activation, and architecture-specific behavior.
  • Add regression tests for telnet compatibility, JSON counter filters, and RFID reader events.

maselhirn-design

This comment was marked as spam.

@benderl
benderl force-pushed the feature-python-compatibility branch from ca4b3f6 to 3449924 Compare November 12, 2026 06:22
@benderl
benderl marked this pull request as ready for review November 12, 2026 10:45

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 7 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path=".github/workflows/github-actions-python.yml" line_range="13-14" />
<code_context>
+    strategy:
+      fail-fast: false
+      matrix:
+        raspberry-pi-os-release: ["Bullseye", "Bookworm", "Trixie"]
+        include:
+          - raspberry-pi-os-release: "Bullseye"
+            glibc: "2_31"
</code_context>
<issue_to_address>
**Bookworm compatibility jobs fail**

When the workflow runs its Bookworm matrix row, `matrix.include` has no Bookworm entry, so `actions/setup-python` receives an empty `matrix.python-version` and the ARM-wheel step gets empty `matrix.glibc` and `matrix.arm-wheel-python-version`; the job fails before compatibility checks.

Add a Bookworm entry to `matrix.include` with its `glibc`, `python-version`, and `arm-wheel-python-version` values.

Also at `.github/workflows/github-actions-python.yml:15-22`.
</issue_to_address>

### Comment 2
<location path="runs/run_python.sh" line_range="17" />
<code_context>
+				exec 8<"$venv/.openwb-lock"
+				flock -s 8
+				exec 9<&-
+				exec "$venv/bin/python" "$@"
+				;;
+		esac
</code_context>
<issue_to_address>
**Integration helpers miss Python requirements**

When a smart-home integration invokes a child script on a fresh installation, `run_python.sh` leaves `PATH` unchanged, so `Sbase.callpro` resolves `python3` to the system interpreter instead of the VEnv interpreter; that interpreter lacks the installed requirements, and integrations fail or return missing or stale results.

Prepend the VEnv's `bin` directory to `PATH` when launching the service.

Also at `runs/setup_python_venv.sh:112`, `openwb-install.sh:122`, `runs/atreboot.sh:320`.
</issue_to_address>

### Comment 3
<location path="runs/install_packages.sh" line_range="57-58" />
<code_context>
-sudo -u "$OPENWB_USER" pip install -r "${OPENWBBASEDIR}/requirements.txt"
+python_build_packages=()
+python_binary_packages=()
+case $(dpkg --print-architecture) in
+	arm64) python_binary_packages=(python3-evdev) ;;
+	arm*) ;;
+	*) python_build_packages=(gcc python3-dev) ;;
</code_context>
<issue_to_address>
**Older ARM64 installs cannot bootstrap**

When a fresh ARM64 installation uses Bullseye or Bookworm repositories that provide an evdev version older than 1.9.1, the package installers select the distribution's unversioned `python3-evdev` on every ARM64 system, but `setup_python_venv.sh` requires its installed metadata to be exactly `evdev-1.9.1.dist-info`. On ARM64 Bullseye or Bookworm systems whose repository package is older, setup exits and a fresh installation fails.

Restrict the Debian-package copy path to distributions that provide evdev 1.9.1, and use a compatible installation path for other ARM64 releases.

Also at `openwb-install.sh:124-125`, `runs/setup_python_venv.sh:100-105`.
</issue_to_address>

### Comment 4
<location path="runs/setup_python_venv.sh" line_range="103-107" />
<code_context>
+	arm64)
+		evdev_source=${OPENWB_EVDEV_SOURCE:-/usr/lib/python3/dist-packages}
+		site_packages=$("$new_env/bin/python" -c 'import sysconfig; print(sysconfig.get_path("purelib"))')
+		if [[ ! -d "$evdev_source/evdev" || ! -d "$evdev_source/evdev-1.9.1.dist-info" ]]; then
+			echo "Debian python3-evdev 1.9.1 is required on ARM64" >&2
+			exit 1
+		fi
+		cp -a "$evdev_source/evdev" "$evdev_source/evdev-1.9.1.dist-info" "$site_packages/"
+		;;
+	arm*) ;;
</code_context>
<issue_to_address>
**ARM64 evdev bootstrap rejects Debian metadata**

When the installed Debian `python3-evdev` package uses `.egg-info` metadata rather than `.dist-info`, `setup_python_venv.sh` requires and copies only `evdev-1.9.1.dist-info`; when Debian’s `python3-evdev` package provides its metadata as `evdev-1.9.1.egg-info`, the check exits before the venv is activated. A fresh ARM64 installation then has no usable Python environment.

Recognize and copy the metadata format actually supplied by Debian, or install evdev into the venv through a mechanism that preserves valid package metadata.
</issue_to_address>

### Comment 5
<location path="runs/run_python.sh" line_range="24" />
<code_context>
+	exec 9<&-
+fi
+if [[ -f "$base_dir/.venvs/.legacy" && ! -e "$base_dir/.venvs/.migrated" ]]; then
+	exec /usr/bin/python3 "$@"
+fi
+
</code_context>
<issue_to_address>
**Legacy fallback lacks new dependencies**

When an existing installation cannot build its new VEnv and has a satellite or dimm-kit integration configured, `run_python.sh` launches the old system interpreter without checking its packages; because the old global requirements did not include `telnetlib3`, importing the satellite or dimm-kit modules then raises `ModuleNotFoundError` and those configured integrations fail during a failed environment migration.

Ensure the legacy interpreter has the newly required dependencies before relying on it as a fallback, or make the fallback use a validated environment that includes them.
</issue_to_address>

### Comment 6
<location path="runs/setup_python_venv.sh" line_range="119-120" />
<code_context>
+printf '%s\n' "$fingerprint" > "$new_env/.openwb-ready"
+temporary_link="$venv_dir/.venv-link.$$"
+ln -s "$new_env" "$temporary_link"
+mv -Tf "$temporary_link" "$venv_link"
+new_env=
+touch "$venv_dir/.migrated"
+rm -f "$venv_dir/.legacy"
</code_context>
<issue_to_address>
**Activation can delete new environment**

When the setup process exits after `mv -Tf` activates the new environment but before `new_env=` runs, the activation rename succeeds, but `new_env` remains set until the next command; if the script exits in between, the EXIT trap removes that now-active directory and leaves `.venv` dangling, so the launcher cannot start Python.

Clear the cleanup ownership before activation or make the EXIT trap preserve the environment whenever `.venv` points to it.
</issue_to_address>

### Comment 7
<location path="runs/tests/test_python_venv.sh" line_range="43" />
<code_context>
+if [[ "${FAIL_INSTALL:-0}" == 1 && "$*" == '-m pip install '* ]]; then
+	exit 1
+fi
+exit 0
+PYTHON
+	chmod +x "$3/bin/python"
</code_context>
<issue_to_address>
**Broken dependencies pass validation**

When the environment's dependency check or import probe would fail, `test_python_venv.sh`'s fake interpreter reaches an unconditional `exit 0` at line 43 for `pip check` and import-probe calls, so bootstrap validation appears successful and the ready-environment assertions pass even when those checks would fail.

Make the fake interpreter record and independently control the results of `pip check` and the import probe, then assert failed validation prevents activation.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread runs/run_python.sh Resolved
Comment thread runs/install_packages.sh Resolved
Comment thread runs/setup_python_venv.sh Resolved
Comment thread runs/run_python.sh Resolved
Comment thread runs/setup_python_venv.sh Resolved
Comment thread runs/tests/test_python_venv.sh Resolved
@benderl

benderl commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

@sourcery-ai review

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 5 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="runs/setup_python_venv.sh" line_range="123-131" />
<code_context>
+	arm*) ;;
+	*) binary_options+=(--no-binary=evdev) ;;
+esac
+"$new_env/bin/python" -m pip install --no-user "${binary_options[@]}" --extra-index-url https://www.piwheels.org/simple -r "$base_dir/requirements.txt"
+"$new_env/bin/python" -m pip check
+"$new_env/bin/python" -c 'import aiohttp, evdev, grpc, jq, lxml, paho.mqtt.client, pymodbus, telnetlib3'
+touch "$new_env/.openwb-lock"
+printf '%s\n' "$fingerprint" > "$new_env/.openwb-ready"
+temporary_link="$venv_dir/.venv-link.$$"
+ln -s "$new_env" "$temporary_link"
+mv -Tf "$temporary_link" "$venv_link"
+new_env=
+touch "$venv_dir/.migrated"
+rm -f "$venv_dir/.legacy"
</code_context>
<issue_to_address>
**Stale dependencies remain active**

When an application update changes its dependency requirements and the new environment build fails or cannot reach package indexes, the failed install leaves the previous `.venv` symlink in place, and `run_python.sh` launches that environment without checking it against the current requirements. After an update that needs a changed or newly added dependency, the updated services run with stale packages and can fail at startup or when that dependency is used.

Before launching updated code with an old environment, verify that it satisfies the current requirements or prevent startup until a compatible environment is available.
</issue_to_address>

### Comment 2
<location path="runs/setup_python_venv.sh" line_range="103-115" />
<code_context>
+"$python" -m venv "$new_env"
+binary_options=(--only-binary=:all:)
+case $(dpkg --print-architecture) in
+	arm64)
+		evdev_source=${OPENWB_EVDEV_SOURCE:-/usr/lib/python3/dist-packages}
+		site_packages=$("$new_env/bin/python" -c 'import sysconfig; print(sysconfig.get_path("purelib"))')
+		evdev_metadata=
+		for metadata_suffix in dist-info egg-info; do
+			candidate="$evdev_source/evdev-1.9.1.$metadata_suffix"
+			if [[ -d "$candidate" ]]; then
+				evdev_metadata=$candidate
+				break
+			fi
+		done
+		if [[ ! -d "$evdev_source/evdev" || -z "$evdev_metadata" ]]; then
+			echo "Debian python3-evdev 1.9.1 is required on ARM64" >&2
+			exit 1
+		fi
</code_context>
<issue_to_address>
**Older ARM64 installs fail**

When an ARM64 installation uses a Debian release whose `python3-evdev` package is older than 1.9.1, the architecture-only package selection installs the OS-provided `python3-evdev`, then the bootstrap requires its metadata to report exactly 1.9.1. On older ARM64 Debian releases where that package is older, setup exits before activating the environment, so Python services cannot start after migration.

Select a compatible evdev source by OS release, or use a matching ARM64 wheel when the system package does not meet the pin.

Also at `runs/install_packages.sh:58`.
</issue_to_address>

### Comment 3
<location path="runs/install_packages.sh" line_range="34" />
<code_context>
+			exit 1
+		fi
+		if ! cmp -s "$source_file" "$temporary_source"; then
+			if ! sudo cp -p -- "$source_file" "$source_file.openwb-backup" || ! sudo cp -- "$temporary_source" "$source_file"; then
+				rm -f "$temporary_source"
+				exit 1
</code_context>
<issue_to_address>
**Interrupted source rewrite breaks APT**

When the process or system stops while the Bullseye security source file is being overwritten, `sudo cp` overwrites the active source file in place, so an interruption can leave it truncated despite the backup copy. Later runs treat the truncated file as the source of truth, and Bullseye security updates can remain disabled.

Write the replacement to a temporary file in the source file's directory and atomically rename it over the active file after the copy succeeds.
</issue_to_address>

### Comment 4
<location path=".github/workflows/github-actions-python.yml" line_range="13" />
<code_context>
+    strategy:
+      fail-fast: false
+      matrix:
+        raspberry-pi-os-release: ["Bullseye", "Trixie"]
+        include:
+          - raspberry-pi-os-release: "Bullseye"
</code_context>
<issue_to_address>
**Bookworm CI coverage is missing**

When the documentation lists Bookworm/Python 3.11 as configured, the workflow matrix contains only Bullseye and Trixie, so no job checks Bookworm/Python 3.11 and readers see a CI coverage claim for a job that never runs.

Add Bookworm to the workflow matrix or remove the Bookworm row from the documentation.

Also at `docs/Python-Kompatibilitaet.md:22-23`.
</issue_to_address>

### Comment 5
<location path="runs/setup_python_venv.sh" line_range="38" />
<code_context>
+	for candidate in "$venv_dir"/env.*; do
+		[[ -d "$candidate" && ! -L "$candidate" && "$candidate" != "$active_env" ]] || continue
+		[[ ${candidate##*/} =~ ^env\.[a-zA-Z0-9]{8}$ ]] || continue
+		[[ -f "$candidate/.openwb-ready" && -f "$candidate/.openwb-lock" && ! -e "$candidate/.openwb-unprotected" ]] || continue
+		if ! exec 8<"$candidate/.openwb-lock"; then
+			echo "Cannot lock Python environment $candidate" >&2
</code_context>
<issue_to_address>
**Interrupted builds leak disk space**

When power is lost or the setup process receives `SIGKILL` after creating an environment directory but before writing its readiness and lock files, `remove_unused_envs` skips environments without both readiness and lock files, so a power loss or `SIGKILL` during environment creation leaves the incomplete directory behind. Repeated interrupted builds accumulate abandoned files under `.venvs` and consume disk space.

During boot cleanup, remove inactive, name-matching environment directories that lack readiness metadata, while holding the setup lock so no build is in progress.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread runs/setup_python_venv.sh Resolved
Comment thread runs/setup_python_venv.sh Resolved
Comment thread runs/install_packages.sh Resolved
Comment thread runs/setup_python_venv.sh Resolved
@benderl
benderl requested a review from LKuemmel November 12, 2026 13:24
@benderl
benderl force-pushed the feature-python-compatibility branch from d1627b1 to a75b72b Compare November 17, 2026 08:01
@benderl benderl added this to the 2.4.0 milestone Nov 17, 2026
@benderl benderl added the ui depends on changes in ui repository label Nov 17, 2026
@benderl

benderl commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@sourcery-ai review

Comment thread docs/Python-Kompatibilitaet.md Resolved

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 5 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="runs/run_python.sh" line_range="20-21" />
<code_context>
+		case "$venv" in
+			"$base_dir"/.venvs/env.*)
+				if [[ $(<"$venv/.openwb-ready") != "$expected_fingerprint" ]]; then
+					echo "The active Python environment does not match the current requirements; refusing to start" >&2
+					exit 1
+				fi
+				exec 8<"$venv/.openwb-lock"
</code_context>
<issue_to_address>
**Services reject the retained environment**

When a requirements change is deployed while dependency installation cannot complete, such as during a boot without network access, the requirements fingerprint no longer matches the active environment, so `run_python.sh` exits instead of launching the service. `setup_python_venv.sh` retains that environment when it cannot rebuild offline, leaving the Python services unable to start until installation succeeds.

Provide a deliberate fallback for an existing usable environment when rebuilding fails, or ensure the application update is not activated until its matching environment is ready.
</issue_to_address>

### Comment 2
<location path="runs/run_python.sh" line_range="15" />
<code_context>
+		sha256sum "$base_dir/requirements.txt"
+		"$base_python" -c 'import os, sys; print(os.path.realpath(sys.executable), sys.version)'
+	} | sha256sum | cut -d ' ' -f 1)
+	if [[ -L "$venv" && -f "$venv/.openwb-ready" && -f "$venv/.openwb-lock" && -x "$venv/bin/python" ]]; then
+		venv=$(readlink -f "$venv")
+		case "$venv" in
</code_context>
<issue_to_address>
**Corrupt environment still launches**

When the active environment's dependencies become incomplete or inconsistent while its requirements and interpreter fingerprint remain unchanged, and setup cannot build a replacement, `run_python.sh` accepts an environment based on its ready marker, fingerprint, and executable, without repeating the `pip check` and import probes used by setup. If those setup checks fail and rebuilding the environment also fails, the retained environment still passes the launcher's checks and services start with broken dependencies.

Have the launcher validate dependency integrity before launching, or invalidate the ready marker when setup detects a failed validation and cannot replace the environment.
</issue_to_address>

### Comment 3
<location path="runs/atreboot.sh" line_range="364" />
<code_context>
+	# update os version
+	if [[ -f "/etc/os-release" ]]; then
+		. /etc/os-release
+		sudo mosquitto_pub -p 1886 -t "openWB/system/os_version" -r -m "$VERSION"
+	fi
+
</code_context>
<issue_to_address>
**Authenticated clients miss system topics**

When user management is enabled on an installation with an existing dynamic-security configuration and stored ACL role version 14, `setup_mosquitto.sh` leaves the existing configuration unchanged, and `update_acls()` skips syncing roles because the template still has version 14. The new topic ACLs are not applied, so authenticated UI clients cannot receive the system topics.

Bump the ACL template version and apply the new topic ACLs to existing dynamic-security configurations.

Also at `packages/helpermodules/update_config.py:787`.
</issue_to_address>

### Comment 4
<location path="runs/tests/test_python_venv.sh" line_range="382-384" />
<code_context>
+deb http://security.debian.org/debian-security bullseye-security main
+EOF
+cp "$workdir/apt/sources.list.d/security.sources" "$workdir/apt-rename-failure/sources.list.d/security.sources"
+if PATH="$workdir/apt-bin:$PATH" OPENWB_APT_DIR="$workdir/apt-rename-failure" FAIL_SOURCE_RENAME=1 \
+	bash "$script_dir/install_packages.sh" > "$workdir/apt-rename-failure.log" 2>&1; then
+	exit 1
+fi
+grep -q '^deb http://security.debian.org/debian-security bullseye-security main$' "$workdir/apt-rename-failure/sources.list"
</code_context>
<issue_to_address>
**Python bootstrap CI fails on Ubuntu**

When the test runs in the configured Ubuntu 24.04 environment, `test_python_venv.sh` expects the command at lines 382–384 to fail, but `install_packages.sh` only reaches the injected `mv` failure on Bullseye. On the Ubuntu runner it skips that rewrite and can succeed, causing the test to exit nonzero with correct code; the Bullseye assertions at line 366 are skipped too.

Run this check in a Bullseye environment or make the operating-system metadata injectable so the test reliably exercises the Bullseye rewrite.

Also at `runs/tests/test_python_venv.sh:366`.
</issue_to_address>

### Comment 5
<location path="packages/helpermodules/update_config.py" line_range="787" />
<code_context>
         ("openWB/system/mqtt/valid_partner_ids", []),
         ("openWB/system/mac_address", "unknown"),
         ("openWB/system/pnp_ip", {"address": "192.168.193.250", "prefix": 24}),
+        ("openWB/system/python_runtime_version", f"{sys.version.split()[0]}"),
         ("openWB/system/release_train", "master"),
         ("openWB/system/secondary_auto_update", True),
</code_context>
<issue_to_address>
**Runtime version stays outdated**

When the Python runtime changes after `openWB/system/python_runtime_version` has been published, `UpdateConfig.__pub_missing_defaults` only publishes missing topics, so it leaves the retained value unchanged and the UI and other consumers report the old runtime version.

Refresh `openWB/system/python_runtime_version` when publishing configuration instead of publishing it only when the topic is missing.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread runs/run_python.sh Resolved
Comment thread runs/run_python.sh Resolved
Comment thread runs/atreboot.sh Resolved
Comment thread runs/tests/test_python_venv.sh Resolved
Comment thread packages/helpermodules/update_config.py Resolved
@benderl
benderl requested a review from LKuemmel November 17, 2026 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ui depends on changes in ui repository

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants