[william-gr]

Why

In HA mode the action gives node B its own machine-id and hostid, then reboots it. Afterwards it read B's IDs over SSH once, with no retry. That raced with B's boot: in webui run 39251437564 the SSH connection was refused 72 s after the reboot, and the HA job failed before any test ran.

The old wait proved neither that B had gone down nor that SSH was back:

• It waited for SSH to stop answering, then for the API. SSH can stop early in the shutdown while the API on the old boot still answers.
• On the new boot, the API can answer before sshd is up.

Change

unique() in ha.sh reads B's /proc/sys/kernel/random/boot_id before the reboot. It then polls over SSH every 5 s, for up to 10 minutes by the clock (SECONDS), until B answers with a different boot id. A refused connection (sshd not started yet) or a timed-out one (ConnectTimeout=10) is simply tried again. Only then does it wait for the API and read the IDs. If B doesn't come back, it fails with a clear message instead of an SSH error.

shellcheck -x appliance.sh ha.sh lib.sh (the CI command) passes. The loop was also run locally against a fake ssh_: refused twice, then the old boot, then a new one → it waits and returns; refused throughout → it fails with the message above once the deadline passes.

[github-actions]

LOW — "within 10 minutes" counts only the sleep 5. Each attempt also runs ssh_, and ssh_ sets ConnectTimeout=10. While B is down, an attempt costs anywhere from about 3 s (no ARP reply, so EHOSTUNREACH) to the full 10 s. So 120 attempts take roughly 16 to 30 minutes, not 10. The loop does still terminate, so it can't hang. But the message understates how long the job waited, and it is easy to misread next to a step timeout. You could bound the loop by wall-clock time instead, e.g. deadline=$((SECONDS + 600)) and while [ "$SECONDS" -lt "$deadline" ]. Or the message could just say "after 120 tries".

[github-actions]

One finding: one LOW. No human review needed.

The change replaces the old "ssh stopped answering" wait with a check on boot_id, which is the right test: B is only back once it answers ssh from a boot that differs from the one before the reboot. Several things hold up:

• local boot now deadline is declared on its own line, so a failure reading the first boot_id still trips set -e.

• The && chain in the loop doesn't trigger errexit.

• now is always assigned, because the loop runs at least once.

• The check after the loop reads the same condition the loop broke on, so a success on the last poll isn't reported as a failure.

• unique is called as a plain statement (ha.sh:313), so errexit is in effect.

• LOW — ha.sh:246: the 10-minute deadline is only checked between polls. ssh_ has no ServerAliveInterval, so one poll that connects just before the guest resets can hang on a dead TCP connection past the deadline. The window is narrow (inline comment).

Callers see no change in inputs, outputs or documented behaviour, so README.md needs no update. The failing check-ticket check is about the PR title or body, not this code.

[github-actions]

Approved. 1 finding(s): 1 LOW.

Nothing blocks and no human review is needed.

[github-actions]

LOW — The 10-minute deadline is only checked between polls, so it doesn't cap a single poll. ConnectTimeout=10 only covers connection setup. ssh_ sets no ServerAliveInterval/ServerAliveCountMax, so if a poll connects just before B's guest resets (the old boot's sshd is still up during shutdown), the reset sends no RST. That ssh call can then block on a dead TCP connection far past the deadline, and the die never fires. The window is small because cat returns at once, so this is unlikely. If you want "by the clock" to actually hold, add -o ServerAliveInterval=5 -o ServerAliveCountMax=2 here (or in ssh_), or wrap the poll in timeout 20.

[github-actions]

Approved. 1 finding(s): 1 LOW.

Nothing blocks and no human review is needed.

[bugclerk]

Jira URL: https://ixsystems.atlassian.net/browse/NAS-144733

[bugclerk]

This PR has been merged and conversations have been locked.
If you would like to discuss more about this issue please use our forums or raise a Jira ticket.