Skip to content

Shrink source routing table to 0 when source routing is disabled - #778

Draft
puddly wants to merge 3 commits into
devfrom
puddly/source-routing-fixes2
Draft

puddly wants to merge 3 commits into
devfrom
puddly/source-routing-fixes2

Conversation

@puddly

@puddly puddly commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Fixes #772.

Source routing in EmberZNet is a little broken/odd and is kind of already half-enabled by default:

  • even when it is not enabled, other EmberZNet devices on the network send Route Status messages until they hear a a MTORR go out with the high-RAM bit set.
  • even when it is not enabled, the coordinator will still populate the source route table with incoming route status messages.
  • the coordinator will still always prefer these sources routes even if a fresher AODV route is known. It doesn't matter if source routing is disabled or enabled. More importantly, they are still preferred even if the source route is broken.
  • there is no mechanism for evicting source routes, even if they are broken. The stack doesn't appear to react to SOURCE_ROUTE_FAILURE properly.

The end result is as the above issue described, bad source routes are used and there's no way to evict them. The only way I can find to do so is to completely disable source routing if it is not enabled by setting the table size to 0 on startup. zbee_nwk.src_route == True in Wireshark finds no source routing packets sent at all.

AODV route discovery seems to help in the case where a source route is unknown and the stack hasn't performed a MTORR to discover a source route to the destination. With our (now) short MTORR intervals this should be less of a problem but it should still help.


Since this is a bugfix PR for source routing, I've rolled in #716.

Should be released along with zigpy/zigpy#1982, as AODV-only routing is less performant 1.

Footnotes

  1. I actually see less popcorning with source routing fully disabled than I did in the "hybrid" mode EmberZNet uses by default, for some reason! Routing in this mode feels more stable and stutters less, somehow. ↩

Copilot AI balanced review requested due to automatic review settings November 19, 2026 22:04
@codecov

codecov Bot commented Nov 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.90909% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 99.59%. Comparing base (1374aae) to head (a74409f).

Files with missing lines Patch % Lines
bellows/zigbee/application.py 90.90% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##              dev     #778   +/-   ##
=======================================
  Coverage   99.59%   99.59%           
=======================================
  Files          64       64           
  Lines        4417     4423    +6     
=======================================
+ Hits         4399     4405    +6     
  Misses         18       18           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified for the source-route configuration and discovery changes.

No open issues 🎉

What changed in this PR

This PR addresses stale source routes by defaulting the NCP source-route table to zero when source routing is disabled, while preserving explicit table-size overrides.

Changes:

  • Apply the adjusted table configuration during connection and reset.
  • Schedule immediate source-route discovery when enabled and honor forced route discovery regardless of source-routing mode.
  • Add configuration and packet-option tests, and update the discovery-mode assertion.
File Description
tests/​test_ezsp.py Expect RESCHEDULE when enabling source routing.
tests/​test_application.py Cover table-size defaults, explicit overrides, and forced discovery with source routing.
bellows/​zigbee/​application.py Adjust table configuration and honor forced route discovery.
bellows/​ezsp/​__init__.py Request immediate discovery with RESCHEDULE.

Review effort: Balanced

@zigpy-review-bot zigpy-review-bot left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix works. Edit: not for joins through routers: an empty table breaks them, see my reply below. The rest still holds for how the NCP handles the table size. The NCP's own EZSP handler accepts a source route table size of 0, and every table function handles 0, on all four SDK lines I checked: Gecko SDK 4.4.4 and 4.5.1 (EmberZNet 7.4/7.5, what most installs and the #772 reporter run) and Simplicity SDK 2025.12.3 and 2026.6.1. An independent A/B test in Koenkk/zigbee2mqtt#32910 on EmberZNet 8.0.2 shows the same effect: size 0 is accepted at runtime and route records are no longer stored. Three requests and two questions below.

Requests

  1. Pin source_routing in the test fixtures. APP_CONFIG in tests/test_application.py doesn't set zigpy.config.CONF_SOURCE_ROUTING, so the tests follow zigpy's default. Against zigpy/zigpy#1982, which this PR is meant to ship with, 15 send tests fail because APS_OPTION_ENABLE_ADDRESS_DISCOVERY is now set on the sent frames, which those tests don't expect. Adding zigpy.config.CONF_SOURCE_ROUTING: False to APP_CONFIG makes all 771 tests pass against both the released zigpy and zigpy/zigpy#1982. The _reset() path with the new config has no test yet either; Codecov reports that line.

  2. Raise the zigpy floor to the release that has zigpy/zigpy#1982. The floor is zigpy>=2.4.0; with today's zigpy, this PR would put every default user on AODV-only routing.

  3. Say how #772 is covered once source routing is on by default. The empty table only applies with source routing off, and after zigpy/zigpy#1982 that is opt-out. With source routing on, recovery depends on the stack, and none of it is guaranteed:

    • After 3 SOURCE_ROUTE_FAILURE errors, the concentrator queues an MTORR. Once that MTORR has gone out, it broadcasts an IEEE_addr_req for the broken node. A parent answering for its child rewrites the child's entry, but only if the parent has a table entry itself.
    • Per Zigbee R23.2 §3.6.4.5.5, a parent sends a route record on behalf of an end-device child toward a many-to-one concentrator. It MAY skip that for children that already received source-routed frames from that concentrator. That could explain the missing route record in #772, though the issue doesn't show it.

    One sentence in the description, or a narrower closing reference, would keep the claim accurate.

Questions

  • Z2M was offered this setting and declined it. Koenkk/zigbee-herdsman#1864 proposed exposing it and was closed. Among the ember adapter maintainer's reasons: it is a side effect rather than disabling concentrator mode, it may change with any EmberZNet release, other routers may source-route anyway, and the firmware build validation won't accept it (Koenkk/zigbee2mqtt#32910). Disabling concentrator mode alone doesn't stop the NCP from using stored routes, though, as the reporter's test there and the stack libraries both show. The build check is most likely the SDK configurator's 2–255 range for SL_ZIGBEE_SOURCE_ROUTE_TABLE_SIZE when the source route library is linked in. At runtime the NCP accepts 0, every table function handles it, and 0 is the SDK default without that library, so I found no bug behind it. It is still a configuration Silicon Labs doesn't ship; worth a line in the description or code comment.
  • Does AODV-only routing hold up after source routing has been on? Per your first caveat in home-assistant/core#142136 (see my review of zigpy/zigpy#1982), many routers stop answering normal route requests once source routing has been on, until they are power-cycled. With this PR, source routing off means the coordinator depends entirely on those route requests, and after zigpy/zigpy#1982 the users who turn it off will mostly be ones who ran with it on first. AN1233 also says a trust center must act as a concentrator and record source routes, or joins and rejoins across multiple hops may fail (quoted in zigbee-herdsman's ember adapter). Your footnote says this mode feels more stable on your network. Had that network run as a concentrator before, and does it include joins and rejoins several hops away?

Optional

  • The new comment says the NCP "never drops a stale route", and the description that "there is no mechanism for evicting source routes". Both are overstated. An entry is replaced by a newer route record for the same destination. It is rewritten by an APS Update-Device or a parent's IEEE_addr_rsp for its child, when that parent has an entry. It is recycled when the table is full, and deleted when the device leaves. What does hold: there is no ageing, and nothing invalidates an entry on SOURCE_ROUTE_FAILURE.
  • RESCHEDULE matches what Z2M does, but changes little here: it schedules an MTORR within minTime (10 s) unless an earlier one is pending, which setConcentrator(on=True) already does. #716 can be closed as superseded, and #717 edits the same call and will need a rebase.
  • The APS_OPTION_ENABLE_ADDRESS_DISCOVERY TODO can be resolved. The stack reads the bit in one place only: when an APS-retried message fails after being sent through the address table or a binding, it forgets the destination's short ID. bellows always sends direct, so the bit does nothing here. Either drop it or keep it for parity with Z2M, with a comment saying so.
Verified (9 checks)
  • Size 0 is accepted: the NCP-side config handler for 0x1A stores any value up to the current table size and rejects anything larger. Checked in the Cortex-M33 builds of Gecko SDK 4.4.4 and 4.5.1 and of Simplicity SDK 2026.6.1. Setting it is allowed before the network comes up, which is when bellows writes its config.
  • Size 0 is safe: the find, add, add-with-closer-next-hop and incoming-route-record functions all return early at size 0. Every other loop is bounded by the entry count, and nothing divides by the size. Same on all four SDK lines.
  • Toggling works: an NCP reset restores the compiled-in size, and bellows resets on connect, so turning source routing back on restores the full table.
  • Override handling: an explicit CONF_EZSP_CONFIG value wins over the empty table. User values reach write_config as RuntimeConfig(minimum=False), so the 0 isn't skipped by the grow-only check, on both connect() and _reset().
  • Stored routes win: a stored source route is appended to every unicast regardless of the route-discovery bits. ENABLE_ROUTE_DISCOVERY is skipped when one is stored, and FORCE_ROUTE_DISCOVERY still starts a discovery, but the frame goes out source-routed anyway. That matches the #772 log, where FORCE_ROUTE_DISCOVERY retries still got SOURCE_ROUTE_FAILURE, and confirms the new comment on the force branch.
  • No concentrator needed: route records are stored even when the coordinator isn't a concentrator. With source routing off, nothing acts on SOURCE_ROUTE_FAILURE: the discovery event exits immediately.
  • XNCP manual routes: the NabuCasa manual route table is separate from the stack's and is consulted first, so manual routes keep working with a table size of 0.
  • Tests: bellows 771 passed on this head. Against zigpy/zigpy#1982, 15 failed and 756 passed; 771 passed with CONF_SOURCE_ROUTING: False pinned in APP_CONFIG.
  • Z2M comparison (zigbee-herdsman 80470887): always a concentrator, high-RAM by default, 5/60 s, RESCHEDULE after setConcentrator. Default APS options include ENABLE_ADDRESS_DISCOVERY. No table-size config. Route errors and route records are only logged, and route discovery is never forced.

@puddly
puddly marked this pull request as draft November 20, 2026 05:25
@puddly

puddly commented Nov 20, 2026

Copy link
Copy Markdown
Member Author

Shrinking the table to 0 entries works great at runtime but unfortunately suffers from a fatal flaw: joins stop working.

I think the only way to do this is with an XNCP command and a small check in nc_zigbee_override_append_source_route, unfortunately.

@zigpy-review-bot

Copy link
Copy Markdown
Collaborator

That fits what the stack libraries show, and it is a gap in my review: "The fix works" only covered how the NCP handles the table size, not joins. Why joins break at size 0, from the prebuilt Simplicity SDK 2026.6.1 and 2026.6.2 libraries:

  • The trust center's own APS commands (Transport Key, Tunnel, Confirm Key, Remove Device) go out with NWK route discovery suppressed: sli_zigbee_send_aps_command builds the header with frame control 0x0008 or 0x0208. Without a neighbour, an active AODV route or a stored source route, sli_zigbee_network_send_unicast gives up with no route.
  • For a joiner behind a router, the NCP's sli_zigbee_af_trust_center_pre_join_callback stores a route to the joiner through its parent with sli_zigbee_source_route_add_entry_with_closer_next_hop. At size 0 nothing is stored, so the trust center's replies in the Zigbee 3.0 link key exchange (Transport Key and Confirm Key, addressed to the joiner itself rather than tunnelled through its parent) normally have no route, and the join doesn't complete. The network key itself is sent via the parent, which only works if the parent is a neighbour or the trust center happens to have an AODV route to it.
  • Joins directly to the coordinator don't depend on it, and a secured rejoin normally gets no action from the trust center, so nothing is sent.

For the XNCP version, the same frames go through nc_zigbee_override_append_source_route: sli_zigbee_make_zigbee_header calls the hook for every unicast NWK data frame, including those APS commands. So a check that simply stops appending stored routes would break joins the same way. What should work is to skip stored routes only for frames whose NWK discover-route bits are set (EmberZNet uses 0x40 to enable and 0x80 to force discovery, and bellows always sets one of them) and leave the discovery-suppressed frames alone. Two limits:

  • The hook also sees buffers that aren't frames: a NULL header (Don't dereference a NULL header in the XNCP source route override NabuCasa/silabs-firmware-builder#269 adds that guard) and the probe buffers from doesSourceRouteExist and storeApplicationKeyRequest. It can't tell those apart by content, so reading the bits from *header is only safe on the sli_zigbee_make_zigbee_header path. Wrapping that function to record its frameControl argument for the duration of the call (the firmware already wraps sli_zigbee_am_multicast_member with the linker's --wrap) would give the hook the bits without reading the buffer.
  • A stored entry with zero relays is out of the hook's reach: nothing is appended for it either way, and when there is no AODV route, sli_zigbee_network_send_unicast reads the table itself and sends such frames direct, without discovery unless it is forced.

Deleting the stored route after a SOURCE_ROUTE_FAILURE (sli_zigbee_source_route_delete_entry) would complement it for the stack's own frames. I haven't tried any of this on hardware; an end device joining through a router one hop and several hops away, plus a trust center rejoin, would be the cases to test.

@kimdelta

Copy link
Copy Markdown

Field report: setting the NCP source route table size to 0 caused widespread delivery failures on my network

I did NOT run this PR branch. On bellows 1.2.1 I only reproduced what I understand to be its core idea, by setting

zha:
  zigpy_config:
    ezsp_config:
      CONFIG_SOURCE_ROUTE_TABLE_SIZE: 0

(source_routing left at default/off). Sharing the numbers in case they are useful before this lands.

Setup

  • Home Assistant 2026.11.1 (Container), ZHA, bellows 1.2.1, zigpy 2.4.1, zha-quirks 2.4.1
  • Coordinator: Sonoff ZBDongle-E (EFR32MG21, EmberZNet)
  • Roughly 80 devices: many Philips Hue routers (bulbs, LOM007 plugs), ~17 Aqara E1 roller blinds (lumi.curtain.acn002), Danfoss Ally TRVs, Aqara/Hue sensors

Why I tried it
I see exactly the problem from #772: after an Aqara blind does a secured rejoin, the coordinator keeps using the old source route. In one night I logged 17,311 ROUTE_ERROR_SOURCE_ROUTE_FAILURE for a single blind over ~9 hours, while the blind kept retrying its Active_EP_req to the coordinator (APS retries, no ACK received). Commands to it failed with "device did not respond" until HA/NCP was restarted.

Result with table size 0 (messageSentHandler status counts, from bellows debug logs)

  • Normal, ~18 h log: 17,892 SUCCESS / 83 DELIVERY_FAILED (<1 %)
  • Normal, ~10 h log: 9,668 SUCCESS / 68 DELIVERY_FAILED (<1 %)
  • Table size 0, ~1 h log: 1,436 SUCCESS / 726 DELIVERY_FAILED (~33 %)

With size 0 the failures were spread over many different devices (routers and end devices), several blinds and a Hue plug stopped responding, and things returned to normal immediately after removing the setting and restarting.

So on this network, disabling the source route table entirely was much worse than the stale-route problem it was meant to fix. Happy to provide more detail if useful.

@puddly

puddly commented Nov 20, 2026

Copy link
Copy Markdown
Member Author

@kimdelta Did you happen to record any of this traffic in a PCAP? I would be interested to know the failure mode for DELIVERY_FAILED and to correlate it with zigpy/bellows logs.

@kimdelta

Copy link
Copy Markdown

@puddly No PCAP unfortunately - I don't have a sniffer set up. Attached are the debug logs (bellows, zigpy, zha, zhaquirks lines only). Timestamps are UTC.

  1. bellows-tablesize0-2026-11-20.log.gz - ~1 hour (15:23-16:24 UTC) running with CONFIG_SOURCE_ROUTE_TABLE_SIZE: 0. This is where the 726 DELIVERY_FAILED came from.

  2. bellows-stale-route-night-2026-11-19.log.gz - normal settings, 2026-11-19 21:29 to 2026-11-20 07:08 UTC. Aqara blind 54:ef:44:10:01:27:c5:31 (nwk 0x1A3E) does a secured rejoin at 22:22:32, after which every unicast to it gets ROUTE_ERROR_SOURCE_ROUTE_FAILURE until the restart at 07:08. The last route record from it before the rejoin was [0x698B, 0x1750].

bellows-stale-route-night-2026-11-19.log.gz
bellows-tablesize0-2026-11-20.log.gz

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix This PR fixes a bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stale NCP source route is never invalidated after SOURCE_ROUTE_FAILURE; end device unreachable until re-pair

5 participants