feat(replication)!: answer a refused fetch with busy instead of silence

[grumbach]

A node that refuses a fetch at admission now tells the requester Busy at once, when the request asked for it and the reply fits within the responder's limits, instead of staying silent. That silence cost every refused requester a 30 s wait and charged the refusing holder a connection-timeout trust event it had not earned.

The problem. A fetch responder admits two outstanding requests per requesting peer and twelve overall. It refuses the rest at admission (Fetch response dropped at admission) and sends nothing back. Then:

• the requester waits its whole fetch timeout (30 s) before trying the next holder;
• saorsa-core records ConnectionTimeout against the holder, which only did what its limits say;
• the requester learns nothing, so it keeps starting new fetches on the same holder.

#252's description reports, from the responders' cumulative admission counters, that 36.6% of inbound fetch requests on 0.21.0 and 44.3% on 0.22.0 were refused this way, and that the WARN line counted 3.38M refusals in the 25-hour window containing the 2026-10-11 join wave of 177 new nodes. #252 keeps a requester within each answering holder's share, which removes most per-peer over-share. Refusals remain for a busy reply to answer: a share still held by requests the requester already gave up on (which #252's description leaves to a busy reply), a twelve-slot pool full of other peers' requests, and a holder #252 currently counts as silent, which it may send requests past the share until an answer clears the mark or the mark expires and the holder is probed under the share again. Older requesters stay on the old silence until they upgrade.

The change.

• Wire. A requester that understands busy replies appends a 5-byte capability trailer (antf and a flags byte) after the encoded FetchRequest. FetchRequest's own layout is unchanged. Older nodes decode with postcard's from_bytes, which ignores trailing bytes, so they read the key and keep refusing silently. FetchResponse::Busy { key } is appended after the existing variants, whose discriminants are unchanged. It is sent only in answer to a request carrying the trailer, so an older requester never receives it.

• Responder. Every message is signed (about 5 KiB with the ML-DSA-65 key and signature), so busy replies are bounded:

  • 48 send permits;
  • per peer, two back to back, then one a second, with at most two in flight;
  • overall, 32 back to back, then 16 a second (about 85 KiB/s sustained);
  • a 2 s send deadline, and the send races shutdown.

  Anything outside those bounds is the old silence. The refusal WARN line keeps its text and counters and gains capability and reply fields.

• Requester. The fetch worker's requests ask for busy replies. A Busy for the requested key carries no trust event; one for any other key is charged as malformed, like every key mismatch. The attempt waits 5-10 s with jitter, never past its own fetch timeout, still holding its fetch slot, then ends as a source failure. So the queue moves on to the next holder exactly as after a timeout, and a busy answer cannot become a fast retry loop. While its cooldown runs, the busy holder is put last when starting or retrying keys, behind every holder that is not cooling; when all are cooling, the first is still tried, so a holder is never withheld. The cooldown is 1, 2, 4, then 8 times the full jittered wait drawn for that answer, even when the attempt's deadline cut the sleep short, capped at 60 s, and a later busy answer never shortens a running cooldown. Serving once resets the multiplier, and so do 10 minutes without a busy answer. The table holds 4096 holders; when it is full, a holder that is not cooling is forgotten first, or, if all are cooling, the one whose cooldown ends soonest.

Merge after #252. #252 bounds how many fetches one requester has open at one answering holder. Without it, a requester with many fetch slots still over-shares, and most of those refusals exceed the per-peer busy allowance and stay silent. The two conflict in src/replication/mod.rs and have not been built or measured together yet. Once #252 merges, this is rebased onto it, the cooldown moves into #252's source choice, and the gates and the real-node measurements below are rerun.

What it does not do.

• Refusals beyond the busy allowances are still silent and still charged.
• A holder that tries to answer busy to everything can still be charged when a request did not ask, a reply was suppressed, or a scheduled send did not complete; otherwise it is put behind every holder that is not cooling (when all are cooling, the first is still tried) and remains subject to storage audits. Charging "never serves" would need a service audit, which this does not add.
• There is no runtime switch. Capability is per request, so mixed versions are safe, and undoing it means a release without it.

Linear issue

Closes V2-1514

Risk tier

• T0 — docs / tooling / CI / pure UX-output. Repo CI only.
• T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
• T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

• Wire: additive and version-gated per request. New FetchResponse::Busy variant, appended, and an optional 5-byte capability trailer after a FetchRequest message. Older nodes ignore the trailer and are never sent Busy. Pinned by tests that decode with mirrors of the previous types.
• Storage: none.
• API: breaking for crate users. FetchRequest gains a public busy_reply field (not serialized), FetchResponse gains a variant, and there is a new public replication::fetch_busy module. Also new and public: ReplicationQueues::retry_fetch_with and protocol::{FETCH_CAPABILITY_MAGIC, FETCH_CAPABILITY_BUSY_REPLY}. Additive test-only hooks under test-utils: ReplicationEngine::{close_fetch_serve_gate_for_test, open_fetch_serve_gate_for_test, fetch_source_cooling_for_test, fetch_source_last_busy_for_test, fetch_admitted_for_test, fetch_pool_full_refusals_for_test}.

Semver impact

• breaking
• feature
• fix

Test evidence

The gates and the real-node runs below were run on upstream main a6e3388 (0.23.0) with this change applied, which is the code and dependency tree CI tests. The full end-to-end suite and the real-node runs were made before the second commit's source change. That change makes the new public SourceBusyTracker::record_busy return the remaining cooldown in force; ant-node uses that value only in a debug line, so node scheduling and wire behaviour are unchanged. After it, the formatting, Clippy, doc and ADR checks, the unit tests, the no-logging build and the two busy-reply end-to-end tests were rerun.

Dev testnet: not run yet. As a T3 change, this needs dev-testnet evidence on the build rebased onto #252 before merge.

Real nodes, against upstream main. tests/e2e/fetch_busy_ab.rs is opt-in: ANT_FETCH_BUSY_AB=1 cargo test --test e2e --features test-utils fetch_busy_ab -- --nocapture --test-threads=1. The same file and the same test-only serve gate were run on main a6e3388. Every run fetched every chunk.

scenario	main a6e3388	this PR
20 requesters, each within its two-request share; the first of five holders keeps its 12-slot pool full for 2 s (28 refusals on main, 26 with this PR). This is one of the refusal cases left after #252.	90.3 s; 14, 9, 9, 4 and 3 requesters charged the five holders	10.5 s; no holder charged
one requester, 8 fetch slots, 48 chunks on three holders in the same order: over-share, which #252 removes	90.2 s, 90.2 s, 90.3 s; first holder charged 6 times, the others 1 to 4 times	38.1 s, 39.2 s, 37.2 s; first holder charged 4 times, the others never

The first scenario was run once per build. On the previous main, 8d25f0c, the same run took 60.4 s with 14 and 4 requesters charging the first two holders, against 10.5 s with none for this PR. The charge counts in the second row are read from the trust scores below: each charge takes a neutral 0.5 down by about 0.06 and then a little less each time, so 0.29 is four charges and 0.23 is six.

Raw output

main a6e3388e pool  run 1: FETCH_BUSY_AB_POOL requesters=20 keys=40 enqueued=40 stored=40 refused_while_full=28 elapsed_ms=90306 requesters_charging_each_holder=[14, 9, 9, 4, 3] trust_lost_per_holder=[1.7638198530638673, 1.002514516809096, 1.9466336090756409, 0.4583219000125878, 0.2945262244958532]
this PR       pool  run 1: FETCH_BUSY_AB_POOL requesters=20 keys=40 enqueued=40 stored=40 refused_while_full=26 elapsed_ms=10535 requesters_charging_each_holder=[0, 0, 0, 0, 0] trust_lost_per_holder=[0.0, 0.0, 0.0, 0.0, 0.0]
main a6e3388e share run 1: FETCH_BUSY_AB slots=8 chunks=48 enqueued=48 stored=48 elapsed_ms=90169 trust_before=[0.5, 0.5, 0.5] trust_after=[0.22617046207604297, 0.3361795380406696, 0.43800011669483147]
main a6e3388e share run 2: FETCH_BUSY_AB slots=8 chunks=48 enqueued=48 stored=48 elapsed_ms=90186 trust_before=[0.5, 0.5, 0.5] trust_after=[0.22617031421379463, 0.33617944994881344, 0.4380000873522496]
main a6e3388e share run 3: FETCH_BUSY_AB slots=8 chunks=48 enqueued=48 stored=48 elapsed_ms=90324 trust_before=[0.5, 0.5, 0.5] trust_after=[0.22617080293489866, 0.2945195203404435, 0.3836883082057919]
this PR       share run 1: FETCH_BUSY_AB slots=8 chunks=48 enqueued=48 stored=48 elapsed_ms=38076 trust_before=[0.5, 0.5, 0.5] trust_after=[0.29445607782725935, 0.5, 0.5]
this PR       share run 2: FETCH_BUSY_AB slots=8 chunks=48 enqueued=48 stored=48 elapsed_ms=39183 trust_before=[0.5, 0.5, 0.5] trust_after=[0.2944590028325731, 0.5, 0.5]
this PR       share run 3: FETCH_BUSY_AB slots=8 chunks=48 enqueued=48 stored=48 elapsed_ms=37164 trust_before=[0.5, 0.5, 0.5] trust_after=[0.2944532978659419, 0.5, 0.5]

End to end (tests/e2e/fetch_busy_reply.rs), deterministic through a test-only fetch serve gate, the same one #252 adds:

• a refused request that asks is told Busy at once, and one that does not ask is refused silently;
• the requester's own fetch worker, refused because its two-request share at the holder is full:
  • records the holder as cooling;
  • holds the attempt for at least the 5 s minimum wait before giving the key up;
  • never lowers the holder's trust.

Unit tests (16 new in fetch_busy, 9 in protocol, 2 in mod):

• mirrors of the previous wire types: both directions decode, and an older requester cannot decode Busy;
• trailer edge cases;
• limiter:
  • burst, refill and the global cap;
  • no spend on a denial;
  • the in-flight limit;
  • the table bound;
• the busy wait's bounds;
• the tracker:
  • cooldown and escalation;
  • not shortened by a later busy answer;
  • reset and expiry;
  • the table bound;
• retry ordering past a cooling holder;
• busy replies are not counted as served bytes.

Simulation. A local, unpublished discrete-event model of the 177-node join wave and 24 edge cases informed the design, including the 5-10 s wait. Its source and raw output are not part of this PR, so its results are not offered as review evidence.

Gates run locally, on main a6e3388 with this change:

• cargo fmt --all -- --check
• cargo clippy --all-targets --all-features -- -D warnings, on stable and on Rust 1.99 (CI's toolchain)
• cargo clippy -- -D warnings
• RUSTDOCFLAGS="--deny=warnings" cargo doc --no-deps
• cargo test --lib --features test-utils: 1215 passed
• RUSTFLAGS="-D warnings" cargo check --lib --no-default-features and RUSTFLAGS="-D warnings" cargo test --lib --no-default-features: 1170 passed
• cargo test --test e2e --features test-utils -- --test-threads=1: 123 passed, plus the three live_testnet tests that are ignored on main
• the bounded-queue and bootstrap-stall regression suites (poc_d1_bounded_queues, poc_bootstrap_stall)
• scripts/adr-governance.py against main

Review. Codex-assisted:

• design: three rounds;
  • round 1 rejected a first design whose retry rotation and expiry charges could storm and punish honest holders;
  • rounds 2 and 3 approved with changes, all made;
• code: five rounds to "No findings", then a fresh full review on 2026-10-16, also "No findings". Its review of the second commit found that a debug line could report a shorter cooldown than the one in force; that is fixed in the same commit, and the commit was then reviewed again with "No findings";
• this description and the ADR: reviewed separately until no findings remained.

New dependency

none

ADR

https://github.com/grumbach/ant-node/blob/fetch-busy-reply/docs/adr/ADR-0025-fetch-busy-reply.md

Mitigation / rollback

Ship a release without it; capability is negotiated per request, so nodes on either side of a mixed fleet keep working.