From 447c000dc7b422dee8f94400035b5821efae6880 Mon Sep 17 00:00:00 2001 From: grumbach Date: Tue, 13 Oct 2026 16:28:51 +0900 Subject: [PATCH] fix(replication): keep fetches within each answering holder's admission share A fetch responder admits two outstanding requests per requesting peer and drops later arrivals without replying while that share is full. The fetch worker ran one fetch per hardware thread and sent each one to the first verified holder of its key, and the nearest-first queue often groups keys that share holders. When more than two fetch slots were free and several queued keys shared a holder, the worker sent more than that holder's share. Each resulting drop cost the full 30 s fetch timeout, a connection-timeout trust event against a holder that did nothing wrong, and a retry that ignored how busy the next holder was. The fetch worker now keeps at most FETCH_MAX_IN_FLIGHT_PER_SOURCE (the responder's share) of its fetches in flight to any one source that is answering: - Each free slot goes to the nearest queued key that can start now, on the first untried source in its preference order that is answering and below the limit, or else on its first silent source. - A key whose untried sources are all answering and full is parked: it leaves the fetch heap, is indexed exactly under each of those sources, and starts again as soon as one frees a share or falls silent, competing with the heap on distance. - A failed fetch moves on by the same rule. If every remaining source is answering and full, it is parked instead of being sent to a full holder, keeping its tried sources and its verification retry reservation. With no untried source left, or a full fetch queue, it returns to verification as before. - The sources a queued or in-flight key is associated with, tried ones included, are capped at close_group_size + paid_list_close_group_size + MAX_HINT_SOURCES_PER_KEY from the running configuration. A re-advertisement merges into such a key's verification retry entry under MAX_HINT_SOURCES_PER_KEY, as into a pending entry, instead of without a bound. A holder that stops answering has no share to keep within, and holding it to two requests per fetch timeout would drain the keys only it holds four times more slowly than before. Each attempt now records whether its source answered. Requests and answers take places in one process-wide sequence, so they are ordered exactly whatever order their outcomes are applied in. An answer reaches the scheduler as soon as it arrives, before its chunk is checked and stored, so a timeout it outdates cannot mark its source while that chunk is still being written. A source is silent while a request sent to it after its latest answer has gone unanswered, and a silent source is fetched from past the limit, as before this change, until it answers again, or until no newer unanswered request to it has been seen for two fetch timeouts (at least one second), after which it is probed under the limit. The table is bounded at 8,192 sources and drops only entries that hold no mark and guard no in-flight fetch; if that frees nothing, untracked sources count as silent for one mark lifetime, which again is the behaviour before this change. A "replication fetch dispatch summary" line every 5 minutes and at shutdown reports parked keys, index size, oldest parked age, queued and in-flight fetches, sources at the limit, parks, attempts by outcome, stored bytes, attempt latency, cancelled attempts, fill-pass lock wait and hold, parked-index maintenance time, and the silent-source counts and events. The limit removes over-share bursts but cannot rule out every refusal: a request this node has already timed out on may still hold its share at the responder until a worker sheds it. Responder admission, the fetch timeout, fetch concurrency and the wire protocol are unchanged. ADR-0024 records the decision, a local risk simulation and a scale benchmark at the fetch queue's hard bound, which runs when ANT_FETCH_SCALE_BENCH names a sources-per-key count. Test-only hooks, compiled only under cfg(test) or the test-utils feature and scoped to one engine: the fetch concurrency, a gate that holds the engine's admitted fetch requests before serving them and that engine shutdown releases, and a fetch-pipeline snapshot. With concurrency eight and twelve keys directed to one holder, the requester records at most two in-flight fetches to it. With a 5 s fetch timeout and sixteen keys directed to a holder that answers nothing, the requester sends past the share once the first requests time out and drains the keys in three timeouts. --- ...uesters-stay-within-the-responder-share.md | 282 ++ docs/adr/README.md | 1 + src/replication/mod.rs | 884 +++++- src/replication/scheduling.rs | 2732 ++++++++++++++++- tests/e2e/fetch_per_source_share.rs | 328 ++ tests/e2e/mod.rs | 3 + tests/e2e/testnet.rs | 8 + 7 files changed, 4005 insertions(+), 233 deletions(-) create mode 100644 docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md create mode 100644 tests/e2e/fetch_per_source_share.rs diff --git a/docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md b/docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md new file mode 100644 index 00000000..909a67f8 --- /dev/null +++ b/docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md @@ -0,0 +1,282 @@ +# ADR-0024: Fetch requesters stay within the responder's per-peer share + +- **Status:** Proposed +- **Date:** 2026-10-13 +- **Decision owners:** +- **Reviewers:** +- **Supersedes:** none +- **Superseded by:** none +- **Related:** [ADR-0005](ADR-0005-replication-repair-hardening.md) (the repair fetch pipeline), + [ADR-0017](ADR-0017-bounded-fresh-offers-and-copy-free-sends.md) (send-path bounds) + +## Context + +A replication fetch is a request/response exchange between two nodes. The responder bounds what +one requester can occupy: + +- `FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER = 2` admitted requests per source, checked before the + 12-slot admission pool, in front of a 3-worker serving pool; +- a request past that share is not answered. The responder logs + `Fetch response dropped at admission: reason=per_peer_cap_full` and returns nothing. + +The requesting side never matched that share: + +- the fetch worker ran one fetch per hardware thread; +- it sent each one to the first verified holder of its key, in source-preference order (the + closest holder that answered Present); +- the fetch queue is nearest-first, so the keys at its head usually share the same few holders. + +When more than two fetch slots were available and several queued keys shared a holder, the worker +could send more than that holder's two-request share. A request arriving while two from the same +requester were still admitted was dropped. Each such drop cost the requester: + +- the full fetch timeout (30 s) before moving on; +- a `ConnectionTimeout` trust event, recorded by the transport against the holder that dropped it, + although that holder had done nothing wrong; +- a retry that went to the next holder without regard to how busy that holder was. + +On the production fleet this was common, according to the per-service cumulative counters: + +| period | inbound fetch requests dropped at admission | +|---|---| +| on 0.21.0 | 36.6% | +| on 0.22.0 | 44.3% | +| 25-hour window containing a wave of 177 freshly installed nodes filling their ranges | 65.8% overall by cumulative counters; 3.22M WARN-line drops came from the new nodes, about 2.8 per chunk attributed to them | + +In that window, 68.5% of drops happened with only 2 of the responder's 12 admission slots in use, +both held by the refused peer. + +## Decision Drivers + +- No wire, storage or production public API change, so the fix ships as a behavioural change on + the release train. +- Do not weaken the responder's protection against one peer occupying the whole serving pool. +- A key must never be lost, tracked twice, or released from bootstrap accounting early because of + scheduling. +- Keep the queue's nearest-first priority and each key's source preference whenever they do not + conflict with the share. +- No scenario may drain more slowly than before the change by more than one fetch timeout, in + particular the keys of a holder that has stopped answering. + +## Considered Options + +1. **Raise the responder's per-peer share.** Rejected. It hides the symptom, lets one requester take + all three serving workers, which is what the share exists to prevent, and still drops requesters + with more threads than the new share. +2. **Answer an over-share request with an explicit "busy" reply.** Deferred. It needs a new + response variant, which peers that cannot decode it treat as malformed and charge an application + failure, so it is a wire change with its own rollout. It removes the cost of a drop, not the + cause. +3. **Limit fetch concurrency to 2.** Rejected. It serialises a node's whole repair and bootstrap fill + through one share even when other holders are idle. +4. **Ship the scheduler switched off, behind a per-node switch, for a later canary.** Rejected. It + merges code that no node runs, so its risks are deferred rather than measured. The risks were + simulated instead (below), and the one regression found was fixed before merge. +5. **Keep at most the share in flight per answering holder on the requesting side, and give each + free slot to the nearest queued key that can start now.** Chosen. + +## Decision + +The fetch worker keeps at most `FETCH_MAX_IN_FLIGHT_PER_SOURCE` fetches in flight to any one source +peer that is answering. The constant is defined equal to `FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER`, +the share every deployed responder grants. It is the worker's only scheduling path. + +**Dispatch.** Each free fetch slot goes to the nearest queued key that can start now. In the key's +source-preference order, over its untried sources, it starts on: + +1. the first source that is answering and below the limit; +2. otherwise the first silent source (below), past the limit. + +**Parking.** + +- A key whose untried sources are all answering and at the limit is parked: it leaves the fetch + heap and is indexed under each of those sources. +- It becomes a candidate again as soon as any of them has a free share or falls silent, and it + competes with the heap on distance. +- Each key leaves the heap once, so a startable key further down is reached in the same pass, and + no source ordering can hide a startable key. + +**Retries.** + +- A failed fetch moves to its next untried source by the same rule. If every untried source is + answering and at the limit, it is parked rather than sent to a full holder. +- It keeps the sources it already tried, so none is asked twice in one verification cycle, and it + keeps its verification retry reservation. +- When no untried source is left, or the fetch queue is full, it returns to verification as before, + so the fetch queue's hard bound holds. + +**Silent sources.** The limit protects a holder's admission share, and only a holder that answers +has one. Without an exception, the keys held only by a holder that stopped answering would be tried +two per fetch timeout instead of one per free slot: four times more slowly than before on a node +with eight slots. + +- Each attempt records whether its source answered. Requests and answers take places in one + process-wide sequence, as a request is sent and as an answer returns, so they can be ordered + exactly, whatever order their outcomes are applied in and whatever the clock reads. +- An answer reaches the scheduler as soon as it arrives, before its chunk is checked and stored, + and again with the attempt's outcome, to no further effect. A timeout that an answer already + outdates therefore never marks the source while that answer's chunk is still being written. +- A source is **silent** while a request sent to it after its latest answer has gone without any + response (a timeout or connection failure). Any answer received after that request was sent ends + it. +- A silent source is fetched from past the limit, as before the change, until it answers again, + or until this node has learned of no newer unanswered request to it for two fetch timeouts (at + least one second). A holder that stays dead while this node keeps fetching from it therefore + stays silent, and one this node stopped fetching from is probed under the limit when it is next + used, marked again only if a request sent then also goes unanswered. +- The table of observations is bounded at 8,192 sources. It drops only entries that hold no mark and + whose source has no fetch in flight. If that frees nothing, a source not in the table counts as + silent for one mark lifetime, which is again the behaviour before the change, never a slower one. + +**Source limit.** + +- The sources a queued or in-flight key is associated with, tried ones included, are bounded by + `close_group_size + paid_list_close_group_size + MAX_HINT_SOURCES_PER_KEY` from the running + configuration. +- That never truncates a verification round's own sources, and it bounds the parked index. +- A re-advertisement of a queued or in-flight key merges into the verification entry kept for its + retry under `MAX_HINT_SOURCES_PER_KEY`, the bound a pending entry has, in the same order: replica + claimants first, displacing paid-only advertisers other than the key's capacity owner, then + paid-only advertisers while there is room. Before the change that merge had no bound. + +**Dispatch summary.** The fetch worker logs `replication fetch dispatch summary` every 5 minutes and +at shutdown, with: + +- the parked count, the index size and the oldest parked age; +- queued and in-flight fetches, and sources at the limit; +- parks and retry parks; +- attempts by outcome, bytes stored, and attempt latency; +- the longest fill-pass lock wait and hold, and the longest parked-index maintenance after a + departure; +- silent sources now and at peak, marks, clears, expiries, ignored stale timeouts, starts on silent + sources, unreachable attempts, tracked sources and table overflows. + +Responder admission, the fetch timeout, fetch concurrency and the wire protocol are unchanged. + +## Risk simulation + +A local 12-node network, eight fetch slots per node and a 5 s fetch timeout, timed each scenario on +this change and on a build with the limit removed and no silent-source handling, which schedules +exactly as before the change: each key to its first untried source. + +| scenario | this change | before the change | +|---|---|---| +| fill: 48 keys held by 3 answering holders | 0.36 s | 10.08 s | +| a non-answering holder first among 3 sources, 2 answering: 24 keys | 5.1 s | 20.1 s | +| a non-answering sole holder: 16 keys | 15.0 s | 10.15 s | + +The third row is the accepted penalty: the first two requests to a holder that just went silent take +one fetch timeout to show it, and the rest then go out past the limit. It is paid once each time +this node starts fetching from a silent holder again after two fetch timeouts without trying it. +For `N` keys held only by that holder, `C ≥ 2` fetch slots and fetch timeout `T`, draining takes +`T × (1 + ceil((N − 2) / C))` against `T × ceil(N / C)` before the change. With one fetch slot the +first wave is one request, and the bound is the same with 1 in place of 2. Without the +silent-source exception the same scenario took 40.1 s. + +The scale benchmark at the fetch queue's hard bound (`scale_benchmark_at_the_fetch_queue_bound`, +release build, 131,072 parked keys) stayed within the thresholds declared before it ran, apart from +dropping the queues (below): + +| sources per key | one park pass, over repeated runs | peak process RSS | +|---|---|---| +| 1 | 89 to 99 ms | 197 MiB | +| 8 | 207 to 216 ms | 342 MiB | +| 35 (the limit) | 754 to 960 ms | 1.13 GiB | + +- Waking and the stats read took under 1 ms; a departure took at most 107 ms. +- Dropping the queues at the limit took 274 to 315 ms over repeated runs. The scheduler before the + silent-source handling took 226 to 295 ms, and the same time as this code in alternating runs on + one machine. It was first held to 250 ms, which a loaded machine did not meet. It happens once, + when the engine shuts down, outside any lock, so it is now held to 1 s. +- The silent-source check makes the park pass about 5% slower than before. + +## Consequences + +### Positive + +- A requester on this release keeps its own in-flight fetches to one answering holder within that + holder's share. That removes the ordinary over-share bursts and their timeout and trust cost; it + does not cover stale work the responder still holds, described below. +- A fill or repair can finish sooner, because fetch slots no longer spend 30 s on over-share + requests that will not be answered. +- Responders should log fewer `per_peer_cap_full` drops as requesters upgrade. +- The dispatch summary shows the scheduler's queue state, lock times and silent sources in + production. + +### Negative / Trade-offs + +- It is not a guarantee of zero refusals. A request this node has already timed out on can still + hold its share at the responder until a worker sheds it, and a new request sent in that window + is refused. Closing that window needs option 2. During the 25-hour wave, responders shed 31,172 + admitted requests after waiting 30 s for a worker; the production data do not show how often + that window caused a later refusal. +- The requester counts a fetch as in flight for its whole attempt, local checks and the local write + included, which is longer than the responder holds its share. A holder can therefore sit below + its share while this node waits, and a fetch whose local work stalls holds its holder's share for + as long as it holds its fetch slot. +- A key whose answering holders are all busy waits in the queue, or goes to a lower-ranked holder + with room, instead of being sent to its first holder. +- Keys held only by a holder that just stopped answering take one fetch timeout longer to drain + than before the change, as in the simulation. A holder that is alive but so overloaded that it + times out is fetched from as before the change while its mark lasts. +- Recovering the slots dropped requests used to waste raises a filling node's instantaneous + bandwidth, disk and CPU use. Each responder still serves at most its share per requester with + three workers; the data a joining node needs to move is unchanged. +- At the source limit, the parked index can reach 4.6 million entries and about 1.1 GiB of process + memory; at 8 sources per key, about 340 MiB. +- Requesters on older releases still overshoot until they upgrade, so the network-wide drop rate + falls only as the release spreads, and the traffic pattern changes during that time. +- There is no runtime switch. Undoing the change means a release without it. + +### Neutral / Operational + +- The effect shows in the responders' cumulative counters (`replication responder admission + summary`): `fetch_admission_dropped_per_peer_share` relative to `fetch_request_rx_count` should + fall as the fleet and the community upgrade. +- Before release, a dev testnet run with a join wave of fresh nodes should compare drain time, + stored bytes, admission drops, trust events and host load against the previous release. + +## Validation + +**Unit tests on the queue:** + +- no answering source exceeds the share; +- preference order is kept; +- a startable key behind hundreds of blocked keys starts in the same pass; +- a freed share starts the nearest parked key, whatever its source; +- parked and retried keys keep their payload, tried sources and retry reservation; +- departures and merges keep the parked index exact; +- the source limit holds across park, retry and merge; +- a silent source is fetched from past the share, an answering source with room is preferred, and + parked keys start when a full source falls silent; +- whether a source is silent does not depend on the order its outcomes are applied in; marks are + set, cleared and expired as described, and the table bound keeps every guard and mark; +- an answer reported when it arrives keeps a source within its share when an earlier request to it + times out while the answered fetch is still storing its chunk; known only from the outcome, the + same answer comes too late and the source is sent every waiting key; +- re-advertisements keep a queued or in-flight key's retry entry within the hint bound, with its + capacity owner; +- sixteen keys held only by a silent holder, with eight slots, start in exactly three waves of 2, 8 + and 6; +- a randomised run of queue operations, re-advertisements and source contacts keeps every + invariant, the retry entries' bound included. + +**End-to-end tests:** + +- with eight fetch slots and twelve keys directed to one holder that holds what it admits, the + requester's own pipeline records at most two in-flight fetches to that holder; with the limit + removed it records eight and the test fails; +- with a 5 s fetch timeout and sixteen keys directed to a holder that answers nothing, the requester + sends past the share once the first requests time out and the keys leave its pipeline within four + timeouts (15.1 s measured); with the silent-source exception removed the test fails. + +**Revisit if:** + +- the responder's per-peer share changes; +- a busy reply is introduced; +- the fleet counters keep showing a high dropped share once most requesters run this release; +- the dispatch summary shows sustained table overflows or a large count of silent sources. + +## Notes for AI-assisted work + +AI tools may help draft this ADR, but **must not mark it Accepted without human review**. Accepted ADRs are immutable: create a new superseding ADR rather than editing an Accepted ADR. diff --git a/docs/adr/README.md b/docs/adr/README.md index b58e9a32..bcde1f6f 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -40,3 +40,4 @@ See [`TOOLING.md`](./TOOLING.md) for `adrs`, `adr-kit`, and AI harness setup. - [ADR-0018: Pointer ownership transfer by final redirection](./ADR-0018-pointer-transfer-by-final-redirection.md) - [ADR-0019: Pointer audits serve the record round 1 bound](./ADR-0019-pointer-audits-serve-the-record-round-one-bound.md) - [ADR-0022: Remove the LMDB Chunk Store](./ADR-0022-remove-the-lmdb-chunk-store.md) +- [ADR-0024: Fetch requesters stay within the responder's per-peer share](./ADR-0024-fetch-requesters-stay-within-the-responder-share.md) diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 62937413..2c50bd27 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -42,6 +42,8 @@ use std::fmt; use std::num::NonZeroUsize; use std::ops::ControlFlow; use std::path::{Path, PathBuf}; +#[cfg(any(test, feature = "test-utils"))] +use std::sync::atomic::AtomicBool; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; use std::time::{Duration, Instant, SystemTime}; @@ -62,6 +64,7 @@ use tokio_util::sync::CancellationToken; use tokio_util::task::TaskTracker; use crate::ant_protocol::XorName; +use crate::client::xor_distance; use crate::error::{Error, Result}; use crate::payment::{ PaymentVerifier, VerificationContext, MAX_PAYMENT_PROOF_SIZE_BYTES, @@ -95,7 +98,12 @@ use crate::replication::protocol::{ }; use crate::replication::quorum::KeyVerificationOutcome; use crate::replication::recent_provers::RecentProvers; -use crate::replication::scheduling::{CapacityDisplacement, DeferralOutcome, ReplicationQueues}; +#[cfg(any(test, feature = "test-utils"))] +pub use crate::replication::scheduling::FetchPipelineSnapshot; +use crate::replication::scheduling::{ + fetch_source_limit, CapacityDisplacement, Contact, DeferralOutcome, FetchDispatchStats, + FetchRetry, ReplicationQueues, +}; use crate::replication::storage_commitment_audit::PointerBindings; use crate::replication::types::{ AuditFailureReason, BootstrapClaimObservation, BootstrapState, FailureEvidence, @@ -116,6 +124,16 @@ use saorsa_pqc::api::sig::{MlDsaSecretKey, MlDsaVariant}; /// counted: it is not a saturation signal. static FIRST_AUDIT_INGRESS_DROPPED: AtomicU64 = AtomicU64::new(0); +/// One sequence over every fetch request this process sends and every answer +/// it receives, so the fetch scheduler can order a request against an answer +/// even when both read the same clock value. See [`Contact`]. +static FETCH_CONTACT_SEQUENCE: AtomicU64 = AtomicU64::new(0); + +/// The next place in [`FETCH_CONTACT_SEQUENCE`]. +fn next_fetch_contact() -> u64 { + FETCH_CONTACT_SEQUENCE.fetch_add(1, Ordering::Relaxed) +} + /// Record one ingress-full drop. Called by the payment verifier's `try_send` /// sites; read by the drainer's periodic summary. pub(crate) fn note_monetized_ingress_drop() { @@ -1129,6 +1147,24 @@ const FETCH_RESPONDER_MAX_OUTSTANDING: usize = FETCH_RESPONDER_WORKER_LIMIT * 4; /// of the three workers available to other peers under a single-source flood. const FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER: u32 = 2; +/// Maximum fetches this node keeps in flight to any one source peer that is +/// answering. +/// +/// The share every responder grants one requester +/// ([`FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER`], unchanged since it was +/// introduced). A request past that share is dropped without an answer, so +/// sending it costs the full fetch timeout and a timeout trust event against a +/// holder that did nothing wrong. Staying within it removes those bursts; it +/// cannot rule out every refusal, because a request this node has already +/// timed out on may still be holding its share at the responder until a worker +/// sheds it. +/// +/// A source whose request got no answer, and which has not answered since that +/// request was sent, has no share to stay within. It is fetched from without +/// this cap, as before the cap existed, until it answers again or two fetch +/// timeouts pass, so a holder that went silent does not hold its keys back. +const FETCH_MAX_IN_FLIGHT_PER_SOURCE: usize = FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER as usize; + /// Maximum verification batches served concurrently. /// /// Point lookups are fast, but a batch can contain 8,192 of them. Two @@ -1222,7 +1258,9 @@ fn paid_notify_payment_context() -> VerificationContext { } /// Boxed future type for in-flight fetch tasks. -type FetchFuture = Pin)> + Send>>; +/// A fetch task's key, its outcome (`None` when the task panicked), and when +/// the task finished. +type FetchFuture = Pin, Instant)> + Send>>; /// Fresh-offer keys currently being handled, each with the proofs still to try. /// @@ -1667,6 +1705,8 @@ pub struct ReplicationEngine { payment_verifier: Arc, /// Replication pipeline queues. queues: Arc>, + /// Fetches the worker runs at once: one per hardware thread. + fetch_concurrency: usize, /// Neighbor sync cycle state. sync_state: Arc>, /// Per-peer sync history (for `RepairOpportunity`). @@ -1787,6 +1827,9 @@ pub struct ReplicationEngine { fetch_responder_admission_semaphore: Arc, /// Per-source fetch responder counts for flood-fair admission. fetch_responder_inflight: Arc>>, + /// Test-only hold on serving admitted fetch requests. + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc, /// Worker permits for lookup-heavy verification batches. verification_responder_worker_semaphore: Arc, /// Admission permits bounding verification workers and queued waiters. @@ -1902,13 +1945,21 @@ impl ReplicationEngine { let (monetized_pin_tx, monetized_pin_rx) = mpsc::channel(config::FIRST_AUDIT_INGRESS_CAPACITY); + let source_limit = + fetch_source_limit(config.close_group_size, config.paid_list_close_group_size); + let engine = Self { config: Arc::clone(&config), p2p_node, storage, paid_list, payment_verifier, - queues: Arc::new(RwLock::new(ReplicationQueues::new())), + queues: Arc::new(RwLock::new( + ReplicationQueues::new() + .with_fetch_source_limit(source_limit) + .with_fetch_request_timeout(config.fetch_request_timeout), + )), + fetch_concurrency: max_parallel_fetch(), sync_state: Arc::new(RwLock::new(initial_neighbors)), sync_history: Arc::new(RwLock::new(HashMap::new())), audit_on_gossip_cooldown: Arc::new(RwLock::new(HashMap::new())), @@ -1940,6 +1991,8 @@ impl ReplicationEngine { FETCH_RESPONDER_MAX_OUTSTANDING, )), fetch_responder_inflight: Arc::new(RwLock::new(HashMap::new())), + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc::new(FetchServeGate::default()), verification_responder_worker_semaphore: Arc::new(Semaphore::new( VERIFICATION_RESPONDER_WORKER_LIMIT, )), @@ -2194,6 +2247,35 @@ impl ReplicationEngine { self.queues.read().await.fetch_queued_or_in_flight(key) } + /// Test-only: run `concurrency` fetches at once instead of one per hardware + /// thread. Call before [`Self::start`], which reads it. Per engine, so it + /// affects no other node or test. + #[cfg(any(test, feature = "test-utils"))] + pub fn set_fetch_concurrency_for_test(&mut self, concurrency: usize) { + self.fetch_concurrency = concurrency; + } + + /// Test-only: hold this engine's admitted fetch requests before they are + /// served, until [`Self::open_fetch_serve_gate_for_test`]. + #[cfg(any(test, feature = "test-utils"))] + pub fn close_fetch_serve_gate_for_test(&self) { + self.fetch_serve_gate.close(); + } + + /// Test-only: release this engine's held fetch requests and stop holding + /// new ones. + #[cfg(any(test, feature = "test-utils"))] + pub fn open_fetch_serve_gate_for_test(&self) { + self.fetch_serve_gate.open(); + } + + /// Test-only: every key queued for fetch and every in-flight fetch with + /// the source it was sent to, read under one lock. + #[cfg(any(test, feature = "test-utils"))] + pub async fn fetch_pipeline_snapshot_for_test(&self) -> FetchPipelineSnapshot { + self.queues.read().await.fetch_pipeline_snapshot() + } + /// Test-only: place `key` into pending verification as though `hinter` had /// just advertised it with a replica hint. Returns whether it was admitted. /// @@ -2434,6 +2516,12 @@ impl ReplicationEngine { } } + // A test may shut down with the fetch serve gate still closed. The + // requests it holds wait on the gate before they reach a worker pool, + // so release them, or the drain below never finishes. + #[cfg(any(test, feature = "test-utils"))] + self.fetch_serve_gate.open(); + // Close the responder worker pools. Detached tasks still queued for a // worker take the `Err` arm of `acquire_owned()` and exit immediately // rather than waiting for a slot that will never free up — without @@ -3063,6 +3151,8 @@ impl ReplicationEngine { let fetch_responder_admission_semaphore = Arc::clone(&self.fetch_responder_admission_semaphore); let fetch_responder_inflight = Arc::clone(&self.fetch_responder_inflight); + #[cfg(any(test, feature = "test-utils"))] + let fetch_serve_gate = Arc::clone(&self.fetch_serve_gate); let verification_responder_worker_semaphore = Arc::clone(&self.verification_responder_worker_semaphore); let verification_responder_admission_semaphore = @@ -3119,6 +3209,8 @@ impl ReplicationEngine { fetch_responder_worker_semaphore, fetch_responder_admission_semaphore, fetch_responder_inflight, + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate, verification_responder_worker_semaphore, verification_responder_admission_semaphore, verification_responder_inflight, @@ -3881,72 +3973,91 @@ impl ReplicationEngine { let is_bootstrapping = Arc::clone(&self.is_bootstrapping); let bootstrap_complete_notify = Arc::clone(&self.bootstrap_complete_notify); let detached_task_tracker = self.detached_task_tracker.clone(); - let concurrency = max_parallel_fetch(); + let concurrency = self.fetch_concurrency; info!("Fetch worker concurrency set to {concurrency} (hardware threads)"); let handle = tokio::spawn(async move { - // Each in-flight future yields (key, Option) so we - // always recover the key — even if the inner task panics. + // Each in-flight future yields (key, Option, when it + // finished), so we always recover the key, even if the inner task + // panics. let mut in_flight = FuturesUnordered::::new(); + let mut dispatch_window = FetchDispatchWindow::new(); + // Each answer as it arrives, ahead of its fetch's outcome. + let (answers, mut early_answers) = mpsc::unbounded_channel(); loop { - // Fill up to `concurrency` slots from the queue. - { - let mut q = queues.write().await; - while in_flight.len() < concurrency { - let Some(candidate) = q.dequeue_fetch() else { - break; - }; - let fetch_key = candidate.key; - let Some(&source) = candidate.sources.first() else { - warn!( - "Fetch candidate {} has no sources; requeueing for verification", - hex::encode(fetch_key) - ); - let _ = q.requeue_candidate_for_verification( - candidate, - config.verification_request_timeout, - ); - continue; - }; - q.start_dequeued_fetch(candidate, source); - - let p2p = Arc::clone(&p2p); - let storage = Arc::clone(&storage); - let config = Arc::clone(&config); - let token = shutdown.clone(); - let tracker = detached_task_tracker.clone(); - in_flight.push(Box::pin(async move { - // Tracked so shutdown() still awaits the task if - // this awaiter is dropped (e.g. the worker is - // aborted): it holds Arc and must - // not outlive the engine. - let handle = tracker.spawn(async move { - // Cancel-aware: abort when the engine shuts down. - tokio::select! { - () = token.cancelled() => FetchOutcome { - key: fetch_key, - result: FetchResult::SourceFailed, - }, - outcome = execute_single_fetch( - p2p, storage, config, fetch_key, source, - ) => outcome, - } - }); - match handle.await { - Ok(outcome) => (outcome.key, Some(outcome)), - Err(e) => { - error!( - "Fetch task for {} panicked: {e}", - hex::encode(fetch_key) - ); - (fetch_key, None) - } + // Fill free slots from the queue, never sending more than + // `FETCH_MAX_IN_FLIGHT_PER_SOURCE` of this node's fetches to any + // one answering source at once. + let lock_requested = Instant::now(); + let mut q = queues.write().await; + let lock_acquired = Instant::now(); + record_early_answers(&mut q, &mut early_answers); + let free_slots = concurrency.saturating_sub(in_flight.len()); + let dispatch = q.dispatch_fetches(free_slots, FETCH_MAX_IN_FLIGHT_PER_SOURCE); + let parked = dispatch.parked; + for candidate in dispatch.sourceless { + warn!( + "Fetch candidate {} has no untried sources; requeueing for verification", + hex::encode(candidate.key) + ); + let _ = q.requeue_candidate_for_verification( + candidate, + config.verification_request_timeout, + ); + } + for (fetch_key, source) in dispatch.started { + let p2p = Arc::clone(&p2p); + let storage = Arc::clone(&storage); + let config = Arc::clone(&config); + let answers = answers.clone(); + let token = shutdown.clone(); + let tracker = detached_task_tracker.clone(); + in_flight.push(Box::pin(async move { + // Tracked so shutdown() still awaits the task if + // this awaiter is dropped (e.g. the worker is + // aborted): it holds Arc and must + // not outlive the engine. + let handle = tracker.spawn(async move { + // Cancel-aware: abort when the engine shuts down. + tokio::select! { + () = token.cancelled() => FetchOutcome { + key: fetch_key, + source, + result: FetchResult::SourceFailed, + contact: Contact::Unknown, + }, + outcome = execute_single_fetch( + p2p, storage, config, fetch_key, source, answers, + ) => outcome, } - })); - } - } // release queues write lock + }); + let joined = handle.await; + let completed_at = Instant::now(); + match joined { + Ok(outcome) => (outcome.key, Some(outcome), completed_at), + Err(e) => { + error!("Fetch task for {} panicked: {e}", hex::encode(fetch_key)); + (fetch_key, None, completed_at) + } + } + })); + } + drop(q); + dispatch_window.parks += parked; + dispatch_window.record_fill( + lock_acquired.duration_since(lock_requested), + lock_acquired.elapsed(), + ); + if dispatch_window.age() >= FETCH_DISPATCH_SUMMARY_INTERVAL { + let stats_requested = Instant::now(); + let stats = queues + .write() + .await + .take_fetch_dispatch_stats(FETCH_MAX_IN_FLIGHT_PER_SOURCE); + dispatch_window.log_and_reset(&stats, stats_requested.elapsed()); + } if in_flight.is_empty() { // No work — wait for new items or shutdown. @@ -3961,22 +4072,32 @@ impl ReplicationEngine { // Wait for the next fetch to complete and process the result. tokio::select! { () = shutdown.cancelled() => break, - Some((key, maybe_outcome)) = in_flight.next() => { + Some((key, maybe_outcome, completed_at)) = in_flight.next() => { let mut q = queues.write().await; + record_early_answers(&mut q, &mut early_answers); + let attempt = attempt_latency(q.in_flight_started_at(&key), completed_at); + dispatch_window.record_outcome(maybe_outcome.as_ref(), attempt); let terminal = if let Some(outcome) = maybe_outcome { + q.record_fetch_contact(outcome.source, outcome.contact); match apply_fetch_result( &mut q, &key, &outcome.result, config.verification_request_timeout, + xor_distance(&key, p2p.peer_id().as_bytes()), ) { FetchFollowUp::Terminal => true, FetchFollowUp::RequeuedForVerification => false, + FetchFollowUp::Parked => { + dispatch_window.retry_parks += 1; + false + } FetchFollowUp::RetryFrom(next_peer) => { // Spawn a new fetch task for the next source. let p2p = Arc::clone(&p2p); let storage = Arc::clone(&storage); let config = Arc::clone(&config); + let answers = answers.clone(); let token = shutdown.clone(); let tracker = detached_task_tracker.clone(); let fetch_key = key; @@ -3987,21 +4108,32 @@ impl ReplicationEngine { tokio::select! { () = token.cancelled() => FetchOutcome { key: fetch_key, + source: next_peer, result: FetchResult::SourceFailed, + contact: Contact::Unknown, }, outcome = execute_single_fetch( - p2p, storage, config, fetch_key, next_peer, + p2p, + storage, + config, + fetch_key, + next_peer, + answers, ) => outcome, } }); - match handle.await { - Ok(outcome) => (outcome.key, Some(outcome)), + let joined = handle.await; + let completed_at = Instant::now(); + match joined { + Ok(outcome) => { + (outcome.key, Some(outcome), completed_at) + } Err(e) => { error!( "Fetch task for {} panicked: {e}", hex::encode(fetch_key) ); - (fetch_key, None) + (fetch_key, None, completed_at) } } })); @@ -4043,7 +4175,22 @@ impl ReplicationEngine { // Cancel and drain remaining in-flight fetches on shutdown. // The CancellationToken is already cancelled by this point, so // spawned tasks will see cancellation via their select! branches. - while in_flight.next().await.is_some() {} + // Their outcomes are counted but not applied, so the final + // summary's in-flight count still includes them. + let mut drained = Vec::new(); + while let Some(resolved) = in_flight.next().await { + drained.push(resolved); + } + let stats_requested = Instant::now(); + let mut q = queues.write().await; + for (key, maybe_outcome, completed_at) in &drained { + let attempt = attempt_latency(q.in_flight_started_at(key), *completed_at); + dispatch_window.record_outcome(maybe_outcome.as_ref(), attempt); + } + let stats = q.take_fetch_dispatch_stats(FETCH_MAX_IN_FLIGHT_PER_SOURCE); + drop(q); + dispatch_window.shutdown = true; + dispatch_window.log_and_reset(&stats, stats_requested.elapsed()); debug!("Fetch worker shut down"); }); self.task_handles.push(handle); @@ -4589,6 +4736,9 @@ struct ReplicationMessageHandlerContext { fetch_responder_worker_semaphore: Arc, fetch_responder_admission_semaphore: Arc, fetch_responder_inflight: Arc>>, + /// Test-only hold on serving admitted fetch requests. + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc, verification_responder_worker_semaphore: Arc, verification_responder_admission_semaphore: Arc, verification_responder_inflight: Arc>>, @@ -7112,6 +7262,45 @@ pub fn verification_requests_for_key_from_for_test(requester: &PeerId, key: &Xor }) } +/// Test-only: holds an engine's admitted fetch requests before they are served. +/// +/// A held request keeps its admission slot and its requester's per-peer share, +/// exactly as a request waiting for a worker does, so a test can read what a +/// requester has outstanding without racing the speed at which the host serves +/// a chunk. One per engine, so closing it affects no other node or test. +#[cfg(any(test, feature = "test-utils"))] +#[derive(Debug, Default)] +struct FetchServeGate { + closed: AtomicBool, + opened: Notify, +} + +#[cfg(any(test, feature = "test-utils"))] +impl FetchServeGate { + fn close(&self) { + self.closed.store(true, Ordering::Release); + } + + fn open(&self) { + self.closed.store(false, Ordering::Release); + self.opened.notify_waiters(); + } + + /// Wait while the gate is closed. The wakeup is registered before the flag + /// is read, so an open that lands between the two is not missed. + async fn wait_open(&self) { + loop { + let opened = self.opened.notified(); + tokio::pin!(opened); + opened.as_mut().enable(); + if !self.closed.load(Ordering::Acquire) { + return; + } + opened.await; + } + } +} + async fn handle_verification_request( source: &PeerId, request: &protocol::VerificationRequest, @@ -7365,8 +7554,12 @@ async fn dispatch_fetch_request( let p2p_node = Arc::clone(&ctx.p2p_node); let request_timeout = ctx.config.fetch_request_timeout; let rr_message_id = rr_message_id.map(ToOwned::to_owned); + #[cfg(any(test, feature = "test-utils"))] + let serve_gate = Arc::clone(&ctx.fetch_serve_gate); ctx.detached_task_tracker.spawn(async move { let _guard = guard; + #[cfg(any(test, feature = "test-utils"))] + serve_gate.wait_open().await; let Ok(_worker_permit) = worker_semaphore.acquire_owned().await else { debug!( responder_class = "fetch", @@ -8931,7 +9124,10 @@ async fn complete_bootstrap( /// Result classification for a single fetch attempt. enum FetchResult { /// Data fetched, integrity-checked, and stored successfully. - Stored, + Stored { + /// Size of the stored chunk, for the per-source scheduler's summary. + bytes: usize, + }, /// Content-address integrity check failed — do not retry. IntegrityFailed, /// Source failed (network error or non-success response) — retryable. @@ -8979,7 +9175,159 @@ enum FetchResult { /// worker loop to update queue state. struct FetchOutcome { key: XorName, + /// The source this attempt was sent to. + source: PeerId, result: FetchResult, + /// Whether `source` answered, for the per-source cap's exemption of + /// sources that stopped answering. + contact: Contact, +} + +/// How often the fetch worker reports its dispatching. +const FETCH_DISPATCH_SUMMARY_INTERVAL: Duration = Duration::from_mins(5); + +/// How long an attempt that started at `started` took, up to when its task +/// finished rather than when the worker got to its outcome. +fn attempt_latency(started: Option, completed_at: Instant) -> Option { + started.map(|started| completed_at.saturating_duration_since(started)) +} + +/// What the fetch worker observed about its own dispatching since the last +/// summary. +#[derive(Default)] +struct FetchDispatchWindow { + /// When this window began. + started: Option, + /// Longest wait for the queues write lock before a fill pass. + fill_lock_wait_max: Duration, + /// Longest hold of the queues write lock by a fill pass, spawning included. + fill_lock_hold_max: Duration, + /// Fill passes in this window. + fills: u64, + /// Queued keys parked by fill passes in this window. + parks: usize, + /// Failed fetches parked because every untried source was full. + retry_parks: u64, + /// Attempts by outcome. + stored: u64, + already_held: u64, + no_longer_responsible: u64, + source_failed: u64, + integrity_failed: u64, + local_write_failed: u64, + panicked: u64, + /// Attempts cancelled by shutdown before they resolved. + cancelled: u64, + /// Bytes of chunks stored by attempts in this window. + stored_bytes: u64, + /// Attempt latency: longest, and total over `timed_attempts`. + attempt_max: Duration, + attempt_total: Duration, + timed_attempts: u64, + /// Set for the summary logged at shutdown, whose in-flight count includes + /// attempts drained without being applied. + shutdown: bool, +} + +impl FetchDispatchWindow { + fn new() -> Self { + Self { + started: Some(Instant::now()), + ..Self::default() + } + } + + /// Time since this window began. + fn age(&self) -> Duration { + self.started + .map_or(Duration::ZERO, |started| started.elapsed()) + } + + fn record_fill(&mut self, wait: Duration, hold: Duration) { + self.fill_lock_wait_max = self.fill_lock_wait_max.max(wait); + self.fill_lock_hold_max = self.fill_lock_hold_max.max(hold); + self.fills += 1; + } + + /// Count one resolved attempt: `None` when its task panicked. A cancelled + /// attempt is counted as cancelled whatever result it carries, and its + /// cut-short latency is left out. + fn record_outcome(&mut self, outcome: Option<&FetchOutcome>, attempt: Option) { + if outcome.is_some_and(|outcome| outcome.contact == Contact::Unknown) { + self.cancelled += 1; + return; + } + match outcome.map(|outcome| &outcome.result) { + Some(FetchResult::Stored { bytes }) => { + self.stored += 1; + self.stored_bytes = self + .stored_bytes + .saturating_add(u64::try_from(*bytes).unwrap_or(u64::MAX)); + } + Some(FetchResult::AlreadyHeld) => self.already_held += 1, + Some(FetchResult::NoLongerResponsible) => self.no_longer_responsible += 1, + Some(FetchResult::SourceFailed) => self.source_failed += 1, + Some(FetchResult::IntegrityFailed) => self.integrity_failed += 1, + Some(FetchResult::LocalWriteFailed) => self.local_write_failed += 1, + None => self.panicked += 1, + } + if let Some(attempt) = attempt { + self.attempt_max = self.attempt_max.max(attempt); + self.attempt_total = self.attempt_total.saturating_add(attempt); + self.timed_attempts += 1; + } + } + + /// Log this window with the queues' state, then start a new window. + /// `stats_lock` is how long reading that state took, lock wait included. + fn log_and_reset(&mut self, stats: &FetchDispatchStats, stats_lock: Duration) { + let attempt_mean_ms = u32::try_from(self.timed_attempts) + .ok() + .filter(|count| *count > 0) + .map_or(0, |count| (self.attempt_total / count).as_millis()); + info!( + per_source_cap = FETCH_MAX_IN_FLIGHT_PER_SOURCE, + window_secs = self.age().as_secs(), + parked = stats.parked, + parked_associations = stats.parked_associations, + oldest_parked_secs = stats.oldest_parked.map_or(0, |age| age.as_secs()), + queued = stats.queued, + in_flight = stats.in_flight, + saturated_sources = stats.saturated_sources, + fills = self.fills, + parks = self.parks, + retry_parks = self.retry_parks, + stored = self.stored, + stored_bytes = self.stored_bytes, + already_held = self.already_held, + no_longer_responsible = self.no_longer_responsible, + source_failed = self.source_failed, + integrity_failed = self.integrity_failed, + local_write_failed = self.local_write_failed, + panicked = self.panicked, + cancelled = self.cancelled, + shutdown = self.shutdown, + attempt_max_ms = self.attempt_max.as_millis(), + attempt_mean_ms, + fill_lock_wait_max_ms = self.fill_lock_wait_max.as_millis(), + fill_lock_hold_max_ms = self.fill_lock_hold_max.as_millis(), + parked_maintenance_max_ms = stats.parked_maintenance_max.as_millis(), + tracked_sources = stats.tracked_sources, + silent_sources = stats.silent_sources, + health_overflow_active = stats.overflow_active, + silent_peak = stats.health.silent_peak, + silent_starts = stats.health.silent_starts, + unreachable = stats.health.unreachable, + silent_marks = stats.health.marks, + silent_clears = stats.health.clears, + silent_expired = stats.health.expired_marks, + silent_ignored_stale = stats.health.ignored_stale, + health_overflows = stats.health.overflows, + stats_lock_ms = stats_lock.as_millis(), + "replication fetch dispatch summary" + ); + *self = Self::new(); + } } /// What the fetch worker must do next for a key whose fetch attempt resolved. @@ -8993,6 +9341,11 @@ enum FetchFollowUp { RequeuedForVerification, /// Retry immediately from this next untried verified source. RetryFrom(PeerId), + /// Every untried source already has this node's per-source share in + /// flight, so the key went back to the fetch queue, parked, keeping its + /// tried sources and retry reservation. Not terminal: the key is still in + /// the pipeline, and the next fill pass starts it once a share frees. + Parked, } /// Apply a resolved [`FetchResult`] to the queues and report the follow-up. @@ -9013,19 +9366,26 @@ fn apply_fetch_result( key: &XorName, result: &FetchResult, verification_retry_after: Duration, + distance: XorName, ) -> FetchFollowUp { match result { - FetchResult::Stored | FetchResult::NoLongerResponsible | FetchResult::AlreadyHeld => { + FetchResult::Stored { .. } + | FetchResult::NoLongerResponsible + | FetchResult::AlreadyHeld => { q.complete_fetch(key); FetchFollowUp::Terminal } FetchResult::IntegrityFailed | FetchResult::SourceFailed => { - if let Some(next_peer) = q.retry_fetch(key) { - FetchFollowUp::RetryFrom(next_peer) - } else if q.requeue_fetch_for_verification(key, verification_retry_after) { - FetchFollowUp::RequeuedForVerification - } else { - FetchFollowUp::Terminal + match q.retry_or_park_fetch(key, distance, FETCH_MAX_IN_FLIGHT_PER_SOURCE) { + FetchRetry::Next(next_peer) => FetchFollowUp::RetryFrom(next_peer), + FetchRetry::Parked => FetchFollowUp::Parked, + FetchRetry::Exhausted => { + if q.requeue_fetch_for_verification(key, verification_retry_after) { + FetchFollowUp::RequeuedForVerification + } else { + FetchFollowUp::Terminal + } + } } } // No `retry_fetch`: the local write is what failed, so every other @@ -9072,7 +9432,6 @@ async fn is_storage_admitted( .await } -#[allow(clippy::too_many_lines)] /// Execute a single fetch request against `source` for `key`. /// /// Handles encoding, network I/O, integrity checking, storage, and trust @@ -9092,17 +9451,64 @@ async fn execute_single_fetch( config: Arc, key: XorName, source: PeerId, + answers: mpsc::UnboundedSender<(PeerId, Contact)>, ) -> FetchOutcome { + let mut contact = Contact::NotAttempted; + let result = fetch_attempt( + p2p_node, + storage, + config, + key, + source, + &mut contact, + &answers, + ) + .await; + FetchOutcome { + key, + source, + result, + contact, + } +} + +/// Record every answer a fetch has reported since the last call. +/// +/// A fetch reports its source's answer as soon as it arrives, before checking +/// and storing the chunk, so the per-source cap never treats an answering +/// source as silent while that work is still under way. The same answer comes +/// again with the fetch's outcome, which records it a second time to no +/// effect. Each is reported while its fetch is in flight, and the worker calls +/// this before applying any outcome, so it is recorded before its fetch ends. +fn record_early_answers( + queues: &mut ReplicationQueues, + early_answers: &mut mpsc::UnboundedReceiver<(PeerId, Contact)>, +) { + while let Ok((source, contact)) = early_answers.try_recv() { + queues.record_fetch_contact(source, contact); + } +} + +/// The body of [`execute_single_fetch`]: the attempt's result, with `contact` +/// set to whether the source answered, stamped when the request returned. An +/// answer is also sent on `answers` at once. +#[allow(clippy::too_many_lines)] +async fn fetch_attempt( + p2p_node: Arc, + storage: Arc, + config: Arc, + key: XorName, + source: PeerId, + contact: &mut Contact, + answers: &mpsc::UnboundedSender<(PeerId, Contact)>, +) -> FetchResult { let self_id = *p2p_node.peer_id(); if !is_storage_admitted(&self_id, &key, &p2p_node, &config).await { debug!( "Skipping fetch for {}: no longer in the storage-admission group", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::NoLongerResponsible, - }; + return FetchResult::NoLongerResponsible; } // Possession, then capacity — both before the dial, and in that order. @@ -9116,10 +9522,7 @@ async fn execute_single_fetch( "Skipping fetch for {}: already held locally", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::AlreadyHeld, - }; + return FetchResult::AlreadyHeld; } // Capacity is checked before the dial rather than after the bytes arrive. @@ -9141,10 +9544,7 @@ async fn execute_single_fetch( "Skipping fetch for {}: local storage cannot accept a write: {e}", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::LocalWriteFailed, - }; + return FetchResult::LocalWriteFailed; } let request = protocol::FetchRequest { key }; @@ -9157,13 +9557,11 @@ async fn execute_single_fetch( Ok(data) => data, Err(e) => { warn!("Failed to encode fetch request: {e}"); - return FetchOutcome { - key, - result: FetchResult::SourceFailed, - }; + return FetchResult::SourceFailed; } }; + let sent = next_fetch_contact(); let result = p2p_node .send_request( &source, @@ -9172,6 +9570,18 @@ async fn execute_single_fetch( config.fetch_request_timeout, ) .await; + *contact = if result.is_ok() { + Contact::Answered { + order: next_fetch_contact(), + } + } else { + Contact::Unreachable { sent } + }; + if matches!(contact, Contact::Answered { .. }) { + // Fails only once the fetch worker has stopped, when no scheduling + // decision is left to inform. + let _ = answers.send((source, *contact)); + } match result { Ok(response) => { @@ -9182,10 +9592,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::SourceFailed, - }; + return FetchResult::SourceFailed; }; match resp_msg.body { @@ -9209,10 +9616,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::IntegrityFailed, - }; + return FetchResult::IntegrityFailed; } // Enforce chunk size invariant on fetched data. @@ -9231,10 +9635,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::IntegrityFailed, - }; + return FetchResult::IntegrityFailed; } // Content-address integrity check. @@ -9251,10 +9652,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::IntegrityFailed, - }; + return FetchResult::IntegrityFailed; } // Responsibility can lapse during the network round-trip. @@ -9268,10 +9666,7 @@ async fn execute_single_fetch( "Fetched {} but responsibility lapsed in transit; not storing", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::NoLongerResponsible, - }; + return FetchResult::NoLongerResponsible; } if let Err(e) = storage.put(&resp_key, &data).await { @@ -9285,16 +9680,10 @@ async fn execute_single_fetch( "Failed to store fetched record {}: {e}", hex::encode(resp_key) ); - return FetchOutcome { - key, - result: FetchResult::LocalWriteFailed, - }; + return FetchResult::LocalWriteFailed; } - FetchOutcome { - key, - result: FetchResult::Stored, - } + FetchResult::Stored { bytes: data.len() } } ReplicationMessageBody::FetchResponse( ref response @ (protocol::FetchResponse::NotFound { .. } @@ -9320,10 +9709,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - FetchOutcome { - key, - result: FetchResult::SourceFailed, - } + FetchResult::SourceFailed } _ => { // Unexpected message type — treat as malformed. @@ -9333,10 +9719,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - FetchOutcome { - key, - result: FetchResult::SourceFailed, - } + FetchResult::SourceFailed } } } @@ -9344,10 +9727,7 @@ async fn execute_single_fetch( debug!("Fetch request to {source} failed: {e}"); // No ApplicationFailure here — P2PNode::send_request() already // reports ConnectionTimeout / ConnectionFailed to the TrustEngine. - FetchOutcome { - key, - result: FetchResult::SourceFailed, - } + FetchResult::SourceFailed } } } @@ -10537,6 +10917,7 @@ mod tests { use crate::replication::audit::AuditTickResult; use crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL; use crate::replication::recent_provers::RecentProvers; + use crate::replication::scheduling::MAX_FETCH_QUEUE; use crate::replication::types::{AuditFailureReason, FailureEvidence}; use lru::LruCache; use saorsa_core::identity::PeerId; @@ -13779,8 +14160,13 @@ mod tests { drive_key_in_flight(&mut q, key, vec![test_peer(0x01), test_peer(0x02)]); assert_eq!(q.retry_reserved_slot_count(), 1); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::NoLongerResponsible, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::NoLongerResponsible, + RETRY_AFTER, + key, + ); assert_eq!( follow_up, @@ -13810,12 +14196,15 @@ mod tests { // Both variants must walk the identical terminal gate so the // battle-tested Stored accounting (retry-slot release + bootstrap // pending-set shrink in the caller) covers lapsed responsibility too. - for result in [FetchResult::Stored, FetchResult::NoLongerResponsible] { + for result in [ + FetchResult::Stored { bytes: 0 }, + FetchResult::NoLongerResponsible, + ] { let mut q = ReplicationQueues::new(); let key = test_key(0xCD); drive_key_in_flight(&mut q, key, vec![test_peer(0x03)]); - let follow_up = apply_fetch_result(&mut q, &key, &result, RETRY_AFTER); + let follow_up = apply_fetch_result(&mut q, &key, &result, RETRY_AFTER, key); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); @@ -13833,7 +14222,8 @@ mod tests { let second = test_peer(0x05); drive_key_in_flight(&mut q, key, vec![first, second]); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); assert_eq!( follow_up, FetchFollowUp::RetryFrom(second), @@ -13841,7 +14231,8 @@ mod tests { ); assert_eq!(q.in_flight_count(), 1, "retry keeps the key in flight"); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); assert_eq!( follow_up, FetchFollowUp::RequeuedForVerification, @@ -13855,6 +14246,199 @@ mod tests { ); } + /// Put two unrelated fetches in flight to `source`, filling its share. + fn saturate_source(q: &mut ReplicationQueues, source: PeerId, first_key_byte: u8) { + for offset in 0..2u8 { + let key = test_key(first_key_byte + offset); + q.start_fetch(key, source, vec![source]); + } + } + + #[test] + fn source_failure_with_every_untried_source_full_parks_the_key() { + const RETRY_AFTER: Duration = Duration::from_mins(1); + + let mut q = ReplicationQueues::new(); + let key = test_key(0x2A); + let (first, second) = (test_peer(0x01), test_peer(0x02)); + assert_eq!(drive_key_in_flight(&mut q, key, vec![first, second]), first); + saturate_source(&mut q, second, 0xE0); + let reserved = q.retry_reserved_slot_count(); + + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); + + assert_eq!( + follow_up, + FetchFollowUp::Parked, + "the only untried source is at its share, so the key must wait rather \ + than be sent a request it will drop" + ); + assert!( + q.contains_key(&key), + "a parked key is still in the pipeline" + ); + assert!(q.fetch_queued_or_in_flight(&key)); + assert_eq!( + q.in_flight_count(), + 2, + "only the two saturating fetches remain" + ); + assert_eq!( + q.pending_count(), + 0, + "parking is not a verification requeue" + ); + assert_eq!(q.retry_reserved_slot_count(), reserved); + } + + #[test] + fn an_answer_reported_before_its_fetch_ends_keeps_a_timed_out_source_capped() { + const RETRY_AFTER: Duration = Duration::from_mins(1); + const SLOTS: usize = 8; + + let source = test_peer(0x0A); + let (dropped, answered) = (test_key(0x50), test_key(0x51)); + let waiting: Vec = (0x60..0x64).map(test_key).collect(); + // Both of `source`'s shares are in flight and four keys only it holds + // wait behind them. + let setup = || { + let mut q = ReplicationQueues::new(); + q.start_fetch(dropped, source, vec![source]); + q.start_fetch(answered, source, vec![source]); + for key in &waiting { + assert!(q.enqueue_fetch(*key, *key, vec![source])); + } + q + }; + let in_flight_to_source = |q: &ReplicationQueues| { + q.fetch_pipeline_snapshot() + .in_flight + .iter() + .filter(|(_, peer)| *peer == source) + .count() + }; + // The first request is dropped and times out. The second, sent after + // it, has been answered and is still checking and storing its chunk. + let dropped_sent = next_fetch_contact(); + let _answered_sent = next_fetch_contact(); + let answer = Contact::Answered { + order: next_fetch_contact(), + }; + let timeout = Contact::Unreachable { sent: dropped_sent }; + + // Known only from the second fetch's outcome, the answer comes too + // late: the timeout marks `source` silent and the next pass sends it + // every waiting key, past its share. + let mut late = setup(); + late.record_fetch_contact(source, timeout); + apply_fetch_result( + &mut late, + &dropped, + &FetchResult::SourceFailed, + RETRY_AFTER, + dropped, + ); + late.dispatch_fetches(SLOTS, FETCH_MAX_IN_FLIGHT_PER_SOURCE); + assert_eq!(in_flight_to_source(&late), 1 + waiting.len()); + + // Reported when it arrives, the answer is recorded before the timeout, + // which it outdates, so `source` stays within its share. + let (answers, mut early_answers) = mpsc::unbounded_channel(); + let mut q = setup(); + assert!(answers.send((source, answer)).is_ok()); + record_early_answers(&mut q, &mut early_answers); + q.record_fetch_contact(source, timeout); + apply_fetch_result( + &mut q, + &dropped, + &FetchResult::SourceFailed, + RETRY_AFTER, + dropped, + ); + record_early_answers(&mut q, &mut early_answers); + let dispatch = q.dispatch_fetches(SLOTS, FETCH_MAX_IN_FLIGHT_PER_SOURCE); + assert_eq!(dispatch.started.len(), 1, "one share is free"); + assert_eq!(in_flight_to_source(&q), FETCH_MAX_IN_FLIGHT_PER_SOURCE); + + // The same answer again, with the second fetch's outcome, changes + // nothing. + q.record_fetch_contact(source, answer); + assert_eq!( + q.dispatch_fetches(SLOTS, FETCH_MAX_IN_FLIGHT_PER_SOURCE) + .started + .len(), + 0 + ); + assert_eq!(in_flight_to_source(&q), FETCH_MAX_IN_FLIGHT_PER_SOURCE); + } + + #[test] + fn the_dispatch_window_times_resolved_attempts_and_counts_cancelled_ones() { + let (key, source) = (test_key(0x31), test_peer(0x01)); + let outcome = |result: FetchResult, contact: Contact| FetchOutcome { + key, + source, + result, + contact, + }; + let started = Instant::now(); + let at = |millis: u64| started + Duration::from_millis(millis); + let mut window = FetchDispatchWindow::new(); + + // Two attempts resolve at different times; the worker reaches both + // later, which must not count. + let stored = outcome( + FetchResult::Stored { bytes: 10 }, + Contact::Answered { order: 2 }, + ); + window.record_outcome(Some(&stored), attempt_latency(Some(started), at(100))); + let failed = outcome(FetchResult::SourceFailed, Contact::Unreachable { sent: 1 }); + window.record_outcome(Some(&failed), attempt_latency(Some(started), at(300))); + // A cancelled attempt is counted as cancelled, not by its placeholder + // result, and its cut-short time is left out. + let cancelled = outcome(FetchResult::SourceFailed, Contact::Unknown); + window.record_outcome(Some(&cancelled), attempt_latency(Some(started), at(5_000))); + + assert_eq!((window.stored, window.source_failed), (1, 1)); + assert_eq!((window.cancelled, window.timed_attempts), (1, 2)); + assert_eq!(window.attempt_max, Duration::from_millis(300)); + assert_eq!(window.attempt_total, Duration::from_millis(400)); + assert_eq!(window.stored_bytes, 10); + assert_eq!(attempt_latency(None, at(100)), None); + } + + #[test] + fn source_failure_with_a_full_fetch_queue_requeues_for_verification() { + const RETRY_AFTER: Duration = Duration::from_mins(1); + + let mut q = ReplicationQueues::new(); + let key = test_key(0x2B); + let (first, second) = (test_peer(0x01), test_peer(0x02)); + drive_key_in_flight(&mut q, key, vec![first, second]); + saturate_source(&mut q, second, 0xE0); + let filler_source = test_peer(0x09); + let mut filler = 0u32; + while q.fetch_queue_count() < MAX_FETCH_QUEUE { + let mut filler_key = [0xC0u8; 32]; + filler_key[..4].copy_from_slice(&filler.to_be_bytes()); + assert!(q.enqueue_fetch(filler_key, filler_key, vec![filler_source])); + filler += 1; + } + + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); + + assert_eq!( + follow_up, + FetchFollowUp::RequeuedForVerification, + "with no room to park, the key returns to verification on its own \ + reserved slot instead of overfilling the fetch queue" + ); + assert_eq!(q.fetch_queue_count(), MAX_FETCH_QUEUE); + assert_eq!(q.pending_count(), 1); + } + #[test] fn source_failure_without_retry_metadata_is_terminal() { const RETRY_AFTER: Duration = Duration::from_mins(1); @@ -13868,7 +14452,8 @@ mod tests { let candidate = q.dequeue_fetch().expect("enqueued key must dequeue"); q.start_dequeued_fetch(candidate, source); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); @@ -13883,7 +14468,8 @@ mod tests { let source = test_peer(0x0B); drive_key_in_flight(&mut q, key, vec![source, test_peer(0x0C)]); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::AlreadyHeld, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::AlreadyHeld, RETRY_AFTER, key); assert_eq!( follow_up, @@ -13912,8 +14498,13 @@ mod tests { let third = test_peer(0x09); drive_key_in_flight(&mut q, key, vec![first, second, third]); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::LocalWriteFailed, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::LocalWriteFailed, + RETRY_AFTER, + key, + ); assert_eq!( follow_up, @@ -13946,8 +14537,13 @@ mod tests { let candidate = q.dequeue_fetch().expect("enqueued key must dequeue"); q.start_dequeued_fetch(candidate, source); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::LocalWriteFailed, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::LocalWriteFailed, + RETRY_AFTER, + key, + ); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); diff --git a/src/replication/scheduling.rs b/src/replication/scheduling.rs index e296606e..95c6b168 100644 --- a/src/replication/scheduling.rs +++ b/src/replication/scheduling.rs @@ -5,13 +5,15 @@ //! time, with strict dedup across all three stages. use std::cmp::Ordering; -use std::collections::{BTreeMap, BinaryHeap, HashMap, HashSet, VecDeque}; +use std::collections::{BTreeMap, BTreeSet, BinaryHeap, HashMap, HashSet, VecDeque}; use std::time::{Duration, Instant}; use crate::logging::debug; -use crate::ant_protocol::XorName; -use crate::replication::config::VERIFICATION_RETRY_BACKOFF_MAX; +use crate::ant_protocol::{XorName, CLOSE_GROUP_SIZE}; +use crate::replication::config::{ + FETCH_REQUEST_TIMEOUT, PAID_LIST_CLOSE_GROUP_SIZE, VERIFICATION_RETRY_BACKOFF_MAX, +}; use crate::replication::types::{ FetchCandidate, FetchOrder, FetchPayload, VerificationEntry, VerificationState, }; @@ -94,6 +96,32 @@ pub const MAX_FETCH_QUEUE: usize = 131_072; /// decides *which* sources survive the cap. pub const MAX_HINT_SOURCES_PER_KEY: usize = 8; +/// Maximum distinct source peers one key may be associated with while it is +/// queued or in flight for fetch, counting the sources it was already fetched +/// from this cycle, for the given close-group widths. +/// +/// A verification round names at most its quorum targets and its paid-list +/// targets as holders, and the replica-hint fallbacks added to them come from a +/// pending entry already bounded by [`MAX_HINT_SOURCES_PER_KEY`], so this never +/// truncates a source list a round produced. It bounds what later +/// advertisements can merge into a queued or in-flight key, and with it the +/// parked-fetch index, which holds one entry per untried source of each parked +/// key. +#[must_use] +pub(crate) const fn fetch_source_limit( + close_group_size: usize, + paid_list_close_group_size: usize, +) -> usize { + close_group_size + .saturating_add(paid_list_close_group_size) + .saturating_add(MAX_HINT_SOURCES_PER_KEY) +} + +/// [`fetch_source_limit`] at the production close-group widths, and the limit +/// [`ReplicationQueues::new`] starts with. +pub(crate) const MAX_FETCH_SOURCES_PER_KEY: usize = + fetch_source_limit(CLOSE_GROUP_SIZE, PAID_LIST_CLOSE_GROUP_SIZE); + /// Outcome of [`ReplicationQueues::add_pending_verify`]. /// /// Distinguishes "the key is already being handled" from "the key was @@ -174,9 +202,13 @@ pub struct InFlightEntry { pub key: XorName, /// The peer we are currently fetching from. pub source: PeerId, - /// When the fetch started. + /// When the current attempt started: the first, or under the per-source + /// scheduler the latest retry. pub started_at: Instant, - /// All verified sources for this key. + /// Verified sources for this key not yet tried before this episode's + /// current attempt. A key that was parked between attempts keeps the + /// sources it already tried only in `tried`, so everything the key is + /// associated with is the union of the two. pub all_sources: Vec, /// Sources already attempted (failed or in progress). pub tried: HashSet, @@ -184,6 +216,409 @@ pub struct InFlightEntry { pub retry_verification: Option, } +// --------------------------------------------------------------------------- +// Per-source fetch dispatch +// --------------------------------------------------------------------------- + +/// A queued key none of whose untried sources had a free share when it was +/// last considered for dispatch. +/// +/// Its payload stays in `fetch_payloads`; only its place in line moves here +/// from the fetch heap. +#[derive(Debug, Clone)] +struct ParkedFetch { + /// XOR distance from self to the key, its priority once startable again. + distance: XorName, + /// Untried sources whose next free share could start the key, in the + /// payload's source-preference order. Each one indexes the key in + /// `parked_by_source`. + wake_sources: Vec, + /// When the key was parked, for the dispatch summary's oldest-parked age. + parked_at: Instant, +} + +/// Fetches started, and candidates found with no untried source, by one +/// [`ReplicationQueues::dispatch_fetches`] pass. +#[derive(Debug, Default)] +pub(crate) struct FetchDispatch { + /// Each started key with the source it was sent to. Already in flight. + pub(crate) started: Vec<(XorName, PeerId)>, + /// Dequeued candidates with no untried source left. The caller decides + /// what to do with them, exactly as for a sourceless dequeue. + pub(crate) sourceless: Vec, + /// Heap keys this pass parked because every untried source was full. + pub(crate) parked: usize, +} + +/// The per-source scheduler's queue state, for its periodic summary. +/// +/// Read only by that summary's log line, which a build without logging +/// compiles out. +#[derive(Debug, Clone, Copy)] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub(crate) struct FetchDispatchStats { + /// Keys parked behind full sources. + pub(crate) parked: usize, + /// How long the longest-parked key has waited, if any is parked. + pub(crate) oldest_parked: Option, + /// Keys queued for fetch, parked ones included. + pub(crate) queued: usize, + /// Fetches in flight. + pub(crate) in_flight: usize, + /// Sources holding at least the per-source cap of this node's fetches. + pub(crate) saturated_sources: usize, + /// Entries in the parked index: one per untried source of each parked key. + pub(crate) parked_associations: usize, + /// Longest parked-index maintenance after a departed source, since the last + /// read. + pub(crate) parked_maintenance_max: Duration, + /// Sources in the health table before the summary's purge. + pub(crate) tracked_sources: usize, + /// Tracked sources silent now. + pub(crate) silent_sources: usize, + /// Whether the health table is overflowing, so every untracked source + /// also counts as silent. + pub(crate) overflow_active: bool, + /// Source-health events since the last read. + pub(crate) health: SourceHealthCounters, +} + +/// What [`ReplicationQueues::retry_or_park_fetch`] did with a failed fetch. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum FetchRetry { + /// Retry now from this untried source, which has a free share. + Next(PeerId), + /// Every untried source is at its share: the key went back to the fetch + /// queue, parked, keeping its tried sources and retry reservation. + Parked, + /// No untried source is left, or the fetch queue is full. The key stays + /// in flight, with its failed source now marked tried, for the caller's + /// verification requeue. + Exhausted, +} + +// --------------------------------------------------------------------------- +// Fetch source health +// --------------------------------------------------------------------------- + +/// How many fetch timeouts a source stays silent after this node last learned +/// of an unanswered request to it. +const SILENCE_LIFETIME_TIMEOUTS: u32 = 2; + +/// Shortest time a mark lasts, whatever the fetch timeout, so a zero or +/// tiny configured timeout cannot turn silent sources off. +const MIN_SILENCE_LIFETIME: Duration = Duration::from_secs(1); + +/// Most fetch sources whose answering this node tracks at once. +/// +/// Far above the sources one node fetches from within a silence lifetime. At +/// the bound, entries that neither hold a mark nor guard an in-flight fetch +/// are dropped. If that frees nothing, a source not in the table counts as +/// silent for one silence lifetime: fetched from as it was before the +/// per-source cap, never more slowly. +const MAX_TRACKED_FETCH_SOURCES: usize = 8_192; + +/// Whether a fetch attempt reached its source. +/// +/// Requests and answers carry their place in one process-wide sequence, taken +/// as the request is sent and as its answer returns, so the table can tell +/// which came first however the fetch worker orders their outcomes and +/// whatever the clock reads. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum Contact { + /// The attempt ended before a request was sent. + NotAttempted, + /// The source answered, whatever the answer then turned out to be. + Answered { + /// Where the answer's arrival falls in the sequence. + order: u64, + }, + /// The request got no answer: a timeout or a connection failure. + Unreachable { + /// Where the request's sending falls in the sequence. + sent: u64, + }, + /// The attempt was cancelled at shutdown, so nothing is known. + Unknown, +} + +/// What this node has observed of one source's answering. +#[derive(Debug, Clone, Copy, Default)] +struct SourceHealth { + /// Sequence place of the latest answer from this source. + last_answer: Option, + /// Sequence place of the latest-sent request to this source that got no + /// answer. + last_unanswered: Option, + /// When this node learned of that request's failure, while the request is + /// newer than the latest answer and the mark has not expired. + marked_since: Option, +} + +impl SourceHealth { + /// Whether a request sent after this source's latest answer went + /// unanswered. It depends only on the two latest events, so it is the same + /// whatever order the outcomes were applied in. + fn unanswered_since_answer(&self) -> bool { + self.last_unanswered + .is_some_and(|sent| self.last_answer.is_none_or(|answer| sent > answer)) + } +} + +/// Source-health events since the dispatch summary last read them. +/// +/// Read only by that summary's log line, which a build without logging +/// compiles out. +#[derive(Debug, Clone, Copy, Default)] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub(crate) struct SourceHealthCounters { + /// Attempts whose request got no answer. + pub(crate) unreachable: u64, + /// Sources that became silent. + pub(crate) marks: u64, + /// Marks cleared by an answer. + pub(crate) clears: u64, + /// Marks that ran out their lifetime. + pub(crate) expired_marks: u64, + /// Unanswered requests sent before their source's latest answer. + pub(crate) ignored_stale: u64, + /// Fetches started or retried on a silent source, outside the cap. + pub(crate) silent_starts: u64, + /// Times the table was full of marks and in-flight guards. + pub(crate) overflows: u64, + /// Most tracked sources silent at once. + pub(crate) silent_peak: usize, +} + +/// What this node has observed of each fetch source's answering. +/// +/// The per-source cap keeps this node within the admission share a responder +/// grants it, and only a responder that answers has a share to keep within. A +/// source is silent while the latest-sent request to it that went unanswered +/// was sent after its latest answer: it is fetched from without the cap, as +/// before the cap existed. Any answer received after that request was sent +/// ends it. So does learning of no newer unanswered request for one silence +/// lifetime, so a source this node has stopped trying is probed again under +/// the cap. +#[derive(Debug)] +struct SourceHealthTable { + entries: HashMap, + /// Every mark as `(marked_since, source)`, so marks expire in order. + marks_by_since: BTreeSet<(Instant, PeerId)>, + /// How long a mark lasts without newer evidence. + silence_lifetime: Duration, + /// When the table last overflowed. For one silence lifetime after, a + /// source not in the table counts as silent. + overflow_since: Option, + counters: SourceHealthCounters, +} + +impl SourceHealthTable { + fn new(fetch_request_timeout: Duration) -> Self { + Self { + entries: HashMap::new(), + marks_by_since: BTreeSet::new(), + silence_lifetime: fetch_request_timeout + .saturating_mul(SILENCE_LIFETIME_TIMEOUTS) + .max(MIN_SILENCE_LIFETIME), + overflow_since: None, + counters: SourceHealthCounters::default(), + } + } + + /// Whether something that began at `since` is still within one silence + /// lifetime at `now`. + fn within_lifetime(&self, since: Instant, now: Instant) -> bool { + now.saturating_duration_since(since) < self.silence_lifetime + } + + fn overflowing(&self, now: Instant) -> bool { + self.overflow_since + .is_some_and(|since| self.within_lifetime(since, now)) + } + + /// Whether `source` counts as silent at `now`. + fn is_silent(&self, source: &PeerId, now: Instant) -> bool { + self.entries.get(source).map_or_else( + || self.overflowing(now), + |health| { + health + .marked_since + .is_some_and(|since| self.within_lifetime(since, now)) + }, + ) + } + + /// Drop the marks that ran out their lifetime by `now`. + fn expire(&mut self, now: Instant) { + while let Some(&(since, source)) = self.marks_by_since.first() { + if self.within_lifetime(since, now) { + break; + } + self.marks_by_since.pop_first(); + if let Some(health) = self.entries.get_mut(&source) { + if health.marked_since == Some(since) { + health.marked_since = None; + } + } + self.counters.expired_marks += 1; + } + } + + /// Drop every entry that neither holds a mark nor guards an in-flight + /// fetch to its source. Every contact, an answer reported ahead of its + /// fetch's outcome included, is applied while its own fetch is still in + /// flight, so none can arrive for a dropped source's past requests: + /// anything later is newer than all of its history. Call after + /// [`Self::expire`]. + fn purge(&mut self, in_flight: &HashMap) { + let guarded: HashSet = in_flight.values().map(|entry| entry.source).collect(); + self.entries + .retain(|source, health| health.marked_since.is_some() || guarded.contains(source)); + } + + /// The entry for `source`, purging the table at the bound. `None` when the + /// table cannot take it, which starts an overflow. + fn entry( + &mut self, + source: PeerId, + now: Instant, + in_flight: &HashMap, + ) -> Option<&mut SourceHealth> { + if !self.entries.contains_key(&source) && self.entries.len() >= MAX_TRACKED_FETCH_SOURCES { + if self.overflowing(now) { + return None; + } + self.purge(in_flight); + if self.entries.len() >= MAX_TRACKED_FETCH_SOURCES { + self.overflow_since = Some(now); + self.counters.overflows += 1; + return None; + } + } + Some(self.entries.entry(source).or_default()) + } + + /// Apply, at `now`, one attempt's contact with `source`. + fn record( + &mut self, + source: PeerId, + contact: Contact, + now: Instant, + in_flight: &HashMap, + ) { + self.expire(now); + match contact { + Contact::NotAttempted | Contact::Unknown => {} + Contact::Answered { order } => { + let Some(health) = self.entry(source, now, in_flight) else { + return; + }; + health.last_answer = health.last_answer.max(Some(order)); + let cleared = if health.unanswered_since_answer() { + None + } else { + health.marked_since.take() + }; + if let Some(since) = cleared { + self.marks_by_since.remove(&(since, source)); + self.counters.clears += 1; + } + } + Contact::Unreachable { sent } => { + self.counters.unreachable += 1; + let Some(health) = self.entry(source, now, in_flight) else { + return; + }; + let stale = health.last_answer.is_some_and(|answer| sent < answer); + let newest = health.last_unanswered.is_none_or(|latest| sent > latest); + if newest { + health.last_unanswered = Some(sent); + } + // The newest unanswered request, if newer than every answer, + // marks the source from now: from when this node learned of it. + let (replaced, marked) = if newest && health.unanswered_since_answer() { + (health.marked_since.replace(now), true) + } else { + (None, false) + }; + if stale { + self.counters.ignored_stale += 1; + } + if marked { + if let Some(since) = replaced { + self.marks_by_since.remove(&(since, source)); + } else { + self.counters.marks += 1; + } + self.marks_by_since.insert((now, source)); + self.counters.silent_peak = + self.counters.silent_peak.max(self.marks_by_since.len()); + } + } + } + } + + /// The counters since the last call, and the table's size before the + /// purge that follows. + fn take_counters( + &mut self, + now: Instant, + in_flight: &HashMap, + ) -> (SourceHealthCounters, usize) { + self.expire(now); + let tracked = self.entries.len(); + self.purge(in_flight); + let counters = std::mem::take(&mut self.counters); + self.counters.silent_peak = self.marks_by_since.len(); + (counters, tracked) + } +} + +/// What lets a parked key start now: an untried source below the per-source +/// cap, or a silent one. +#[derive(Clone, Copy)] +struct Startable<'a> { + in_flight: &'a HashMap, + per_source_cap: usize, + now: Instant, +} + +/// The next queued key, taken off the heap or out of the parked set. +/// +/// Returned and matched at once, never stored, so the size gap between the +/// variants costs nothing; boxing would add an allocation per dispatched fetch. +#[allow(clippy::large_enum_variant)] +enum TakenFetch { + /// The key, rejoined with its payload. + Candidate(FetchCandidate), + /// An inconsistent entry was dropped instead; try again. + Skipped, + /// The heap is empty and no parked key was asked for. + Empty, +} + +/// How [`ReplicationQueues::dispatch_fetches`] may start one candidate. +enum SourceChoice { + /// This untried source has a free share. + Start(PeerId), + /// Untried sources exist, but every one is at its share. + AllBusy, + /// No untried source is left. + NoSource, +} + +/// Test-only: every key queued or parked for fetch, and every in-flight fetch +/// with its source, read under one lock. +#[cfg(any(test, feature = "test-utils"))] +#[derive(Debug, Clone, Default)] +pub struct FetchPipelineSnapshot { + /// Keys queued for fetch, on the heap or parked. + pub queued: Vec, + /// In-flight fetches with the source each one was sent to. + pub in_flight: Vec<(XorName, PeerId)>, +} + // --------------------------------------------------------------------------- // Central queue manager // --------------------------------------------------------------------------- @@ -205,10 +640,11 @@ pub struct ReplicationQueues { /// Capacity-bounded by [`MAX_PENDING_VERIFY`]. At capacity, max-min sender /// accounting reclaims borrowed entries before rejecting new work. pending_verify: HashMap, - /// Nearest-first priority order over the keys in `fetch_payloads`. + /// Nearest-first priority order over the queued keys that are not parked. /// /// Holds ordering data only: every entry has exactly one matching - /// `fetch_payloads` entry, and vice versa. Splitting the mutable half of a + /// `fetch_payloads` entry, and every `fetch_payloads` entry is represented + /// exactly once, here or in `fetch_parked`. Splitting the mutable half of a /// queued fetch out into `fetch_payloads` keeps this heap's ordering /// immutable for as long as a key is queued, so a source merge never has /// to rebuild it. @@ -221,8 +657,30 @@ pub struct ReplicationQueues { /// Capacity-bounded by [`MAX_FETCH_QUEUE`]: enqueues are dropped once /// full, preventing unbounded growth under a network hint flood. fetch_payloads: HashMap, + /// Queued keys taken off `fetch_queue` because every untried source was at + /// its per-source share. Every key in `fetch_payloads` is in exactly one of + /// `fetch_queue` and this map. + fetch_parked: HashMap, + /// Exact reverse index over `fetch_parked`: `(distance, key)` is in the set + /// for `source` if and only if `key` is parked with `source` among its + /// wake sources. Ordered so the nearest parked key per source is `first()`; + /// empty sets are removed. + parked_by_source: HashMap>, + /// Sources a queued key was already fetched from in the current + /// verification cycle. Present only for a key that went back to the queue + /// from flight, and carried into its next in-flight entry. + fetch_tried: HashMap>, /// Active downloads keyed by `XorName`. in_flight_fetch: HashMap, + /// Most distinct sources a queued or in-flight key may be associated with, + /// tried ones included. See [`fetch_source_limit`]. + max_fetch_sources: usize, + /// Longest parked-index maintenance after a departed source since the + /// dispatch summary last read it. + parked_maintenance_max: Duration, + /// Which fetch sources stopped answering, so the per-source cap can leave + /// them out. + source_health: SourceHealthTable, /// Reverse index for removing a departed peer from every pending hint /// without scanning the entire pending table. pending_keys_by_source: HashMap>, @@ -272,7 +730,13 @@ impl ReplicationQueues { pending_verify: HashMap::new(), fetch_queue: BinaryHeap::new(), fetch_payloads: HashMap::new(), + fetch_parked: HashMap::new(), + parked_by_source: HashMap::new(), + fetch_tried: HashMap::new(), in_flight_fetch: HashMap::new(), + max_fetch_sources: MAX_FETCH_SOURCES_PER_KEY, + parked_maintenance_max: Duration::ZERO, + source_health: SourceHealthTable::new(FETCH_REQUEST_TIMEOUT), pending_keys_by_source: HashMap::new(), capacity_owner_by_key: HashMap::new(), pending_keys_by_owner: HashMap::new(), @@ -286,6 +750,39 @@ impl ReplicationQueues { } } + /// Use `limit` as the most distinct sources a queued or in-flight key may + /// be associated with, for close-group widths other than the production + /// ones. See [`fetch_source_limit`]. + #[must_use] + pub(crate) fn with_fetch_source_limit(mut self, limit: usize) -> Self { + self.set_fetch_source_limit(limit); + self + } + + /// Change the source limit. Only for queues no key has entered yet, such + /// as an engine's before it starts. + pub(crate) fn set_fetch_source_limit(&mut self, limit: usize) { + self.max_fetch_sources = limit.max(1); + } + + /// Use `timeout`, the fetch request timeout, to time how long a source + /// that stopped answering stays silent. Only for queues no fetch has + /// resolved in yet. + #[must_use] + pub(crate) fn with_fetch_request_timeout(mut self, timeout: Duration) -> Self { + self.source_health = SourceHealthTable::new(timeout); + self + } + + /// Apply whether a fetch attempt's `source` answered. Called for every + /// resolved attempt, before the attempt's result moves its key on, and for + /// an answer as soon as it arrives, while its attempt is still in flight. + /// Applying the same answer twice changes nothing. + pub(crate) fn record_fetch_contact(&mut self, source: PeerId, contact: Contact) { + self.source_health + .record(source, contact, Instant::now(), &self.in_flight_fetch); + } + // ----------------------------------------------------------------------- // PendingVerify // ----------------------------------------------------------------------- @@ -302,6 +799,7 @@ impl ReplicationQueues { /// source-side retry) MUST treat this as still-outstanding work, not as /// "done". Without this distinction a bootstrap snapshot whose hints /// are capacity-rejected would silently mark itself drained. + #[allow(clippy::too_many_lines)] pub fn add_pending_verify( &mut self, key: XorName, @@ -373,32 +871,66 @@ impl ReplicationQueues { return AdmissionResult::AlreadyPresent; } // Merging a source touches only `FetchPayload`, which no ordering - // reads, so the fetch heap is left completely untouched here. + // reads, so the fetch heap is left completely untouched here. A source + // this key was already fetched from this cycle is not merged back: it + // would be asked twice. A parked key can start from a merged source as + // soon as that source has a free share, so it is indexed there too. if let Some(payload) = self.fetch_payloads.get_mut(&key) { + let tried = self.fetch_tried.get(&key); + let mut merged = Vec::new(); for source in &entry.replica_hint_sources { - if !payload.sources.contains(source) { + // Payload sources and tried sources are disjoint, so this is + // the size of everything the key has been associated with. + if payload.sources.len() + tried.map_or(0, HashSet::len) >= self.max_fetch_sources { + break; + } + if !payload.sources.contains(source) && !tried.is_some_and(|t| t.contains(source)) { payload.sources.push(*source); + merged.push(*source); + } + } + if let Some(parked) = self.fetch_parked.get_mut(&key) { + for source in merged { + parked.wake_sources.push(source); + self.parked_by_source + .entry(source) + .or_default() + .insert((parked.distance, key)); } } if let Some(retry) = &mut payload.retry_verification { - retry - .replica_hint_sources - .extend(entry.replica_hint_sources); - retry.hint_sources.extend(entry.hint_sources); + Self::merge_retry_hints( + retry, + &entry.replica_hint_sources, + &entry.hint_sources, + self.capacity_owner_by_key.get(&key).copied(), + ); } return AdmissionResult::AlreadyPresent; } if let Some(in_flight) = self.in_flight_fetch.get_mut(&key) { + let mut associated = in_flight.all_sources.len() + + in_flight + .tried + .iter() + .filter(|peer| !in_flight.all_sources.contains(peer)) + .count(); for source in &entry.replica_hint_sources { - if !in_flight.all_sources.contains(source) { + if associated >= self.max_fetch_sources { + break; + } + if !in_flight.all_sources.contains(source) && !in_flight.tried.contains(source) { in_flight.all_sources.push(*source); + associated += 1; } } if let Some(retry) = &mut in_flight.retry_verification { - retry - .replica_hint_sources - .extend(entry.replica_hint_sources); - retry.hint_sources.extend(entry.hint_sources); + Self::merge_retry_hints( + retry, + &entry.replica_hint_sources, + &entry.hint_sources, + self.capacity_owner_by_key.get(&key).copied(), + ); } return AdmissionResult::AlreadyPresent; } @@ -652,10 +1184,7 @@ impl ReplicationQueues { key: &XorName, protected_owner: Option, ) -> bool { - let victim = entry.hint_sources.iter().copied().find(|source| { - !entry.replica_hint_sources.contains(source) && protected_owner != Some(*source) - }); - let Some(victim) = victim else { + let Some(victim) = Self::paid_only_victim(entry, protected_owner) else { return false; }; entry.hint_sources.remove(&victim); @@ -668,6 +1197,53 @@ impl ReplicationQueues { true } + /// The paid-only advertiser [`Self::displace_paid_only_source`] would drop + /// from `entry`: never a replica claimant, never `protected_owner`. + fn paid_only_victim( + entry: &VerificationEntry, + protected_owner: Option, + ) -> Option { + entry.hint_sources.iter().copied().find(|source| { + !entry.replica_hint_sources.contains(source) && protected_owner != Some(*source) + }) + } + + /// Merge a re-advertisement into the verification entry a fetch-stage key + /// keeps for a later retry, under the bound and in the order + /// [`Self::add_pending_verify`] applies to a pending entry: replica + /// claimants first, displacing paid-only advertisers other than + /// `protected_owner` when the set is full, then paid-only advertisers + /// while it is below [`MAX_HINT_SOURCES_PER_KEY`]. A fetch-stage key is in + /// no pending index, so only the entry changes. + fn merge_retry_hints( + retry: &mut VerificationEntry, + replica_hint_sources: &HashSet, + hint_sources: &HashSet, + protected_owner: Option, + ) { + for source in replica_hint_sources { + if retry.replica_hint_sources.contains(source) { + continue; + } + if !retry.hint_sources.contains(source) + && retry.hint_sources.len() >= MAX_HINT_SOURCES_PER_KEY + { + let Some(victim) = Self::paid_only_victim(retry, protected_owner) else { + break; + }; + retry.hint_sources.remove(&victim); + } + retry.replica_hint_sources.insert(*source); + retry.hint_sources.insert(*source); + } + for source in hint_sources { + if retry.hint_sources.len() >= MAX_HINT_SOURCES_PER_KEY { + break; + } + retry.hint_sources.insert(*source); + } + } + /// Reserve a pending slot for `key`, charged to its capacity owner. /// /// Must be called while `capacity_owner_by_key` still holds `key`, which is @@ -951,8 +1527,9 @@ impl ReplicationQueues { .retain(|key, _| !orphaned_fetch_keys.contains(key)); self.fetch_queue .retain(|order| !orphaned_fetch_keys.contains(&order.key)); - orphaned.extend(orphaned_fetch_keys); } + self.drop_departed_source_from_parked(source, &orphaned_fetch_keys); + orphaned.extend(orphaned_fetch_keys); for entry in self.in_flight_fetch.values_mut() { entry.all_sources.retain(|peer| peer != source); @@ -1114,7 +1691,7 @@ impl ReplicationQueues { &mut self, key: XorName, distance: XorName, - sources: Vec, + mut sources: Vec, retry_verification: Option, ) -> bool { if self.pending_verify.contains_key(&key) @@ -1130,6 +1707,7 @@ impl ReplicationQueues { ); return false; } + sources.truncate(self.max_fetch_sources); self.fetch_payloads.insert( key, FetchPayload { @@ -1138,6 +1716,10 @@ impl ReplicationQueues { }, ); self.fetch_queue.push(FetchOrder { key, distance }); + // A fresh enqueue starts a new verification cycle's fetch, so nothing + // asked in an earlier cycle is excluded. Cleared only now, after the + // insert succeeded: a refused enqueue leaves an existing entry alone. + self.fetch_tried.remove(&key); true } @@ -1184,39 +1766,84 @@ impl ReplicationQueues { enqueued } - /// Dequeue the nearest fetch candidate. + /// Dequeue the nearest fetch candidate, whether it is on the fetch heap or + /// parked. /// /// Returns `None` when the queue is empty. Silently skips candidates /// that are somehow already in-flight. Concurrency is enforced by the /// fetch worker, not by this method. /// /// A returned candidate may carry a live verification retry-slot - /// reservation. Callers must consume it with - /// [`Self::start_dequeued_fetch`], [`Self::discard_fetch_candidate`], or + /// reservation, and the sources it was already fetched from this cycle. + /// Callers must consume it with [`Self::start_dequeued_fetch`], + /// [`Self::discard_fetch_candidate`], or /// [`Self::requeue_candidate_for_verification`] so that reservation is /// either transferred, released, or restored to `pending_verify`. pub fn dequeue_fetch(&mut self) -> Option { - while let Some(order) = self.fetch_queue.pop() { - let Some(payload) = self.fetch_payloads.remove(&order.key) else { - debug_assert!( - false, - "fetch order without payload for {}", - hex::encode(order.key) - ); - continue; - }; - let candidate = FetchCandidate { - key: order.key, - distance: order.distance, - sources: payload.sources, - retry_verification: payload.retry_verification, + loop { + let parked = self.nearest_parked(None); + let candidate = match self.take_queued(self.parked_if_nearer(parked)) { + TakenFetch::Candidate(candidate) => candidate, + TakenFetch::Skipped => continue, + TakenFetch::Empty => return None, }; if !self.in_flight_fetch.contains_key(&candidate.key) { return Some(candidate); } - self.release_retry_slot_for_candidate(&candidate); + self.drop_duplicate_of_in_flight(&candidate); } - None + } + + /// `parked`'s key when it is nearer than the heap head, or the heap is + /// empty; `None` to take the heap head. + fn parked_if_nearer(&self, parked: Option<(XorName, XorName)>) -> Option { + let heap_head = self + .fetch_queue + .peek() + .map(|order| (order.distance, order.key)); + match (heap_head, parked) { + (Some(head), Some(parked)) if parked < head => Some(parked.1), + (None, Some(parked)) => Some(parked.1), + _ => None, + } + } + + /// Remove a queued key from where it waits, the parked key `parked` or + /// otherwise the heap head, and rejoin it with its payload. + fn take_queued(&mut self, parked: Option) -> TakenFetch { + let (key, distance) = if let Some(key) = parked { + let Some(distance) = self.unpark(&key) else { + self.purge_parked_index_entry(&key); + return TakenFetch::Skipped; + }; + (key, distance) + } else { + let Some(order) = self.fetch_queue.pop() else { + return TakenFetch::Empty; + }; + (order.key, order.distance) + }; + let Some(payload) = self.fetch_payloads.remove(&key) else { + debug_assert!( + false, + "queued fetch without payload for {}", + hex::encode(key) + ); + return TakenFetch::Skipped; + }; + TakenFetch::Candidate(FetchCandidate { + key, + distance, + sources: payload.sources, + retry_verification: payload.retry_verification, + }) + } + + /// Discard a dequeued candidate whose key is somehow already in flight, + /// releasing what it carried. + fn drop_duplicate_of_in_flight(&mut self, candidate: &FetchCandidate) { + self.release_retry_slot_for_candidate(candidate); + self.fetch_tried.remove(&candidate.key); } /// Number of keys waiting in the fetch queue. @@ -1253,8 +1880,41 @@ impl ReplicationQueues { all_sources: Vec, retry_verification: Option, ) { - let mut tried = HashSet::new(); + self.start_fetch_after(key, source, all_sources, retry_verification, HashSet::new()); + } + + /// Start `key` from `source`, carrying the sources it was already fetched + /// from this cycle into the in-flight entry's tried set. + fn start_fetch_after( + &mut self, + key: XorName, + source: PeerId, + mut all_sources: Vec, + retry_verification: Option, + mut tried: HashSet, + ) { tried.insert(source); + // Keep the distinct union of tried and listed sources within the + // limit by dropping untried sources from the end of the list. Every + // dispatched start is already within it; this binds direct starts. + let union = all_sources.len() + + tried + .iter() + .filter(|peer| !all_sources.contains(peer)) + .count(); + let mut excess = union.saturating_sub(self.max_fetch_sources); + if excess > 0 { + let mut kept = Vec::with_capacity(all_sources.len()); + for peer in all_sources.into_iter().rev() { + if excess > 0 && !tried.contains(&peer) { + excess -= 1; + } else { + kept.push(peer); + } + } + kept.reverse(); + all_sources = kept; + } let replaced = self.in_flight_fetch.insert( key, InFlightEntry { @@ -1271,8 +1931,9 @@ impl ReplicationQueues { } } - /// Consume a dequeued fetch candidate and transfer its retry reservation - /// into the in-flight entry. + /// Consume a dequeued fetch candidate and transfer its retry reservation, + /// and the sources it was already fetched from this cycle, into the + /// in-flight entry. pub fn start_dequeued_fetch(&mut self, candidate: FetchCandidate, source: PeerId) { let FetchCandidate { key, @@ -1280,7 +1941,8 @@ impl ReplicationQueues { retry_verification, .. } = candidate; - self.start_fetch_with_retry(key, source, sources, retry_verification); + let tried = self.fetch_tried.remove(&key).unwrap_or_default(); + self.start_fetch_after(key, source, sources, retry_verification, tried); } /// Mark a fetch as completed (success or permanent failure). @@ -1300,6 +1962,7 @@ impl ReplicationQueues { retry_verification, .. } = candidate; + self.fetch_tried.remove(&key); if retry_verification.is_some() { self.release_retry_slot(&key); self.capacity_owner_by_key.remove(&key); @@ -1329,59 +1992,454 @@ impl ReplicationQueues { } } - /// Consume a dequeued candidate and restore its verification entry for a - /// later retry when retry metadata exists. - pub fn requeue_candidate_for_verification( + // ----------------------------------------------------------------------- + // Per-source dispatch + // ----------------------------------------------------------------------- + + /// Start up to `slots` queued fetches without putting more than + /// `per_source_cap` of this node's fetches in flight to any one source + /// that is answering. + /// + /// Each slot goes to the nearest queued key that can start now: one with + /// an untried source below the cap, or an untried silent source. It starts + /// on the first untried source below the cap that is not silent, in its + /// source-preference order, or else on its first untried silent source, + /// past the cap. A heap key that cannot start is parked under each of its + /// untried sources, so it leaves the heap once and a startable key behind + /// it is reached in the same pass; a parked key becomes a candidate again + /// as soon as one of those sources has a free share or falls silent, and + /// competes on distance with the heap. + /// + /// With `per_source_cap = usize::MAX` nothing parks and every key starts on + /// its first untried source that is not silent, in plain nearest-first + /// order. + pub(crate) fn dispatch_fetches( &mut self, - candidate: FetchCandidate, - retry_after: Duration, - ) -> bool { + slots: usize, + per_source_cap: usize, + ) -> FetchDispatch { + let now = Instant::now(); + self.source_health.expire(now); + let mut dispatch = FetchDispatch::default(); + let mut in_flight = self.in_flight_counts_by_source(None); + let mut best_parked = self.nearest_parked(Some(Startable { + in_flight: &in_flight, + per_source_cap, + now, + })); + while dispatch.started.len() < slots { + let from_parked = self.parked_if_nearer(best_parked); + let candidate = match self.take_queued(from_parked) { + TakenFetch::Candidate(candidate) => Some(candidate), + TakenFetch::Skipped => None, + TakenFetch::Empty => break, + }; + let mut started = false; + if let Some(candidate) = candidate { + if self.in_flight_fetch.contains_key(&candidate.key) { + self.drop_duplicate_of_in_flight(&candidate); + } else { + let startable = Startable { + in_flight: &in_flight, + per_source_cap, + now, + }; + match self.first_startable_source(&candidate, startable) { + SourceChoice::Start(source) => { + if self.source_health.is_silent(&source, now) { + self.source_health.counters.silent_starts += 1; + } + *in_flight.entry(source).or_default() += 1; + dispatch.started.push((candidate.key, source)); + self.start_dequeued_fetch(candidate, source); + started = true; + } + SourceChoice::NoSource => dispatch.sourceless.push(candidate), + SourceChoice::AllBusy => { + if from_parked.is_none() { + dispatch.parked += 1; + } + self.park_candidate(candidate); + } + } + } + } + // A start changes the per-source counts, and a key taken from the + // parked set changes the set itself. A heap key that parked is + // indexed only under full sources that are not silent, so it + // cannot change the answer. + if started || from_parked.is_some() { + best_parked = self.nearest_parked(Some(Startable { + in_flight: &in_flight, + per_source_cap, + now, + })); + } + } + dispatch + } + + /// Put a dequeued candidate back in the queue, parked under its untried + /// sources. + fn park_candidate(&mut self, candidate: FetchCandidate) { let FetchCandidate { key, + distance, + sources, retry_verification, - .. } = candidate; - let Some(mut verification) = retry_verification else { - return false; - }; - verification.state = VerificationState::PendingVerify; - verification.verified_sources.clear(); - verification.tried_sources.clear(); - verification.next_verify_at = Instant::now() + retry_after; + self.fetch_payloads.insert( + key, + FetchPayload { + sources, + retry_verification, + }, + ); + self.park(key, distance); + } - let owner = self - .capacity_owner_by_key - .get(&key) + /// Mark the current fetch attempt as failed and move `key` on, without + /// sending to a source that is answering and already has `per_source_cap` + /// of this node's fetches in flight. + /// + /// The next untried source with a free share that is not silent is + /// returned to retry from now, as [`Self::retry_fetch`] would, or else the + /// first untried silent source, past the cap. If every untried source is + /// answering and at its share, the key goes back to the fetch queue + /// parked, with only its untried sources and with its tried set kept, so + /// no source is asked twice this cycle; its retry reservation travels with + /// it. If no untried source is left, or the fetch queue is at + /// [`MAX_FETCH_QUEUE`], the in-flight entry is left for the caller's + /// verification requeue, which keeps `MAX_FETCH_QUEUE` a hard bound. + pub(crate) fn retry_or_park_fetch( + &mut self, + key: &XorName, + distance: XorName, + per_source_cap: usize, + ) -> FetchRetry { + let now = Instant::now(); + self.source_health.expire(now); + let in_flight = self.in_flight_counts_by_source(Some(key)); + let queue_full = self.fetch_payloads.len() >= MAX_FETCH_QUEUE; + let Some(entry) = self.in_flight_fetch.get_mut(key) else { + return FetchRetry::Exhausted; + }; + entry.tried.insert(entry.source); + let untried: Vec = entry + .all_sources + .iter() + .filter(|peer| !entry.tried.contains(peer)) .copied() - .or_else(|| self.least_loaded_source(&verification.hint_sources)); - let Some(owner) = owner else { - self.release_retry_slot(&key); - return false; + .collect(); + if untried.is_empty() { + return FetchRetry::Exhausted; + } + let health = &self.source_health; + let free = untried + .iter() + .find(|peer| { + !health.is_silent(peer, now) + && in_flight.get(peer).copied().unwrap_or(0) < per_source_cap + }) + .copied(); + let silent = || { + untried + .iter() + .find(|peer| health.is_silent(peer, now)) + .copied() }; - // Release before re-inserting: the reservation is charged to the owner - // recorded now, which `insert_pending_owned_unchecked` may replace. - self.release_retry_slot(&key); - self.insert_pending_owned_unchecked(key, verification, owner, true); - true + let next = free.or_else(silent); + if let Some(next) = next { + if free.is_none() { + self.source_health.counters.silent_starts += 1; + } + entry.source = next; + entry.tried.insert(next); + entry.started_at = now; + return FetchRetry::Next(next); + } + if queue_full { + return FetchRetry::Exhausted; + } + let Some(entry) = self.in_flight_fetch.remove(key) else { + return FetchRetry::Exhausted; + }; + self.fetch_payloads.insert( + *key, + FetchPayload { + sources: untried, + retry_verification: entry.retry_verification, + }, + ); + self.fetch_tried.insert(*key, entry.tried); + self.park(*key, distance); + FetchRetry::Parked } - /// Complete an exhausted fetch and restore its verification entry for a - /// later retry when retry metadata exists. - pub fn requeue_fetch_for_verification(&mut self, key: &XorName, retry_after: Duration) -> bool { - let Some(mut entry) = self.in_flight_fetch.remove(key) else { - return false; - }; - let Some(mut verification) = entry.retry_verification.take() else { - self.capacity_owner_by_key.remove(key); - return false; - }; - verification.state = VerificationState::PendingVerify; - verification.verified_sources.clear(); - verification.tried_sources.clear(); - verification.next_verify_at = Instant::now() + retry_after; + /// In-flight fetches per source, optionally leaving out `exclude`'s own + /// entry. + fn in_flight_counts_by_source(&self, exclude: Option<&XorName>) -> HashMap { + let mut counts = HashMap::new(); + for (key, entry) in &self.in_flight_fetch { + if exclude != Some(key) { + *counts.entry(entry.source).or_default() += 1; + } + } + counts + } - let owner = self - .capacity_owner_by_key + /// The first untried source of `candidate`, in its source-preference + /// order, that is not silent and is below the cap, or else its first + /// untried silent source. + fn first_startable_source( + &self, + candidate: &FetchCandidate, + startable: Startable<'_>, + ) -> SourceChoice { + let tried = self.fetch_tried.get(&candidate.key); + let mut any_untried = false; + let mut first_silent = None; + for source in &candidate.sources { + if tried.is_some_and(|t| t.contains(source)) { + continue; + } + any_untried = true; + if self.source_health.is_silent(source, startable.now) { + first_silent = first_silent.or(Some(*source)); + } else if startable.in_flight.get(source).copied().unwrap_or(0) + < startable.per_source_cap + { + return SourceChoice::Start(*source); + } + } + match (first_silent, any_untried) { + (Some(source), _) => SourceChoice::Start(source), + (None, true) => SourceChoice::AllBusy, + (None, false) => SourceChoice::NoSource, + } + } + + /// Park a queued key whose payload is in `fetch_payloads`, indexed under + /// each of its untried sources. A key with no untried source goes back on + /// the heap instead, so it is never parked with nothing to wake it. + fn park(&mut self, key: XorName, distance: XorName) { + self.unpark(&key); + let tried = self.fetch_tried.get(&key); + let wake_sources: Vec = self + .fetch_payloads + .get(&key) + .map(|payload| { + payload + .sources + .iter() + .filter(|source| !tried.is_some_and(|t| t.contains(source))) + .copied() + .collect() + }) + .unwrap_or_default(); + if wake_sources.is_empty() { + self.fetch_queue.push(FetchOrder { key, distance }); + return; + } + for source in &wake_sources { + self.parked_by_source + .entry(*source) + .or_default() + .insert((distance, key)); + } + self.fetch_parked.insert( + key, + ParkedFetch { + distance, + wake_sources, + parked_at: Instant::now(), + }, + ); + } + + /// The per-source scheduler's state for its periodic summary, resetting + /// the maintenance maximum and source-health counters it reports and + /// purging the health table. Walks the parked set, the index's per-source + /// sets, the health table and the in-flight fetches once each. + pub(crate) fn take_fetch_dispatch_stats( + &mut self, + per_source_cap: usize, + ) -> FetchDispatchStats { + let now = Instant::now(); + let parked_maintenance_max = std::mem::take(&mut self.parked_maintenance_max); + let (health, tracked_sources) = + self.source_health.take_counters(now, &self.in_flight_fetch); + FetchDispatchStats { + tracked_sources, + silent_sources: self.source_health.marks_by_since.len(), + overflow_active: self.source_health.overflowing(now), + health, + parked_associations: self.parked_by_source.values().map(BTreeSet::len).sum(), + parked_maintenance_max, + parked: self.fetch_parked.len(), + oldest_parked: self + .fetch_parked + .values() + .map(|parked| now.saturating_duration_since(parked.parked_at)) + .max(), + queued: self.fetch_payloads.len(), + in_flight: self.in_flight_fetch.len(), + saturated_sources: self + .in_flight_counts_by_source(None) + .values() + .filter(|count| **count >= per_source_cap) + .count(), + } + } + + /// The parked-state part of [`Self::remove_hint_source`]: orphaned keys + /// leave it entirely, and a parked key can no longer start from the + /// departed source. One left with no source to wake it goes back on the + /// heap, where dispatch hands it out as sourceless, exactly as a queued key + /// that lost its last source. Timed for the dispatch summary. + fn drop_departed_source_from_parked( + &mut self, + source: &PeerId, + orphaned_fetch_keys: &HashSet, + ) { + if self.fetch_parked.is_empty() && self.fetch_tried.is_empty() { + return; + } + let started = Instant::now(); + for key in orphaned_fetch_keys { + self.unpark(key); + self.fetch_tried.remove(key); + } + if let Some(parked_here) = self.parked_by_source.remove(source) { + for (distance, key) in parked_here { + let Some(parked) = self.fetch_parked.get_mut(&key) else { + continue; + }; + parked.wake_sources.retain(|peer| peer != source); + if parked.wake_sources.is_empty() { + self.fetch_parked.remove(&key); + self.fetch_queue.push(FetchOrder { key, distance }); + } + } + } + self.parked_maintenance_max = self.parked_maintenance_max.max(started.elapsed()); + } + + /// Take `key` out of the parked state and the index, returning its + /// distance. The payload is left in `fetch_payloads`. + fn unpark(&mut self, key: &XorName) -> Option { + let parked = self.fetch_parked.remove(key)?; + for source in &parked.wake_sources { + if let Some(keys) = self.parked_by_source.get_mut(source) { + keys.remove(&(parked.distance, *key)); + if keys.is_empty() { + self.parked_by_source.remove(source); + } + } + } + Some(parked.distance) + } + + /// The nearest parked key, as `(distance, key)`. With `startable`, only + /// keys indexed under a source below the cap or silent count, so the + /// result is the nearest parked key that could start now. + fn nearest_parked(&self, startable: Option>) -> Option<(XorName, XorName)> { + self.parked_by_source + .iter() + .filter(|(source, _)| { + startable.is_none_or(|startable| { + startable.in_flight.get(source).copied().unwrap_or(0) < startable.per_source_cap + || self.source_health.is_silent(source, startable.now) + }) + }) + .filter_map(|(_, keys)| keys.first().copied()) + .min() + } + + /// Defensive: drop every index entry for `key` when the index names a key + /// that is not parked. Unreachable while the index stays exact; without it + /// such an entry would be offered forever. + fn purge_parked_index_entry(&mut self, key: &XorName) { + debug_assert!( + false, + "parked index names an unparked key {}", + hex::encode(key) + ); + self.parked_by_source.retain(|_, keys| { + keys.retain(|(_, parked)| parked != key); + !keys.is_empty() + }); + } + + /// Test-only: every queued or parked key, and every in-flight fetch with + /// its source, read together. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn fetch_pipeline_snapshot(&self) -> FetchPipelineSnapshot { + FetchPipelineSnapshot { + queued: self.fetch_payloads.keys().copied().collect(), + in_flight: self + .in_flight_fetch + .iter() + .map(|(key, entry)| (*key, entry.source)) + .collect(), + } + } + + /// Consume a dequeued candidate and restore its verification entry for a + /// later retry when retry metadata exists. + pub fn requeue_candidate_for_verification( + &mut self, + candidate: FetchCandidate, + retry_after: Duration, + ) -> bool { + let FetchCandidate { + key, + retry_verification, + .. + } = candidate; + self.fetch_tried.remove(&key); + let Some(mut verification) = retry_verification else { + return false; + }; + verification.state = VerificationState::PendingVerify; + verification.verified_sources.clear(); + verification.tried_sources.clear(); + verification.next_verify_at = Instant::now() + retry_after; + + let owner = self + .capacity_owner_by_key + .get(&key) + .copied() + .or_else(|| self.least_loaded_source(&verification.hint_sources)); + let Some(owner) = owner else { + self.release_retry_slot(&key); + return false; + }; + // Release before re-inserting: the reservation is charged to the owner + // recorded now, which `insert_pending_owned_unchecked` may replace. + self.release_retry_slot(&key); + self.insert_pending_owned_unchecked(key, verification, owner, true); + true + } + + /// Complete an exhausted fetch and restore its verification entry for a + /// later retry when retry metadata exists. + pub fn requeue_fetch_for_verification(&mut self, key: &XorName, retry_after: Duration) -> bool { + let Some(mut entry) = self.in_flight_fetch.remove(key) else { + return false; + }; + let Some(mut verification) = entry.retry_verification.take() else { + self.capacity_owner_by_key.remove(key); + return false; + }; + verification.state = VerificationState::PendingVerify; + verification.verified_sources.clear(); + verification.tried_sources.clear(); + verification.next_verify_at = Instant::now() + retry_after; + + let owner = self + .capacity_owner_by_key .get(key) .copied() .or_else(|| self.least_loaded_source(&verification.hint_sources)); @@ -1396,6 +2454,11 @@ impl ReplicationQueues { true } + /// When `key`'s current fetch attempt started, if it is in flight. + pub(crate) fn in_flight_started_at(&self, key: &XorName) -> Option { + self.in_flight_fetch.get(key).map(|entry| entry.started_at) + } + /// Number of in-flight fetches. #[must_use] pub fn in_flight_count(&self) -> usize { @@ -1516,6 +2579,8 @@ fn max_min_allocations( mod tests { use super::*; use crate::replication::config::CAPACITY_BLOCKED_RETRY; + use rand::rngs::StdRng; + use rand::{Rng, SeedableRng}; /// Build a `PeerId` from a single byte (zero-padded to 32 bytes). fn peer_id_from_byte(b: u8) -> PeerId { @@ -3026,4 +4091,1493 @@ mod tests { ); assert_eq!(queues.pending_count(), 1); } + + // -- per-source fetch dispatch --------------------------------------------- + + const CAP: usize = 2; + + fn peer(b: u8) -> PeerId { + peer_id_from_byte(b) + } + + /// A key whose byte order matches the numeric order of `distance`. + fn key_at(distance: u32) -> XorName { + let mut name = [0u8; 32]; + name[..4].copy_from_slice(&distance.to_be_bytes()); + name + } + + /// Queue `key` at `distance` with `sources`, keyed at its own distance so + /// tests can reason about nearest-first order directly. + fn queue(queues: &mut ReplicationQueues, distance: u32, sources: &[PeerId]) -> XorName { + let key = key_at(distance); + assert!(queues.enqueue_fetch(key, key, sources.to_vec())); + key + } + + /// Queue `key` through pending verification and promotion, so it carries a + /// verification retry reservation owned by `hinter`. + fn queue_promoted( + queues: &mut ReplicationQueues, + distance: u32, + hinter: u8, + sources: &[PeerId], + ) -> XorName { + let key = key_at(distance); + assert!(queues + .add_pending_verify(key, test_entry(hinter)) + .admitted()); + assert!(queues.promote_pending_to_fetch(key, key, sources.to_vec())); + key + } + + fn in_flight_to(queues: &ReplicationQueues, source: &PeerId) -> usize { + queues + .in_flight_fetch + .values() + .filter(|entry| entry.source == *source) + .count() + } + + fn heap_keys(queues: &ReplicationQueues) -> HashSet { + queues.fetch_queue.iter().map(|order| order.key).collect() + } + + /// I1 and I2: every queued payload is on the heap or parked, never both, + /// and the reverse index is exactly what `fetch_parked` implies. + fn assert_queue_invariants(queues: &ReplicationQueues) { + let heap = heap_keys(queues); + assert_eq!(heap.len(), queues.fetch_queue.len(), "duplicate heap entry"); + for key in queues.fetch_payloads.keys() { + let on_heap = heap.contains(key); + let parked = queues.fetch_parked.contains_key(key); + assert!( + on_heap != parked, + "queued key must be on exactly one of heap/parked" + ); + } + for key in heap.iter().chain(queues.fetch_parked.keys()) { + assert!( + queues.fetch_payloads.contains_key(key), + "order without payload" + ); + } + let mut expected: HashMap> = HashMap::new(); + for (key, parked) in &queues.fetch_parked { + assert!( + !parked.wake_sources.is_empty(), + "parked key with nothing to wake it" + ); + for source in &parked.wake_sources { + expected + .entry(*source) + .or_default() + .insert((parked.distance, *key)); + } + } + assert_eq!( + queues.parked_by_source, expected, + "parked index is not exact" + ); + for key in queues.fetch_tried.keys() { + assert!( + queues.fetch_payloads.contains_key(key), + "tried sources kept for a key that is not queued" + ); + } + } + + #[test] + fn dispatch_spreads_fetches_across_sources_within_the_cap() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let keys: Vec = (0..8).map(|d| queue(&mut queues, d, &[a, b, c])).collect(); + + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!(dispatch.started.len(), 6, "two per source, three sources"); + assert!(dispatch.sourceless.is_empty()); + for source in [a, b, c] { + assert_eq!(in_flight_to(&queues, &source), CAP); + } + // Preference order is kept: the nearest keys take the first source. + let expected: Vec<(XorName, PeerId)> = keys + .iter() + .take(6) + .zip([a, a, b, b, c, c]) + .map(|(key, source)| (*key, source)) + .collect(); + assert_eq!(dispatch.started, expected); + assert_eq!(queues.fetch_parked.len(), 2); + assert_queue_invariants(&queues); + } + + #[test] + fn uncapped_dispatch_keeps_the_first_source_order() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + for d in 0..5 { + queue(&mut queues, d, &[a, b]); + } + + let dispatch = queues.dispatch_fetches(8, usize::MAX); + + assert_eq!(dispatch.started.len(), 5); + assert!(dispatch.started.iter().all(|(_, source)| *source == a)); + assert!(queues.fetch_parked.is_empty()); + assert_queue_invariants(&queues); + } + + #[test] + fn a_startable_key_behind_blocked_keys_starts_in_the_same_pass() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + assert_eq!(queues.dispatch_fetches(8, CAP).started.len(), 2); + for d in 10..310 { + queue(&mut queues, d, &[a]); + } + let deeper = queue(&mut queues, 1_000, &[b]); + + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!(dispatch.started, vec![(deeper, b)]); + assert_eq!( + queues.fetch_parked.len(), + 300, + "every blocked key left the heap" + ); + assert!(queues.fetch_queue.is_empty()); + assert_eq!( + queues.fetch_queue_count(), + 300, + "parked keys are still queued" + ); + assert_queue_invariants(&queues); + } + + #[test] + fn a_freed_share_starts_the_nearest_parked_key_for_that_source() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let first = queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + for d in [40, 20, 30] { + queue(&mut queues, d, &[a]); + } + assert_eq!(queues.dispatch_fetches(8, CAP).started.len(), 2); + assert_eq!(queues.fetch_parked.len(), 3); + + queues.complete_fetch(&first); + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!(dispatch.started, vec![(key_at(20), a)]); + assert_eq!(in_flight_to(&queues, &a), CAP); + assert_queue_invariants(&queues); + } + + #[test] + fn the_nearest_startable_parked_key_wins_whatever_its_source() { + let mut queues = ReplicationQueues::new(); + let sources: Vec = (1..=5).map(peer).collect(); + // Saturate every source with two blockers, then park one key per source. + let mut blockers = Vec::new(); + for (index, source) in sources.iter().enumerate() { + let base = u32::try_from(index).unwrap() * 2; + blockers.push(queue(&mut queues, base, &[*source])); + blockers.push(queue(&mut queues, base + 1, &[*source])); + } + assert_eq!(queues.dispatch_fetches(10, CAP).started.len(), 10); + let parked_distances = [500, 100, 400, 200, 300]; + for (source, distance) in sources.iter().zip(parked_distances) { + queue(&mut queues, distance, &[*source]); + } + assert_eq!(queues.dispatch_fetches(5, CAP).started, Vec::new()); + assert_eq!(queues.fetch_parked.len(), 5); + + // Free one share on every source: the two slots go to the two nearest. + for pair in blockers.chunks(2) { + queues.complete_fetch(&pair[0]); + } + let dispatch = queues.dispatch_fetches(2, CAP); + + assert_eq!( + dispatch.started, + vec![(key_at(100), sources[1]), (key_at(200), sources[3])] + ); + assert_queue_invariants(&queues); + } + + #[test] + fn a_parked_key_starts_on_its_preferred_free_source() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + let blockers: Vec = (0..4) + .map(|d| queue(&mut queues, d, &[if d < 2 { a } else { b }])) + .collect(); + assert_eq!(queues.dispatch_fetches(4, CAP).started.len(), 4); + let key = queue(&mut queues, 50, &[a, b]); + assert_eq!(queues.dispatch_fetches(1, CAP).started, Vec::new()); + + // Free b first, then a: both have room when dispatch runs. + queues.complete_fetch(&blockers[2]); + queues.complete_fetch(&blockers[0]); + let dispatch = queues.dispatch_fetches(1, CAP); + + assert_eq!(dispatch.started, vec![(key, a)]); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_moves_to_a_free_untried_source() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + queue(&mut queues, 10, &[b]); + queue(&mut queues, 11, &[b]); + let key = queue(&mut queues, 0, &[a, b, c]); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 3); + assert_eq!(queues.in_flight_fetch[&key].source, a); + + let retry = queues.retry_or_park_fetch(&key, key, CAP); + + assert_eq!(retry, FetchRetry::Next(c), "b is at its share"); + let entry = &queues.in_flight_fetch[&key]; + assert_eq!(entry.source, c); + assert_eq!(entry.tried, HashSet::from([a, c])); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_with_every_untried_source_full_parks_and_keeps_its_history() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = queue_promoted(&mut queues, 0, 9, &[a, b, c]); + let blockers: Vec = (10..14) + .map(|d| queue(&mut queues, d, &[if d < 12 { b } else { c }])) + .collect(); + assert_eq!(queues.dispatch_fetches(5, CAP).started.len(), 5); + assert_eq!(queues.in_flight_fetch[&key].source, a); + let reserved = queues.retry_reserved_slot_count(); + let owner = queues.capacity_owner_by_key.get(&key).copied(); + let reserved_by_owner = queues.retry_reserved_by_owner.clone(); + assert_eq!(reserved_by_owner.get(&peer(9)), Some(&1)); + + let retry = queues.retry_or_park_fetch(&key, key, CAP); + + assert_eq!(retry, FetchRetry::Parked); + assert!(!queues.in_flight_fetch.contains_key(&key)); + assert_eq!(queues.fetch_payloads[&key].sources, vec![b, c]); + assert!(queues.fetch_payloads[&key].retry_verification.is_some()); + assert_eq!(queues.fetch_tried[&key], HashSet::from([a])); + assert_eq!(queues.fetch_parked[&key].wake_sources, vec![b, c]); + assert_eq!(queues.retry_reserved_slot_count(), reserved); + assert_eq!(queues.capacity_owner_by_key.get(&key).copied(), owner); + assert_eq!(queues.retry_reserved_by_owner, reserved_by_owner); + assert!(queues.contains_key(&key)); + assert_queue_invariants(&queues); + + // A share on b frees: the key restarts there, still remembering a. + queues.complete_fetch(&blockers[0]); + let dispatch = queues.dispatch_fetches(1, CAP); + assert_eq!(dispatch.started, vec![(key, b)]); + assert_eq!(queues.in_flight_fetch[&key].tried, HashSet::from([a, b])); + assert!(!queues.fetch_tried.contains_key(&key)); + assert_eq!(queues.retry_reserved_slot_count(), reserved); + assert_eq!(queues.capacity_owner_by_key.get(&key).copied(), owner); + assert_eq!(queues.retry_reserved_by_owner, reserved_by_owner); + assert_queue_invariants(&queues); + } + + #[test] + fn a_parked_key_wakes_through_any_of_its_sources_beyond_the_eighth() { + let mut queues = ReplicationQueues::new(); + let sources: Vec = (1..=9).map(peer).collect(); + let mut blockers = Vec::new(); + for (index, source) in sources.iter().enumerate() { + let base = u32::try_from(index).unwrap() * 2; + blockers.push(queue(&mut queues, base, &[*source])); + blockers.push(queue(&mut queues, base + 1, &[*source])); + } + assert_eq!(queues.dispatch_fetches(18, CAP).started.len(), 18); + let key = queue(&mut queues, 100, &sources); + assert_eq!(queues.dispatch_fetches(1, CAP).started, Vec::new()); + assert_eq!(queues.fetch_parked[&key].wake_sources, sources); + + // Only the ninth source frees a share. + queues.complete_fetch(&blockers[16]); + let dispatch = queues.dispatch_fetches(1, CAP); + + assert_eq!(dispatch.started, vec![(key, sources[8])]); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_with_no_untried_source_is_exhausted_and_left_in_flight() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let key = queue(&mut queues, 0, &[a]); + assert_eq!(queues.dispatch_fetches(1, CAP).started.len(), 1); + + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Exhausted + ); + assert!(queues.in_flight_fetch.contains_key(&key)); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_with_a_full_fetch_queue_is_exhausted_rather_than_overfilling_it() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + let key = queue(&mut queues, 0, &[a, b]); + queue(&mut queues, 1, &[b]); + queue(&mut queues, 2, &[b]); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 3); + let mut filler = 1_000_u32; + while queues.fetch_queue_count() < MAX_FETCH_QUEUE { + queue(&mut queues, filler, &[peer(7)]); + filler += 1; + } + + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Exhausted + ); + assert!(queues.in_flight_fetch.contains_key(&key)); + assert_eq!(queues.fetch_queue_count(), MAX_FETCH_QUEUE); + } + + /// Park `key` (sources a, b, c) with a already tried and b, c full. + fn park_after_trying_a( + queues: &mut ReplicationQueues, + a: PeerId, + b: PeerId, + c: PeerId, + ) -> XorName { + let key = queue(queues, 0, &[a, b, c]); + for d in 10..14 { + queue(queues, d, &[if d < 12 { b } else { c }]); + } + assert_eq!(queues.dispatch_fetches(5, CAP).started.len(), 5); + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Parked + ); + key + } + + #[test] + fn a_tried_source_readvertised_for_a_parked_key_is_not_merged_back() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + + let result = queues.add_pending_verify(key, test_entry(1)); + + assert_eq!(result, AdmissionResult::AlreadyPresent); + assert_eq!(queues.fetch_payloads[&key].sources, vec![b, c]); + assert_eq!(queues.fetch_parked[&key].wake_sources, vec![b, c]); + assert_queue_invariants(&queues); + } + + #[test] + fn a_new_source_merged_into_a_parked_key_can_start_it() { + let mut queues = ReplicationQueues::new(); + let (a, b, c, d) = (peer(1), peer(2), peer(3), peer(4)); + let key = park_after_trying_a(&mut queues, a, b, c); + + assert_eq!( + queues.add_pending_verify(key, test_entry(4)), + AdmissionResult::AlreadyPresent + ); + assert_eq!(queues.fetch_parked[&key].wake_sources, vec![b, c, d]); + assert_queue_invariants(&queues); + + let dispatch = queues.dispatch_fetches(1, CAP); + assert_eq!(dispatch.started, vec![(key, d)]); + assert_eq!(queues.in_flight_fetch[&key].tried, HashSet::from([a, d])); + assert_queue_invariants(&queues); + } + + #[test] + fn an_in_flight_merge_skips_tried_sources_and_respects_the_limit() { + let mut queues = ReplicationQueues::new().with_fetch_source_limit(4); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + // Restart it on b, carrying a as tried: associated sources are a, b, c. + let blocker = queues + .in_flight_fetch + .iter() + .find(|(_, entry)| entry.source == b) + .map(|(key, _)| *key) + .unwrap(); + queues.complete_fetch(&blocker); + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, b)]); + + // a was tried: not merged. 5 is new and fits (4 associated). 6 does not. + for hinter in [1, 5, 6] { + queues.add_pending_verify(key, test_entry(hinter)); + } + + assert_eq!( + queues.in_flight_fetch[&key].all_sources, + vec![b, c, peer(5)] + ); + assert_queue_invariants(&queues); + } + + /// A re-advertisement from `sender` that claims no replica. + fn paid_only_entry(sender: u8) -> VerificationEntry { + let mut entry = test_entry(sender); + entry.replica_hint_sources.clear(); + entry + } + + /// Every fetch-stage key's retry entry is within the hint bound, as a + /// pending entry is, and still holds the key's capacity owner. + fn assert_retry_entries_bounded(queues: &ReplicationQueues) { + let retries = queues + .fetch_payloads + .iter() + .filter_map(|(key, payload)| Some((key, payload.retry_verification.as_ref()?))) + .chain( + queues + .in_flight_fetch + .iter() + .filter_map(|(key, entry)| Some((key, entry.retry_verification.as_ref()?))), + ); + for (key, retry) in retries { + assert!( + retry.hint_sources.len() <= MAX_HINT_SOURCES_PER_KEY, + "{} retry entry over the hint bound", + hex::encode(key) + ); + assert!(retry.replica_hint_sources.is_subset(&retry.hint_sources)); + if let Some(owner) = queues.capacity_owner_by_key.get(key) { + assert!( + retry.hint_sources.contains(owner), + "{} retry entry lost its capacity owner", + hex::encode(key) + ); + } + } + } + + #[test] + fn readvertisements_keep_a_fetch_stage_retry_entry_within_the_hint_bound() { + let mut queues = ReplicationQueues::new(); + let (owner, holder) = (peer(1), peer(2)); + let key = key_at(0); + // The owner advertised the key without claiming a replica, so only + // the capacity accounting keeps it in the set. + assert!(queues + .add_pending_verify(key, paid_only_entry(1)) + .admitted()); + assert!(queues.promote_pending_to_fetch(key, key, vec![holder])); + + // Queued: paid-only advertisers fill the set to the bound, then replica + // claimants displace every one of them except the owner. + for sender in 20..40 { + queues.add_pending_verify(key, paid_only_entry(sender)); + } + for sender in 40..60 { + queues.add_pending_verify(key, test_entry(sender)); + } + assert_retry_entries_bounded(&queues); + let retry_sizes = queues + .fetch_payloads + .get(&key) + .and_then(|payload| payload.retry_verification.as_ref()) + .map(|retry| (retry.hint_sources.len(), retry.replica_hint_sources.len())); + assert_eq!( + retry_sizes, + Some((MAX_HINT_SOURCES_PER_KEY, MAX_HINT_SOURCES_PER_KEY - 1)), + "a full set keeps the owner and fills the rest with replica claimants" + ); + + // In flight: the same bound. + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, holder)]); + for sender in 60..80 { + queues.add_pending_verify(key, test_entry(sender)); + } + assert_retry_entries_bounded(&queues); + + // Back in pending verification, the restored entry keeps the bound and + // the owner its reservation was charged to. + assert!(queues.requeue_fetch_for_verification(&key, Duration::ZERO)); + let pending = queues.get_pending(&key).map(|entry| { + ( + entry.hint_sources.len(), + entry.hint_sources.contains(&owner), + ) + }); + assert_eq!(pending, Some((MAX_HINT_SOURCES_PER_KEY, true))); + assert_eq!(queues.capacity_owner_by_key.get(&key), Some(&owner)); + } + + #[test] + fn source_lists_are_capped_at_the_limit_including_tried_sources() { + let mut queues = ReplicationQueues::new().with_fetch_source_limit(3); + let sources: Vec = (1..=6).map(peer).collect(); + let key = queue(&mut queues, 0, &sources); + assert_eq!(queues.fetch_payloads[&key].sources, sources[..3].to_vec()); + + // Park it with one tried source: payload b, c plus tried a is the limit. + let mut queues = ReplicationQueues::new().with_fetch_source_limit(3); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + queues.add_pending_verify(key, test_entry(4)); + assert_eq!(queues.fetch_payloads[&key].sources, vec![b, c]); + assert_queue_invariants(&queues); + + // A direct start with an oversized list is cut to the limit too. + let mut queues = ReplicationQueues::new().with_fetch_source_limit(2); + let direct = key_at(9); + queues.start_fetch(direct, sources[0], sources.clone()); + assert_eq!( + queues.in_flight_fetch[&direct].all_sources, + sources[..2].to_vec() + ); + + // A start from a source outside the list counts that source too. + let mut queues = ReplicationQueues::new().with_fetch_source_limit(3); + let outside = peer(9); + queues.start_fetch(direct, outside, sources[..4].to_vec()); + let entry = &queues.in_flight_fetch[&direct]; + assert_eq!(entry.all_sources, sources[..2].to_vec()); + let union: HashSet<&PeerId> = entry.all_sources.iter().chain(&entry.tried).collect(); + assert_eq!(union.len(), 3); + } + + #[test] + fn a_parked_key_that_loses_its_last_wake_source_returns_to_the_heap() { + let mut queues = ReplicationQueues::new(); + let (b, c) = (peer(2), peer(3)); + queue(&mut queues, 0, &[b]); + queue(&mut queues, 1, &[b]); + queue(&mut queues, 2, &[c]); + // Hinted by 9, so the retry metadata survives b leaving. + let key = queue_promoted(&mut queues, 10, 9, &[b]); + let started = queues.dispatch_fetches(4, CAP).started; + assert!(started.iter().all(|(started_key, _)| *started_key != key)); + assert!(queues.fetch_parked.contains_key(&key)); + + let orphaned = queues.remove_hint_source(&b); + + assert!(!orphaned.contains(&key)); + assert!(!queues.fetch_parked.contains_key(&key)); + assert!(heap_keys(&queues).contains(&key)); + assert_queue_invariants(&queues); + let dispatch = queues.dispatch_fetches(4, CAP); + assert!(dispatch + .sourceless + .iter() + .any(|candidate| candidate.key == key)); + } + + #[test] + fn an_orphaned_parked_key_leaves_every_structure() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + queues.remove_hint_source(&b); + assert!( + queues.fetch_parked.contains_key(&key), + "c can still wake it" + ); + + let orphaned = queues.remove_hint_source(&c); + + assert!(orphaned.contains(&key)); + assert!(!queues.fetch_payloads.contains_key(&key)); + assert!(!queues.fetch_parked.contains_key(&key)); + assert!(!queues.fetch_tried.contains_key(&key)); + assert!(!queues.contains_key(&key)); + assert_queue_invariants(&queues); + } + + #[test] + fn tried_sources_survive_dequeue_and_move_into_the_next_start() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + + let candidate = queues.dequeue_fetch().unwrap(); + assert_eq!( + candidate.key, key, + "the parked key is the nearest queued key" + ); + assert!(queues.fetch_tried.contains_key(&key)); + queues.start_dequeued_fetch(candidate, c); + + assert_eq!(queues.in_flight_fetch[&key].tried, HashSet::from([a, c])); + assert!(!queues.fetch_tried.contains_key(&key)); + assert_queue_invariants(&queues); + } + + #[test] + fn tried_sources_are_dropped_with_a_discarded_or_requeued_candidate() { + for requeue in [false, true] { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + let candidate = queues.dequeue_fetch().unwrap(); + assert_eq!(candidate.key, key); + if requeue { + queues.requeue_candidate_for_verification(candidate, Duration::from_secs(1)); + } else { + queues.discard_fetch_candidate(candidate); + } + assert!(!queues.fetch_tried.contains_key(&key)); + assert_queue_invariants(&queues); + } + } + + #[test] + fn tried_sources_are_kept_by_a_refused_enqueue_and_cleared_by_a_fresh_one() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + assert!(!queues.enqueue_fetch(key, key, vec![a]), "already queued"); + assert_eq!(queues.fetch_tried[&key], HashSet::from([a])); + + let stray = key_at(77); + queues.fetch_tried.insert(stray, HashSet::from([a])); + assert!(queues.enqueue_fetch(stray, stray, vec![a])); + assert!(!queues.fetch_tried.contains_key(&stray)); + } + + #[test] + fn a_duplicate_of_an_in_flight_key_is_dropped_with_its_tried_sources() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let key = key_at(3); + queues.start_fetch(key, a, vec![a]); + // Force the impossible state the defensive skip exists for. + queues.fetch_payloads.insert( + key, + FetchPayload { + sources: vec![a], + retry_verification: None, + }, + ); + queues.fetch_queue.push(FetchOrder { key, distance: key }); + queues.fetch_tried.insert(key, HashSet::from([a])); + + assert!(queues.dequeue_fetch().is_none()); + assert!(!queues.fetch_tried.contains_key(&key)); + assert!(queues.in_flight_fetch.contains_key(&key)); + } + + #[test] + fn dequeue_returns_a_parked_key_nearer_than_the_heap_head() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + let parked = queue(&mut queues, 5, &[a]); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 2); + let farther = queue(&mut queues, 9, &[peer(2)]); + + assert_eq!(queues.dequeue_fetch().map(|c| c.key), Some(parked)); + assert_eq!(queues.dequeue_fetch().map(|c| c.key), Some(farther)); + } + + #[test] + fn a_parked_key_holds_bootstrap_drain_open() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + let parked = queue(&mut queues, 2, &[a]); + queues.dispatch_fetches(3, CAP); + assert!(queues.fetch_parked.contains_key(&parked)); + + assert!(!queues.is_bootstrap_work_empty(&HashSet::from([parked]))); + } + + // -- silent sources --------------------------------------------------------- + + /// A fetch request timeout for the table tests: a mark lasts twice this. + const TEST_TIMEOUT: Duration = Duration::from_secs(10); + + /// A peer distinct from every `peer(b)` and every other index. + fn distinct_peer(index: u32) -> PeerId { + let mut bytes = [0xA5_u8; 32]; + bytes[..4].copy_from_slice(&index.to_be_bytes()); + PeerId::from_bytes(bytes) + } + + /// `base` moved forward by `millis`. + fn at(base: Instant, millis: u64) -> Instant { + base + Duration::from_millis(millis) + } + + fn unreachable(sent: u64) -> Contact { + Contact::Unreachable { sent } + } + + fn answered(order: u64) -> Contact { + Contact::Answered { order } + } + + /// Mark `source` silent from now with a request at sequence place `sent`. + fn silence(queues: &mut ReplicationQueues, source: PeerId, sent: u64) { + queues.record_fetch_contact(source, unreachable(sent)); + } + + fn in_flight_entry(key: XorName, source: PeerId, started_at: Instant) -> InFlightEntry { + InFlightEntry { + key, + source, + started_at, + all_sources: vec![source], + tried: HashSet::from([source]), + retry_verification: None, + } + } + + /// The health table's mark index matches its entries exactly, and only a + /// source with an unanswered request since its latest answer is marked. + fn assert_health_invariants(table: &SourceHealthTable) { + assert!(table.entries.len() <= MAX_TRACKED_FETCH_SOURCES); + let marked: BTreeSet<(Instant, PeerId)> = table + .entries + .iter() + .filter_map(|(source, health)| health.marked_since.map(|since| (since, *source))) + .collect(); + assert_eq!(table.marks_by_since, marked, "mark index is not exact"); + for health in table.entries.values() { + assert!( + health.marked_since.is_none() || health.unanswered_since_answer(), + "a mark outlived the evidence for it" + ); + } + } + + #[test] + fn an_unanswered_request_marks_its_source_until_the_mark_expires() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + table.record(a, unreachable(1), at(t0, 100), &none); + assert!(table.is_silent(&a, at(t0, 100))); + assert!(table.is_silent(&a, at(t0, 20_099))); + assert!( + !table.is_silent(&a, at(t0, 20_100)), + "a mark lasts two fetch timeouts from when the failure was learned" + ); + + table.expire(at(t0, 20_100)); + assert_eq!(table.counters.expired_marks, 1); + assert!(table.marks_by_since.is_empty()); + + // An older unanswered request learned after the expiry is not new + // evidence: the source stays under the cap for its probe. + table.record(a, unreachable(0), at(t0, 20_200), &none); + assert!(!table.is_silent(&a, at(t0, 20_200))); + + // A request sent later that also goes unanswered marks it again. + table.record(a, unreachable(2), at(t0, 30_200), &none); + assert!(table.is_silent(&a, at(t0, 30_200))); + assert_eq!(table.counters.marks, 2); + assert_eq!(table.counters.unreachable, 3); + assert_health_invariants(&table); + } + + #[test] + fn a_source_that_keeps_failing_stays_silent_past_one_lifetime() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + for (sent, millis) in [(1, 0), (2, 15_000), (3, 30_000)] { + table.record(a, unreachable(sent), at(t0, millis), &none); + } + + assert!(table.is_silent(&a, at(t0, 49_999))); + assert!(!table.is_silent(&a, at(t0, 50_000))); + assert_eq!( + table.counters.marks, 1, + "one mark, refreshed by newer failures" + ); + assert_eq!(table.marks_by_since.len(), 1); + assert_health_invariants(&table); + } + + #[test] + fn a_failure_learned_late_still_marks_for_a_whole_lifetime() { + let none = HashMap::new(); + let a = peer(1); + let learned = at(Instant::now(), 600_000); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + table.record(a, unreachable(1), learned, &none); + + assert!(table.is_silent(&a, learned)); + assert!(table.is_silent(&a, at(learned, 19_999))); + assert!(!table.is_silent(&a, at(learned, 20_000))); + } + + /// Every order of applying `events` to a fresh table, all learned at + /// `now`, so only the sequence places can decide. + fn apply_in_every_order(events: &[Contact], now: Instant) -> Vec { + let none = HashMap::new(); + let a = peer(1); + let mut orders: Vec> = vec![Vec::new()]; + for _ in events { + orders = orders + .into_iter() + .flat_map(|prefix| { + (0..events.len()) + .filter(|index| !prefix.contains(index)) + .map(|index| { + let mut next = prefix.clone(); + next.push(index); + next + }) + .collect::>() + }) + .collect(); + } + orders + .iter() + .map(|order| { + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + for event in order.iter().filter_map(|index| events.get(*index)) { + table.record(a, *event, now, &none); + } + assert_health_invariants(&table); + table + }) + .collect() + } + + #[test] + fn whether_a_source_is_silent_does_not_depend_on_the_order_outcomes_arrive() { + let a = peer(1); + let now = Instant::now(); + let lifetime_ms = + u64::try_from((TEST_TIMEOUT * SILENCE_LIFETIME_TIMEOUTS).as_millis()).unwrap(); + + // Unanswered at 1, an answer at 2, unanswered again at 3: silent, as + // the request sent after the answer went unanswered, for one lifetime. + let silent = [unreachable(1), answered(2), unreachable(3)]; + let tables = apply_in_every_order(&silent, now); + assert_eq!(tables.len(), 6); + for table in &tables { + assert!(table.is_silent(&a, now)); + assert!(table.is_silent(&a, at(now, lifetime_ms - 1))); + assert!(!table.is_silent(&a, at(now, lifetime_ms))); + } + + // Unanswered at 1 and 2, then an answer at 3: answering. + let answering = [unreachable(1), unreachable(2), answered(3)]; + for table in &apply_in_every_order(&answering, now) { + assert!(!table.is_silent(&a, now)); + } + } + + #[test] + fn an_answer_older_than_the_unanswered_request_neither_clears_nor_blocks_its_mark() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(a, unreachable(5), t0, &none); + table.record(a, answered(4), t0, &none); + assert!(table.is_silent(&a, t0)); + assert_eq!(table.counters.clears, 0); + + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(a, answered(4), t0, &none); + table.record(a, unreachable(5), t0, &none); + assert!(table.is_silent(&a, t0)); + assert_eq!(table.counters.ignored_stale, 0); + + // A request sent before the latest answer is stale and cannot mark. + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(a, answered(6), t0, &none); + table.record(a, unreachable(5), t0, &none); + assert!(!table.is_silent(&a, t0)); + assert_eq!((table.counters.marks, table.counters.ignored_stale), (0, 1)); + assert_health_invariants(&table); + } + + #[test] + fn an_expired_mark_is_kept_only_while_its_source_has_a_fetch_in_flight() { + let t0 = Instant::now(); + let (guarded, idle) = (peer(1), peer(2)); + let key = key_at(0); + let in_flight = HashMap::from([(key, in_flight_entry(key, guarded, t0))]); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(guarded, unreachable(1), t0, &in_flight); + table.record(idle, unreachable(2), t0, &in_flight); + + let later = at(t0, 20_000); + let (counters, tracked) = table.take_counters(later, &in_flight); + + assert_eq!((counters.marks, counters.expired_marks, tracked), (2, 2, 2)); + assert!(table.entries.contains_key(&guarded)); + assert!(!table.entries.contains_key(&idle)); + // The guarded source keeps its history: its older request is no new + // evidence, its newer one marks it again. + table.record(guarded, unreachable(0), later, &in_flight); + assert!(!table.is_silent(&guarded, later)); + table.record(guarded, unreachable(3), later, &in_flight); + assert!(table.is_silent(&guarded, later)); + assert_health_invariants(&table); + } + + #[test] + fn attempts_that_sent_nothing_or_were_cancelled_change_nothing() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + table.record(a, Contact::NotAttempted, t0, &none); + table.record(a, Contact::Unknown, t0, &none); + + assert!(table.entries.is_empty()); + assert!(!table.is_silent(&a, t0)); + assert_eq!(table.counters.unreachable, 0); + } + + #[test] + fn a_zero_fetch_timeout_still_leaves_silent_sources_a_lifetime() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(Duration::ZERO); + + table.record(a, unreachable(1), t0, &none); + + assert!(table.is_silent(&a, t0)); + assert!(table.is_silent(&a, at(t0, 999))); + assert!(!table.is_silent(&a, at(t0, 1_000))); + } + + /// A table at its bound: `guarded` answered and has a fetch in flight, + /// `idle` answered and has none, and every other entry is marked. + fn full_table( + t0: Instant, + marked: usize, + ) -> ( + SourceHealthTable, + HashMap, + PeerId, + PeerId, + ) { + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + let guarded = distinct_peer(0); + let idle = distinct_peer(1); + let key = key_at(0); + let in_flight = HashMap::from([(key, in_flight_entry(key, guarded, t0))]); + table.record(guarded, answered(1), t0, &in_flight); + if marked < MAX_TRACKED_FETCH_SOURCES - 1 { + table.record(idle, answered(1), t0, &in_flight); + } + for index in 0..marked { + let source = distinct_peer(u32::try_from(index).unwrap() + 2); + table.record(source, unreachable(1), t0, &in_flight); + } + assert_eq!(table.entries.len(), MAX_TRACKED_FETCH_SOURCES); + (table, in_flight, guarded, idle) + } + + #[test] + fn a_full_table_makes_room_by_dropping_answers_that_guard_nothing() { + let t0 = Instant::now(); + let (mut table, in_flight, guarded, idle) = full_table(t0, MAX_TRACKED_FETCH_SOURCES - 2); + let newcomer = distinct_peer(u32::MAX); + + table.record(newcomer, unreachable(1), t0, &in_flight); + + assert!(table.is_silent(&newcomer, t0)); + assert!( + table.entries.contains_key(&guarded), + "an in-flight guard stays" + ); + assert!(!table.entries.contains_key(&idle)); + assert_eq!(table.counters.overflows, 0); + assert_health_invariants(&table); + } + + #[test] + fn a_table_full_of_marks_and_guards_counts_untracked_sources_as_silent() { + let t0 = Instant::now(); + let (mut table, in_flight, guarded, _) = full_table(t0, MAX_TRACKED_FETCH_SOURCES - 1); + let newcomer = distinct_peer(u32::MAX); + let bystander = distinct_peer(u32::MAX - 1); + + table.record(newcomer, unreachable(1), t0, &in_flight); + + assert_eq!(table.counters.overflows, 1); + assert!(!table.entries.contains_key(&newcomer)); + assert!( + table.is_silent(&newcomer, t0), + "an untracked source that stopped answering is fetched from as before the cap" + ); + assert!(table.is_silent(&bystander, t0)); + assert!( + table.entries.contains_key(&guarded), + "an in-flight guard is never dropped" + ); + assert!( + !table.is_silent(&guarded, t0), + "a tracked source keeps its own state" + ); + let lifetime = TEST_TIMEOUT * SILENCE_LIFETIME_TIMEOUTS; + assert!( + !table.is_silent(&bystander, t0 + lifetime), + "the overflow lasts one silence lifetime" + ); + assert_health_invariants(&table); + } + + #[test] + fn a_silent_source_is_fetched_from_past_the_cap() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + for d in 0..6 { + queue(&mut queues, d, &[a]); + } + assert_eq!(queues.dispatch_fetches(8, CAP).started.len(), CAP); + assert_eq!(queues.fetch_parked.len(), 4); + + silence(&mut queues, a, 1); + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!( + dispatch.started.len(), + 4, + "every parked key starts on the silent source" + ); + assert_eq!(in_flight_to(&queues, &a), 6); + assert!(queues.fetch_parked.is_empty()); + assert_eq!(queues.source_health.counters.silent_starts, 4); + assert_queue_invariants(&queues); + } + + #[test] + fn a_sole_silent_holder_drains_one_timeout_after_it_is_noticed() { + // Sixteen keys held only by `a`, eight fetch slots, and every request + // to `a` unanswered: one wave of two at the share, then the rest past + // it, eight per wave, as many waves as before the cap plus one. + let mut queues = ReplicationQueues::new(); + let a = peer(1); + for d in 0..16 { + queue(&mut queues, d, &[a]); + } + let mut sequence = 0; + let mut waves = Vec::new(); + loop { + let started = queues.dispatch_fetches(8, CAP).started; + if started.is_empty() { + break; + } + waves.push(started.len()); + // Every request in the wave times out. + for (key, source) in started { + sequence += 1; + queues.record_fetch_contact(source, unreachable(sequence)); + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Exhausted + ); + queues.complete_fetch(&key); + } + assert_queue_invariants(&queues); + } + assert_eq!(waves, vec![2, 8, 6]); + assert_eq!(queues.fetch_queue_count(), 0); + } + + #[test] + fn an_answering_source_with_room_is_preferred_over_a_silent_one() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + silence(&mut queues, a, 1); + let keys: Vec = (0..3).map(|d| queue(&mut queues, d, &[a, b])).collect(); + + let dispatch = queues.dispatch_fetches(8, CAP); + + let expected: Vec<(XorName, PeerId)> = keys.iter().copied().zip([b, b, a]).collect(); + assert_eq!( + dispatch.started, expected, + "b until its share is full, then the silent a rather than parking" + ); + assert!(queues.fetch_parked.is_empty()); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_prefers_an_answering_source_with_room_then_a_silent_one() { + let (a, b, c) = (peer(1), peer(2), peer(3)); + for c_full in [false, true] { + let mut queues = ReplicationQueues::new(); + let key = queue(&mut queues, 0, &[a, b, c]); + if c_full { + queue(&mut queues, 10, &[c]); + queue(&mut queues, 11, &[c]); + } + let started = if c_full { 3 } else { 1 }; + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), started); + assert_eq!( + queues.in_flight_fetch.get(&key).map(|entry| entry.source), + Some(a) + ); + silence(&mut queues, b, 1); + + let retry = queues.retry_or_park_fetch(&key, key, CAP); + + let expected = if c_full { b } else { c }; + assert_eq!(retry, FetchRetry::Next(expected)); + assert_eq!( + queues.source_health.counters.silent_starts, + u64::from(c_full) + ); + assert_queue_invariants(&queues); + } + } + + #[test] + fn parked_keys_start_on_a_source_that_falls_silent_while_the_others_stay_full() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + for d in 0..4 { + queue(&mut queues, d, &[if d < 2 { a } else { b }]); + } + assert_eq!(queues.dispatch_fetches(4, CAP).started.len(), 4); + let parked: Vec = (10..20).map(|d| queue(&mut queues, d, &[a, b])).collect(); + assert_eq!(queues.dispatch_fetches(8, CAP).started, Vec::new()); + assert_eq!(queues.fetch_parked.len(), 10); + + silence(&mut queues, b, 1); + let dispatch = queues.dispatch_fetches(8, CAP); + + // a stays full and answering: every slot goes to b, and the pass ends. + let expected: Vec<(XorName, PeerId)> = parked.iter().take(8).map(|key| (*key, b)).collect(); + assert_eq!(dispatch.started, expected); + assert_eq!(queues.fetch_parked.len(), 2); + assert_queue_invariants(&queues); + } + + #[test] + fn an_answer_puts_a_silent_source_back_under_the_cap() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + for d in 0..6 { + queue(&mut queues, d, &[a]); + } + silence(&mut queues, a, 1); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 3); + + queues.record_fetch_contact(a, answered(2)); + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!( + dispatch.started, + Vec::new(), + "a answers again and already has three fetches in flight" + ); + assert_eq!(queues.fetch_parked.len(), 3); + assert_queue_invariants(&queues); + } + + #[test] + fn a_departure_keeps_what_was_observed_of_the_source() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let key = queue(&mut queues, 0, &[a]); + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, a)]); + silence(&mut queues, a, 1); + + queues.remove_hint_source(&a); + + assert!(queues.source_health.is_silent(&a, Instant::now())); + // The attempt still in flight answers after the departure: applied + // like any other outcome. + queues.record_fetch_contact(a, answered(2)); + assert!(!queues.source_health.is_silent(&a, Instant::now())); + // A late unanswered request sent before that answer cannot mark it. + queues.record_fetch_contact(a, unreachable(1)); + assert!(!queues.source_health.is_silent(&a, Instant::now())); + assert_health_invariants(&queues.source_health); + } + + #[test] + fn the_summary_drops_answers_that_guard_nothing_and_keeps_marks() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = queue(&mut queues, 0, &[a]); + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, a)]); + queues.record_fetch_contact(a, answered(1)); + queues.record_fetch_contact(b, answered(2)); + silence(&mut queues, c, 3); + + let stats = queues.take_fetch_dispatch_stats(CAP); + + assert_eq!(stats.tracked_sources, 3); + assert_eq!(stats.silent_sources, 1); + assert!(!stats.overflow_active); + assert_eq!(stats.health.marks, 1); + assert_eq!(stats.health.silent_peak, 1); + let kept: HashSet = queues.source_health.entries.keys().copied().collect(); + assert_eq!( + kept, + HashSet::from([a, c]), + "a guards a fetch in flight and c is marked" + ); + assert_eq!(queues.source_health.counters.marks, 0, "counters restart"); + assert_eq!( + queues.source_health.counters.silent_peak, 1, + "the peak restarts at the sources silent now" + ); + assert_health_invariants(&queues.source_health); + } + + #[test] + #[allow(clippy::too_many_lines)] + fn queue_invariants_hold_under_random_operations() { + let mut rng = StdRng::seed_from_u64(0x1514); + let mut queues = ReplicationQueues::new().with_fetch_source_limit(5); + let sources: Vec = (1..=6).map(peer).collect(); + let mut sequence = 0_u64; + for step in 0..4_000_u32 { + match rng.gen_range(0..10) { + 0 | 1 => { + let count = rng.gen_range(1..=4); + let mut chosen: Vec = Vec::new(); + while chosen.len() < count { + let source = sources[rng.gen_range(0..sources.len())]; + if !chosen.contains(&source) { + chosen.push(source); + } + } + let key = key_at(rng.gen_range(0..400)); + if rng.gen_bool(0.25) { + // Through verification, so it carries a retry entry. + let hinter = rng.gen_range(1..=30); + if queues + .add_pending_verify(key, test_entry(hinter)) + .admitted() + { + queues.promote_pending_to_fetch(key, key, chosen); + } + } else { + queues.enqueue_fetch(key, key, chosen); + } + } + 2 => { + // As the fetch worker does, a candidate with no untried + // source goes back to verification. + let dispatch = queues.dispatch_fetches(rng.gen_range(0..6), CAP); + for candidate in dispatch.sourceless { + queues.requeue_candidate_for_verification(candidate, Duration::ZERO); + } + } + 3 => { + if let Some(key) = queues.in_flight_fetch.keys().min().copied() { + queues.complete_fetch(&key); + } + } + 4 => { + let mut in_flight: Vec = + queues.in_flight_fetch.keys().copied().collect(); + in_flight.sort_unstable(); + if !in_flight.is_empty() { + let key = in_flight[rng.gen_range(0..in_flight.len())]; + if queues.retry_or_park_fetch(&key, key, CAP) == FetchRetry::Exhausted { + queues.complete_fetch(&key); + } + } + } + 5 => { + let mut waiting: Vec = queues.fetch_payloads.keys().copied().collect(); + waiting.sort_unstable(); + if !waiting.is_empty() { + let key = waiting[rng.gen_range(0..waiting.len())]; + let hinter = rng.gen_range(1..=30); + let entry = if rng.gen_bool(0.5) { + test_entry(hinter) + } else { + paid_only_entry(hinter) + }; + queues.add_pending_verify(key, entry); + } + } + 6 => { + if rng.gen_range(0..10) == 0 { + queues.remove_hint_source(&sources[rng.gen_range(0..sources.len())]); + } + } + 7 => { + if let Some(candidate) = queues.dequeue_fetch() { + match candidate.sources.first().copied() { + Some(source) if rng.gen_bool(0.5) => { + queues.start_dequeued_fetch(candidate, source); + } + _ => queues.discard_fetch_candidate(candidate), + } + } + } + _ => { + // A source answers, or a request to it goes unanswered, + // applied a little out of sequence order. + if let Some(source) = sources.get(rng.gen_range(0..sources.len())).copied() { + sequence += 1; + let place = sequence.saturating_sub(rng.gen_range(0..4)); + let contact = if rng.gen_bool(0.5) { + Contact::Answered { order: place } + } else { + Contact::Unreachable { sent: place } + }; + queues.record_fetch_contact(source, contact); + } + } + } + assert_queue_invariants(&queues); + assert_health_invariants(&queues.source_health); + assert_retry_entries_bounded(&queues); + for (key, entry) in &queues.in_flight_fetch { + let associated: HashSet<&PeerId> = + entry.all_sources.iter().chain(entry.tried.iter()).collect(); + assert!( + associated.len() <= 5, + "step {step}: {} over the source limit", + hex::encode(key) + ); + } + for (key, payload) in &queues.fetch_payloads { + let tried = queues.fetch_tried.get(key).map_or(0, HashSet::len); + assert!( + payload.sources.len() + tried <= 5, + "step {step}: queued over the limit" + ); + } + } + } + + /// Scale benchmark for the per-source scheduler at the fetch queue's hard + /// bound. Runs only when `ANT_FETCH_SCALE_BENCH` names a sources-per-key + /// count (it allocates hundreds of MiB and takes seconds); otherwise it + /// returns at once. One count per process, so `/usr/bin/time -l` reports + /// that scenario's peak memory: + /// + /// `ANT_FETCH_SCALE_BENCH=8 cargo test --release --features test-utils --lib + /// scale_benchmark -- --nocapture` + /// + /// Thresholds, declared before the release runs: one pass parking every key + /// holds the queue for at most 1 s at up to 8 sources per key and at most + /// 5 s at the 35-source limit; waking, the stats read and a departure each + /// stay under 250 ms; the process's peak RSS is at most 1 GiB at 8 sources + /// per key and 3 GiB at 35. Dropping the queues was first held to 250 ms + /// too, but measured 274 to 315 ms at 35 sources per key, and 226 to 295 ms + /// with the scheduler before the silent-source handling. It happens once, + /// when the engine shuts down, outside any lock, so it is held to 1 s. + #[test] + fn scale_benchmark_at_the_fetch_queue_bound() { + let Some(sources_per_key) = std::env::var("ANT_FETCH_SCALE_BENCH") + .ok() + .and_then(|raw| raw.parse::().ok()) + else { + return; + }; + let park_limit = if sources_per_key <= 8 { + Duration::from_secs(1) + } else { + Duration::from_secs(5) + }; + { + let other_limit = Duration::from_millis(250); + let teardown_limit = Duration::from_secs(1); + let sources: Vec = (0..sources_per_key) + .map(|index| { + let mut bytes = [0u8; 32]; + bytes[..8].copy_from_slice(&(index as u64 + 1).to_be_bytes()); + PeerId::from_bytes(bytes) + }) + .collect(); + let mut queues = ReplicationQueues::new(); + // Saturate every source with this node's own fetches first. + let mut blockers = Vec::new(); + for (index, source) in sources.iter().enumerate() { + for offset in 0..CAP { + let key = key_at(u32::MAX - u32::try_from(index * CAP + offset).unwrap()); + queues.start_fetch(key, *source, vec![*source]); + blockers.push(key); + } + } + let mut distance = 0u32; + while queues.fetch_queue_count() < MAX_FETCH_QUEUE { + let key = key_at(distance); + assert!(queues.enqueue_fetch(key, key, sources.clone())); + distance += 1; + } + + let started = Instant::now(); + let dispatch = queues.dispatch_fetches(8, CAP); + let park_pass = started.elapsed(); + assert_eq!(dispatch.started, Vec::new()); + assert_eq!(queues.fetch_parked.len(), MAX_FETCH_QUEUE); + + let associations: usize = queues.parked_by_source.values().map(BTreeSet::len).sum(); + + let started = Instant::now(); + let stats = queues.take_fetch_dispatch_stats(CAP); + let stats_pass = started.elapsed(); + assert_eq!(stats.parked, MAX_FETCH_QUEUE); + + let started = Instant::now(); + queues.complete_fetch(&blockers[0]); + let wake = queues.dispatch_fetches(8, CAP); + let wake_pass = started.elapsed(); + assert_eq!(wake.started.len(), 1); + + let started = Instant::now(); + queues.remove_hint_source(&sources[0]); + let departure = started.elapsed(); + + let started = Instant::now(); + drop(queues); + let teardown = started.elapsed(); + + println!( + "FETCH-SCALE sources_per_key={sources_per_key} keys={MAX_FETCH_QUEUE} \ + associations={associations} park_pass_ms={} stats_ms={} wake_ms={} \ + departure_ms={} teardown_ms={}", + park_pass.as_millis(), + stats_pass.as_millis(), + wake_pass.as_millis(), + departure.as_millis(), + teardown.as_millis() + ); + assert!( + park_pass <= park_limit, + "park pass {park_pass:?} over {park_limit:?}" + ); + for (name, took) in [ + ("stats", stats_pass), + ("wake", wake_pass), + ("departure", departure), + ] { + assert!( + took <= other_limit, + "{name} took {took:?}, over {other_limit:?}" + ); + } + assert!( + teardown <= teardown_limit, + "teardown took {teardown:?}, over {teardown_limit:?}" + ); + } + } } diff --git a/tests/e2e/fetch_per_source_share.rs b/tests/e2e/fetch_per_source_share.rs new file mode 100644 index 00000000..7b9b87ef --- /dev/null +++ b/tests/e2e/fetch_per_source_share.rs @@ -0,0 +1,328 @@ +//! A fetching node keeps no more of its fetches in flight to one answering +//! holder than that holder will admit from it. +//! +//! A fetch responder admits two outstanding requests per requesting peer and +//! answers a third with nothing at all, so a requester that sends more than two +//! at once to one holder loses the rest to its own fetch timeout. The fetch +//! worker used to do exactly that: it ran one fetch per slot and sent every one +//! to the first verified holder of its key, so a node whose queued keys shared +//! one holder had all of them outstanding there at once. +//! +//! The observation is made on the requester, which is what the scheduler +//! governs, and made deterministic by holding the holder's work in place: the +//! holder's test-only serve gate holds every fetch request it admits before +//! serving it, so whatever the requester sends stays outstanding for as long as +//! the test looks at it, however fast the host serves a chunk. Every node runs +//! eight fetch slots, more than one holder's share on any host. With the +//! per-source limit removed the requester holds all eight in flight to the one +//! holder and this test fails. Both hooks are per engine, so no other node or +//! test is affected. +//! +//! The limit is a share of a holder that answers. A holder whose requests go +//! unanswered has no share to keep within, and keeping its keys at two per +//! fetch timeout would drain them four times more slowly than before the limit. +//! The second test closes the holder's gate for good, with a short fetch +//! timeout, and checks that once the first requests time out the requester +//! sends past the share and the keys leave its fetch pipeline in a bounded time. +//! +//! Both tests tear the network down with the holder's gate still closed and +//! requests held at it, which the holder's engine shutdown must release. + +#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] + +use super::testnet::TestNetworkConfig; +use super::verify_storage_gate::find_key_for_target; +use super::TestHarness; +use ant_node::client::XorName; +use ant_node::replication::config::storage_admission_width; +use ant_node::{ReplicationConfig, ReplicationEngine}; +use saorsa_core::identity::PeerId; +use serial_test::serial; +use std::collections::HashSet; +use std::time::{Duration, Instant}; + +/// Above `storage_admission_width` (9), so the network has a real close group. +const NODE_COUNT: usize = 12; +/// Production close-group size. +const CLOSE_GROUP_SIZE: usize = 7; +/// Node whose fetch worker is observed. +const REQUESTER_INDEX: usize = 5; +/// The one node hosting every chunk. +const HOLDER_INDEX: usize = 6; +/// More keys than fetch slots, all held by the one holder. +const KEY_COUNT: usize = 12; +/// Fetch slots per node, above the per-source share on every host. +const FETCH_CONCURRENCY: usize = 8; +/// The responder's per-peer fetch share. +const PER_SOURCE_SHARE: usize = 2; +/// How long the requester's fetch pipeline must stay empty before the test +/// starts, so earlier traffic cannot occupy its slots. +const QUIET_FOR: Duration = Duration::from_secs(1); +/// Upper bound on waiting for that quiet period. +const QUIET_DEADLINE: Duration = Duration::from_secs(30); +/// How long the requester is observed with the gate closed. Far below the 30 s +/// fetch timeout, so no held request times out and frees its slot early. +const OBSERVATION_WINDOW: Duration = Duration::from_secs(3); +const OBSERVATION_POLL: Duration = Duration::from_millis(50); +/// The silent-holder test's fetch timeout, short so silence is seen in seconds. +const SILENT_FETCH_TIMEOUT: Duration = Duration::from_secs(5); +/// Keys held only by the silent holder: two fetch slots' worth. +const SILENT_KEY_COUNT: usize = 16; +/// The longest the silent holder's keys may take to leave the requester's +/// fetch pipeline: four fetch timeouts. This scheduler takes three, one more +/// than fetching as before the limit, to see the silence; keeping the holder +/// at its share would take eight (sixteen keys, two per timeout). The exact +/// three waves are checked without timing in the scheduler's unit tests. +const SILENT_DRAIN_LIMIT: Duration = Duration::from_secs(20); +/// The longest tearing the network down may take with requests still held at +/// the holder's closed serve gate. Without the release at engine shutdown it +/// waits for them for ever. +const TEARDOWN_LIMIT: Duration = Duration::from_secs(120); + +#[tokio::test] +#[serial] +async fn requester_keeps_at_most_its_share_in_flight_to_one_holder() { + let max_in_flight = max_in_flight_to_one_holder().await; + assert!( + max_in_flight <= PER_SOURCE_SHARE, + "the requester had {max_in_flight} fetches outstanding at one holder, which \ + admits {PER_SOURCE_SHARE} from it and answers the rest with nothing" + ); + assert_eq!( + max_in_flight, PER_SOURCE_SHARE, + "the requester never filled the holder's share, so the run says nothing \ + about the limit: the worker did not dispatch" + ); +} + +#[tokio::test] +#[serial] +async fn a_holder_that_stops_answering_is_fetched_from_past_its_share() { + let harness = start_network(SILENT_FETCH_TIMEOUT).await; + let chunks = chunks_for_requester(&harness, "silent-holder", SILENT_KEY_COUNT).await; + let keys: HashSet = chunks.iter().map(|(_, key)| *key).collect(); + + let requester = harness.test_node(REQUESTER_INDEX).expect("requester"); + let requester_engine = requester + .replication_engine + .as_ref() + .expect("requester engine"); + let holder = harness.test_node(HOLDER_INDEX).expect("holder"); + let holder_peer = *holder.p2p_node.as_ref().expect("holder p2p").peer_id(); + let holder_engine = holder.replication_engine.as_ref().expect("holder engine"); + + wait_until_fetch_pipeline_is_quiet(requester_engine).await; + + // The holder answers none of these fetches, ever. + holder_engine.close_fetch_serve_gate_for_test(); + let started = Instant::now(); + for key in &keys { + assert!( + requester_engine + .enqueue_fetch_for_test(*key, vec![holder_peer]) + .await, + "every key must be queued for the silent holder alone" + ); + } + + let mut max_in_flight_to_holder = 0; + let drained_after = loop { + let snapshot = requester_engine.fetch_pipeline_snapshot_for_test().await; + let in_flight_to_holder = snapshot + .in_flight + .iter() + .filter(|(key, source)| keys.contains(key) && *source == holder_peer) + .count(); + max_in_flight_to_holder = max_in_flight_to_holder.max(in_flight_to_holder); + let pending = snapshot + .queued + .iter() + .chain(snapshot.in_flight.iter().map(|(key, _)| key)) + .filter(|key| keys.contains(*key)) + .count(); + if pending == 0 { + break started.elapsed(); + } + assert!( + started.elapsed() < SILENT_DRAIN_LIMIT, + "{pending} of {SILENT_KEY_COUNT} keys held only by a silent holder were still \ + in the fetch pipeline after {SILENT_DRAIN_LIMIT:?}" + ); + tokio::time::sleep(OBSERVATION_POLL).await; + }; + println!( + "SILENT-HOLDER-RESULT drained_after={drained_after:?} \ + max_in_flight_to_holder={max_in_flight_to_holder} timeout={SILENT_FETCH_TIMEOUT:?}" + ); + + teardown_with_the_gate_closed(harness).await; + assert!( + drained_after < SILENT_DRAIN_LIMIT, + "the silent holder's keys took {drained_after:?} to leave the fetch pipeline" + ); + assert!( + max_in_flight_to_holder > PER_SOURCE_SHARE, + "the requester kept a holder that answers nothing at its share of \ + {PER_SOURCE_SHARE}, which drains that holder's keys at two per timeout" + ); +} + +/// Build the network with `fetch_request_timeout` and warm its DHT. +async fn start_network(fetch_request_timeout: Duration) -> TestHarness { + let config = TestNetworkConfig { + node_count: NODE_COUNT, + replication_config: Some(ReplicationConfig { + close_group_size: CLOSE_GROUP_SIZE, + fetch_request_timeout, + ..ReplicationConfig::default() + }), + fetch_concurrency: Some(FETCH_CONCURRENCY), + ..TestNetworkConfig::default() + }; + let harness = TestHarness::setup_with_config(config) + .await + .expect("setup per-source share network"); + harness.warmup_dht().await.expect("warmup"); + harness +} + +/// `count` distinct chunks the requester is responsible for storing. +async fn chunks_for_requester( + harness: &TestHarness, + label: &str, + count: usize, +) -> Vec<(Vec, XorName)> { + let width = storage_admission_width(CLOSE_GROUP_SIZE); + let mut chunks = Vec::with_capacity(count); + for index in 0..count { + let label = format!("{label}-{index}"); + chunks.push(find_key_for_target(harness, REQUESTER_INDEX, width, true, &label).await); + } + let keys: HashSet<&XorName> = chunks.iter().map(|(_, key)| key).collect(); + assert_eq!(keys.len(), count, "every key must be distinct"); + chunks +} + +/// Build the network, direct twelve keys at one holder while that holder holds +/// what it admits, and return the most fetches the requester had in flight to +/// it. +async fn max_in_flight_to_one_holder() -> usize { + let harness = start_network(ReplicationConfig::default().fetch_request_timeout).await; + let chunks = chunks_for_requester(&harness, "per-source-share", KEY_COUNT).await; + let keys: HashSet = chunks.iter().map(|(_, key)| *key).collect(); + + let requester = harness.test_node(REQUESTER_INDEX).expect("requester"); + let requester_engine = requester + .replication_engine + .as_ref() + .expect("requester engine"); + let holder = harness.test_node(HOLDER_INDEX).expect("holder"); + let holder_peer = *holder.p2p_node.as_ref().expect("holder p2p").peer_id(); + let holder_engine = holder.replication_engine.as_ref().expect("holder engine"); + let holder_storage = holder + .ant_protocol + .as_ref() + .expect("holder protocol") + .storage(); + + // Start from an idle requester, so its fetch slots are all free. + wait_until_fetch_pipeline_is_quiet(requester_engine).await; + + // Hold every fetch the holder admits, then make the chunks available. + holder_engine.close_fetch_serve_gate_for_test(); + for (content, key) in &chunks { + holder_storage.put(key, content).await.expect("host chunk"); + } + let before = requester_engine.fetch_pipeline_snapshot_for_test().await; + let unrelated: Vec = before + .queued + .iter() + .chain(before.in_flight.iter().map(|(key, _)| key)) + .filter(|key| !keys.contains(*key)) + .copied() + .collect(); + assert!( + unrelated.is_empty(), + "unrelated fetch work could occupy the requester's slots: {unrelated:?}" + ); + for key in &keys { + // A replica hint may already have queued the key; either way it is + // headed to the one holder. + let _ = requester_engine + .enqueue_fetch_for_test(*key, vec![holder_peer]) + .await; + } + + let max_in_flight_to_holder = + observe_max_in_flight_to(requester_engine, &keys, &holder_peer).await; + println!( + "PER-SOURCE-SHARE-RESULT max_in_flight_to_holder={max_in_flight_to_holder} \ + concurrency={FETCH_CONCURRENCY} keys={KEY_COUNT}" + ); + + teardown_with_the_gate_closed(harness).await; + max_in_flight_to_holder +} + +/// Tear the network down while the holder's serve gate is still closed. +async fn teardown_with_the_gate_closed(harness: TestHarness) { + let torn_down = tokio::time::timeout(TEARDOWN_LIMIT, harness.teardown()).await; + assert!( + matches!(torn_down, Ok(Ok(()))), + "teardown with requests held at a closed serve gate did not finish within \ + {TEARDOWN_LIMIT:?}: {torn_down:?}" + ); +} + +/// Wait until `engine` has nothing queued or in flight for fetch, continuously +/// for [`QUIET_FOR`]. +async fn wait_until_fetch_pipeline_is_quiet(engine: &ReplicationEngine) { + let deadline = tokio::time::Instant::now() + QUIET_DEADLINE; + let mut quiet_since = tokio::time::Instant::now(); + loop { + let snapshot = engine.fetch_pipeline_snapshot_for_test().await; + let now = tokio::time::Instant::now(); + if !snapshot.queued.is_empty() || !snapshot.in_flight.is_empty() { + quiet_since = now; + } else if now.duration_since(quiet_since) >= QUIET_FOR { + return; + } + assert!( + now < deadline, + "the requester's fetch pipeline never went quiet: {snapshot:?}" + ); + tokio::time::sleep(OBSERVATION_POLL).await; + } +} + +/// The most of `keys` that `engine` had in flight to `holder` at once over +/// [`OBSERVATION_WINDOW`], failing if any other fetch work appears. +async fn observe_max_in_flight_to( + engine: &ReplicationEngine, + keys: &HashSet, + holder: &PeerId, +) -> usize { + let deadline = tokio::time::Instant::now() + OBSERVATION_WINDOW; + let mut max_in_flight = 0; + while tokio::time::Instant::now() < deadline { + let snapshot = engine.fetch_pipeline_snapshot_for_test().await; + let unrelated: Vec<&XorName> = snapshot + .queued + .iter() + .chain(snapshot.in_flight.iter().map(|(key, _)| key)) + .filter(|key| !keys.contains(*key)) + .collect(); + assert!( + unrelated.is_empty(), + "unrelated fetch work appeared during the observation: {unrelated:?}" + ); + let to_holder = snapshot + .in_flight + .iter() + .filter(|(key, source)| keys.contains(key) && source == holder) + .count(); + max_in_flight = max_in_flight.max(to_holder); + tokio::time::sleep(OBSERVATION_POLL).await; + } + max_in_flight +} diff --git a/tests/e2e/mod.rs b/tests/e2e/mod.rs index ec2a74c7..295b022d 100644 --- a/tests/e2e/mod.rs +++ b/tests/e2e/mod.rs @@ -75,6 +75,9 @@ mod fetch_local_write_guard; #[cfg(test)] mod fetch_responsibility_recheck; +#[cfg(test)] +mod fetch_per_source_share; + #[cfg(test)] mod first_audit_ab; diff --git a/tests/e2e/testnet.rs b/tests/e2e/testnet.rs index ca33a901..051e42b4 100644 --- a/tests/e2e/testnet.rs +++ b/tests/e2e/testnet.rs @@ -230,6 +230,10 @@ pub struct TestNetworkConfig { /// use this to shorten timers — e.g. the ADR-0003 possession-check delay. pub replication_config: Option, + /// Fetches each node's worker runs at once. `None` uses one per hardware + /// thread. + pub fetch_concurrency: Option, + /// Optional per-node storage disk-reserve overrides. /// /// Tests can set a node's reserve above available disk space to make its @@ -283,6 +287,7 @@ impl Default for TestNetworkConfig { payment_enforcement: false, evm_network: None, replication_config: None, + fetch_concurrency: None, storage_disk_reserve_overrides: HashMap::new(), } } @@ -1388,6 +1393,9 @@ impl TestNetwork { .await { Ok(mut engine) => { + if let Some(concurrency) = self.config.fetch_concurrency { + engine.set_fetch_concurrency_for_test(concurrency); + } // Pointers replicate through the same engine (ADR-0016), // which the service also asks before a final state // (ADR-0018).