diff --git a/docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md b/docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md new file mode 100644 index 00000000..909a67f8 --- /dev/null +++ b/docs/adr/ADR-0024-fetch-requesters-stay-within-the-responder-share.md @@ -0,0 +1,282 @@ +# ADR-0024: Fetch requesters stay within the responder's per-peer share + +- **Status:** Proposed +- **Date:** 2026-10-13 +- **Decision owners:** +- **Reviewers:** +- **Supersedes:** none +- **Superseded by:** none +- **Related:** [ADR-0005](ADR-0005-replication-repair-hardening.md) (the repair fetch pipeline), + [ADR-0017](ADR-0017-bounded-fresh-offers-and-copy-free-sends.md) (send-path bounds) + +## Context + +A replication fetch is a request/response exchange between two nodes. The responder bounds what +one requester can occupy: + +- `FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER = 2` admitted requests per source, checked before the + 12-slot admission pool, in front of a 3-worker serving pool; +- a request past that share is not answered. The responder logs + `Fetch response dropped at admission: reason=per_peer_cap_full` and returns nothing. + +The requesting side never matched that share: + +- the fetch worker ran one fetch per hardware thread; +- it sent each one to the first verified holder of its key, in source-preference order (the + closest holder that answered Present); +- the fetch queue is nearest-first, so the keys at its head usually share the same few holders. + +When more than two fetch slots were available and several queued keys shared a holder, the worker +could send more than that holder's two-request share. A request arriving while two from the same +requester were still admitted was dropped. Each such drop cost the requester: + +- the full fetch timeout (30 s) before moving on; +- a `ConnectionTimeout` trust event, recorded by the transport against the holder that dropped it, + although that holder had done nothing wrong; +- a retry that went to the next holder without regard to how busy that holder was. + +On the production fleet this was common, according to the per-service cumulative counters: + +| period | inbound fetch requests dropped at admission | +|---|---| +| on 0.21.0 | 36.6% | +| on 0.22.0 | 44.3% | +| 25-hour window containing a wave of 177 freshly installed nodes filling their ranges | 65.8% overall by cumulative counters; 3.22M WARN-line drops came from the new nodes, about 2.8 per chunk attributed to them | + +In that window, 68.5% of drops happened with only 2 of the responder's 12 admission slots in use, +both held by the refused peer. + +## Decision Drivers + +- No wire, storage or production public API change, so the fix ships as a behavioural change on + the release train. +- Do not weaken the responder's protection against one peer occupying the whole serving pool. +- A key must never be lost, tracked twice, or released from bootstrap accounting early because of + scheduling. +- Keep the queue's nearest-first priority and each key's source preference whenever they do not + conflict with the share. +- No scenario may drain more slowly than before the change by more than one fetch timeout, in + particular the keys of a holder that has stopped answering. + +## Considered Options + +1. **Raise the responder's per-peer share.** Rejected. It hides the symptom, lets one requester take + all three serving workers, which is what the share exists to prevent, and still drops requesters + with more threads than the new share. +2. **Answer an over-share request with an explicit "busy" reply.** Deferred. It needs a new + response variant, which peers that cannot decode it treat as malformed and charge an application + failure, so it is a wire change with its own rollout. It removes the cost of a drop, not the + cause. +3. **Limit fetch concurrency to 2.** Rejected. It serialises a node's whole repair and bootstrap fill + through one share even when other holders are idle. +4. **Ship the scheduler switched off, behind a per-node switch, for a later canary.** Rejected. It + merges code that no node runs, so its risks are deferred rather than measured. The risks were + simulated instead (below), and the one regression found was fixed before merge. +5. **Keep at most the share in flight per answering holder on the requesting side, and give each + free slot to the nearest queued key that can start now.** Chosen. + +## Decision + +The fetch worker keeps at most `FETCH_MAX_IN_FLIGHT_PER_SOURCE` fetches in flight to any one source +peer that is answering. The constant is defined equal to `FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER`, +the share every deployed responder grants. It is the worker's only scheduling path. + +**Dispatch.** Each free fetch slot goes to the nearest queued key that can start now. In the key's +source-preference order, over its untried sources, it starts on: + +1. the first source that is answering and below the limit; +2. otherwise the first silent source (below), past the limit. + +**Parking.** + +- A key whose untried sources are all answering and at the limit is parked: it leaves the fetch + heap and is indexed under each of those sources. +- It becomes a candidate again as soon as any of them has a free share or falls silent, and it + competes with the heap on distance. +- Each key leaves the heap once, so a startable key further down is reached in the same pass, and + no source ordering can hide a startable key. + +**Retries.** + +- A failed fetch moves to its next untried source by the same rule. If every untried source is + answering and at the limit, it is parked rather than sent to a full holder. +- It keeps the sources it already tried, so none is asked twice in one verification cycle, and it + keeps its verification retry reservation. +- When no untried source is left, or the fetch queue is full, it returns to verification as before, + so the fetch queue's hard bound holds. + +**Silent sources.** The limit protects a holder's admission share, and only a holder that answers +has one. Without an exception, the keys held only by a holder that stopped answering would be tried +two per fetch timeout instead of one per free slot: four times more slowly than before on a node +with eight slots. + +- Each attempt records whether its source answered. Requests and answers take places in one + process-wide sequence, as a request is sent and as an answer returns, so they can be ordered + exactly, whatever order their outcomes are applied in and whatever the clock reads. +- An answer reaches the scheduler as soon as it arrives, before its chunk is checked and stored, + and again with the attempt's outcome, to no further effect. A timeout that an answer already + outdates therefore never marks the source while that answer's chunk is still being written. +- A source is **silent** while a request sent to it after its latest answer has gone without any + response (a timeout or connection failure). Any answer received after that request was sent ends + it. +- A silent source is fetched from past the limit, as before the change, until it answers again, + or until this node has learned of no newer unanswered request to it for two fetch timeouts (at + least one second). A holder that stays dead while this node keeps fetching from it therefore + stays silent, and one this node stopped fetching from is probed under the limit when it is next + used, marked again only if a request sent then also goes unanswered. +- The table of observations is bounded at 8,192 sources. It drops only entries that hold no mark and + whose source has no fetch in flight. If that frees nothing, a source not in the table counts as + silent for one mark lifetime, which is again the behaviour before the change, never a slower one. + +**Source limit.** + +- The sources a queued or in-flight key is associated with, tried ones included, are bounded by + `close_group_size + paid_list_close_group_size + MAX_HINT_SOURCES_PER_KEY` from the running + configuration. +- That never truncates a verification round's own sources, and it bounds the parked index. +- A re-advertisement of a queued or in-flight key merges into the verification entry kept for its + retry under `MAX_HINT_SOURCES_PER_KEY`, the bound a pending entry has, in the same order: replica + claimants first, displacing paid-only advertisers other than the key's capacity owner, then + paid-only advertisers while there is room. Before the change that merge had no bound. + +**Dispatch summary.** The fetch worker logs `replication fetch dispatch summary` every 5 minutes and +at shutdown, with: + +- the parked count, the index size and the oldest parked age; +- queued and in-flight fetches, and sources at the limit; +- parks and retry parks; +- attempts by outcome, bytes stored, and attempt latency; +- the longest fill-pass lock wait and hold, and the longest parked-index maintenance after a + departure; +- silent sources now and at peak, marks, clears, expiries, ignored stale timeouts, starts on silent + sources, unreachable attempts, tracked sources and table overflows. + +Responder admission, the fetch timeout, fetch concurrency and the wire protocol are unchanged. + +## Risk simulation + +A local 12-node network, eight fetch slots per node and a 5 s fetch timeout, timed each scenario on +this change and on a build with the limit removed and no silent-source handling, which schedules +exactly as before the change: each key to its first untried source. + +| scenario | this change | before the change | +|---|---|---| +| fill: 48 keys held by 3 answering holders | 0.36 s | 10.08 s | +| a non-answering holder first among 3 sources, 2 answering: 24 keys | 5.1 s | 20.1 s | +| a non-answering sole holder: 16 keys | 15.0 s | 10.15 s | + +The third row is the accepted penalty: the first two requests to a holder that just went silent take +one fetch timeout to show it, and the rest then go out past the limit. It is paid once each time +this node starts fetching from a silent holder again after two fetch timeouts without trying it. +For `N` keys held only by that holder, `C ≥ 2` fetch slots and fetch timeout `T`, draining takes +`T × (1 + ceil((N − 2) / C))` against `T × ceil(N / C)` before the change. With one fetch slot the +first wave is one request, and the bound is the same with 1 in place of 2. Without the +silent-source exception the same scenario took 40.1 s. + +The scale benchmark at the fetch queue's hard bound (`scale_benchmark_at_the_fetch_queue_bound`, +release build, 131,072 parked keys) stayed within the thresholds declared before it ran, apart from +dropping the queues (below): + +| sources per key | one park pass, over repeated runs | peak process RSS | +|---|---|---| +| 1 | 89 to 99 ms | 197 MiB | +| 8 | 207 to 216 ms | 342 MiB | +| 35 (the limit) | 754 to 960 ms | 1.13 GiB | + +- Waking and the stats read took under 1 ms; a departure took at most 107 ms. +- Dropping the queues at the limit took 274 to 315 ms over repeated runs. The scheduler before the + silent-source handling took 226 to 295 ms, and the same time as this code in alternating runs on + one machine. It was first held to 250 ms, which a loaded machine did not meet. It happens once, + when the engine shuts down, outside any lock, so it is now held to 1 s. +- The silent-source check makes the park pass about 5% slower than before. + +## Consequences + +### Positive + +- A requester on this release keeps its own in-flight fetches to one answering holder within that + holder's share. That removes the ordinary over-share bursts and their timeout and trust cost; it + does not cover stale work the responder still holds, described below. +- A fill or repair can finish sooner, because fetch slots no longer spend 30 s on over-share + requests that will not be answered. +- Responders should log fewer `per_peer_cap_full` drops as requesters upgrade. +- The dispatch summary shows the scheduler's queue state, lock times and silent sources in + production. + +### Negative / Trade-offs + +- It is not a guarantee of zero refusals. A request this node has already timed out on can still + hold its share at the responder until a worker sheds it, and a new request sent in that window + is refused. Closing that window needs option 2. During the 25-hour wave, responders shed 31,172 + admitted requests after waiting 30 s for a worker; the production data do not show how often + that window caused a later refusal. +- The requester counts a fetch as in flight for its whole attempt, local checks and the local write + included, which is longer than the responder holds its share. A holder can therefore sit below + its share while this node waits, and a fetch whose local work stalls holds its holder's share for + as long as it holds its fetch slot. +- A key whose answering holders are all busy waits in the queue, or goes to a lower-ranked holder + with room, instead of being sent to its first holder. +- Keys held only by a holder that just stopped answering take one fetch timeout longer to drain + than before the change, as in the simulation. A holder that is alive but so overloaded that it + times out is fetched from as before the change while its mark lasts. +- Recovering the slots dropped requests used to waste raises a filling node's instantaneous + bandwidth, disk and CPU use. Each responder still serves at most its share per requester with + three workers; the data a joining node needs to move is unchanged. +- At the source limit, the parked index can reach 4.6 million entries and about 1.1 GiB of process + memory; at 8 sources per key, about 340 MiB. +- Requesters on older releases still overshoot until they upgrade, so the network-wide drop rate + falls only as the release spreads, and the traffic pattern changes during that time. +- There is no runtime switch. Undoing the change means a release without it. + +### Neutral / Operational + +- The effect shows in the responders' cumulative counters (`replication responder admission + summary`): `fetch_admission_dropped_per_peer_share` relative to `fetch_request_rx_count` should + fall as the fleet and the community upgrade. +- Before release, a dev testnet run with a join wave of fresh nodes should compare drain time, + stored bytes, admission drops, trust events and host load against the previous release. + +## Validation + +**Unit tests on the queue:** + +- no answering source exceeds the share; +- preference order is kept; +- a startable key behind hundreds of blocked keys starts in the same pass; +- a freed share starts the nearest parked key, whatever its source; +- parked and retried keys keep their payload, tried sources and retry reservation; +- departures and merges keep the parked index exact; +- the source limit holds across park, retry and merge; +- a silent source is fetched from past the share, an answering source with room is preferred, and + parked keys start when a full source falls silent; +- whether a source is silent does not depend on the order its outcomes are applied in; marks are + set, cleared and expired as described, and the table bound keeps every guard and mark; +- an answer reported when it arrives keeps a source within its share when an earlier request to it + times out while the answered fetch is still storing its chunk; known only from the outcome, the + same answer comes too late and the source is sent every waiting key; +- re-advertisements keep a queued or in-flight key's retry entry within the hint bound, with its + capacity owner; +- sixteen keys held only by a silent holder, with eight slots, start in exactly three waves of 2, 8 + and 6; +- a randomised run of queue operations, re-advertisements and source contacts keeps every + invariant, the retry entries' bound included. + +**End-to-end tests:** + +- with eight fetch slots and twelve keys directed to one holder that holds what it admits, the + requester's own pipeline records at most two in-flight fetches to that holder; with the limit + removed it records eight and the test fails; +- with a 5 s fetch timeout and sixteen keys directed to a holder that answers nothing, the requester + sends past the share once the first requests time out and the keys leave its pipeline within four + timeouts (15.1 s measured); with the silent-source exception removed the test fails. + +**Revisit if:** + +- the responder's per-peer share changes; +- a busy reply is introduced; +- the fleet counters keep showing a high dropped share once most requesters run this release; +- the dispatch summary shows sustained table overflows or a large count of silent sources. + +## Notes for AI-assisted work + +AI tools may help draft this ADR, but **must not mark it Accepted without human review**. Accepted ADRs are immutable: create a new superseding ADR rather than editing an Accepted ADR. diff --git a/docs/adr/README.md b/docs/adr/README.md index b58e9a32..bcde1f6f 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -40,3 +40,4 @@ See [`TOOLING.md`](./TOOLING.md) for `adrs`, `adr-kit`, and AI harness setup. - [ADR-0018: Pointer ownership transfer by final redirection](./ADR-0018-pointer-transfer-by-final-redirection.md) - [ADR-0019: Pointer audits serve the record round 1 bound](./ADR-0019-pointer-audits-serve-the-record-round-one-bound.md) - [ADR-0022: Remove the LMDB Chunk Store](./ADR-0022-remove-the-lmdb-chunk-store.md) +- [ADR-0024: Fetch requesters stay within the responder's per-peer share](./ADR-0024-fetch-requesters-stay-within-the-responder-share.md) diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 62937413..2c50bd27 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -42,6 +42,8 @@ use std::fmt; use std::num::NonZeroUsize; use std::ops::ControlFlow; use std::path::{Path, PathBuf}; +#[cfg(any(test, feature = "test-utils"))] +use std::sync::atomic::AtomicBool; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; use std::time::{Duration, Instant, SystemTime}; @@ -62,6 +64,7 @@ use tokio_util::sync::CancellationToken; use tokio_util::task::TaskTracker; use crate::ant_protocol::XorName; +use crate::client::xor_distance; use crate::error::{Error, Result}; use crate::payment::{ PaymentVerifier, VerificationContext, MAX_PAYMENT_PROOF_SIZE_BYTES, @@ -95,7 +98,12 @@ use crate::replication::protocol::{ }; use crate::replication::quorum::KeyVerificationOutcome; use crate::replication::recent_provers::RecentProvers; -use crate::replication::scheduling::{CapacityDisplacement, DeferralOutcome, ReplicationQueues}; +#[cfg(any(test, feature = "test-utils"))] +pub use crate::replication::scheduling::FetchPipelineSnapshot; +use crate::replication::scheduling::{ + fetch_source_limit, CapacityDisplacement, Contact, DeferralOutcome, FetchDispatchStats, + FetchRetry, ReplicationQueues, +}; use crate::replication::storage_commitment_audit::PointerBindings; use crate::replication::types::{ AuditFailureReason, BootstrapClaimObservation, BootstrapState, FailureEvidence, @@ -116,6 +124,16 @@ use saorsa_pqc::api::sig::{MlDsaSecretKey, MlDsaVariant}; /// counted: it is not a saturation signal. static FIRST_AUDIT_INGRESS_DROPPED: AtomicU64 = AtomicU64::new(0); +/// One sequence over every fetch request this process sends and every answer +/// it receives, so the fetch scheduler can order a request against an answer +/// even when both read the same clock value. See [`Contact`]. +static FETCH_CONTACT_SEQUENCE: AtomicU64 = AtomicU64::new(0); + +/// The next place in [`FETCH_CONTACT_SEQUENCE`]. +fn next_fetch_contact() -> u64 { + FETCH_CONTACT_SEQUENCE.fetch_add(1, Ordering::Relaxed) +} + /// Record one ingress-full drop. Called by the payment verifier's `try_send` /// sites; read by the drainer's periodic summary. pub(crate) fn note_monetized_ingress_drop() { @@ -1129,6 +1147,24 @@ const FETCH_RESPONDER_MAX_OUTSTANDING: usize = FETCH_RESPONDER_WORKER_LIMIT * 4; /// of the three workers available to other peers under a single-source flood. const FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER: u32 = 2; +/// Maximum fetches this node keeps in flight to any one source peer that is +/// answering. +/// +/// The share every responder grants one requester +/// ([`FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER`], unchanged since it was +/// introduced). A request past that share is dropped without an answer, so +/// sending it costs the full fetch timeout and a timeout trust event against a +/// holder that did nothing wrong. Staying within it removes those bursts; it +/// cannot rule out every refusal, because a request this node has already +/// timed out on may still be holding its share at the responder until a worker +/// sheds it. +/// +/// A source whose request got no answer, and which has not answered since that +/// request was sent, has no share to stay within. It is fetched from without +/// this cap, as before the cap existed, until it answers again or two fetch +/// timeouts pass, so a holder that went silent does not hold its keys back. +const FETCH_MAX_IN_FLIGHT_PER_SOURCE: usize = FETCH_RESPONDER_MAX_OUTSTANDING_PER_PEER as usize; + /// Maximum verification batches served concurrently. /// /// Point lookups are fast, but a batch can contain 8,192 of them. Two @@ -1222,7 +1258,9 @@ fn paid_notify_payment_context() -> VerificationContext { } /// Boxed future type for in-flight fetch tasks. -type FetchFuture = Pin)> + Send>>; +/// A fetch task's key, its outcome (`None` when the task panicked), and when +/// the task finished. +type FetchFuture = Pin, Instant)> + Send>>; /// Fresh-offer keys currently being handled, each with the proofs still to try. /// @@ -1667,6 +1705,8 @@ pub struct ReplicationEngine { payment_verifier: Arc, /// Replication pipeline queues. queues: Arc>, + /// Fetches the worker runs at once: one per hardware thread. + fetch_concurrency: usize, /// Neighbor sync cycle state. sync_state: Arc>, /// Per-peer sync history (for `RepairOpportunity`). @@ -1787,6 +1827,9 @@ pub struct ReplicationEngine { fetch_responder_admission_semaphore: Arc, /// Per-source fetch responder counts for flood-fair admission. fetch_responder_inflight: Arc>>, + /// Test-only hold on serving admitted fetch requests. + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc, /// Worker permits for lookup-heavy verification batches. verification_responder_worker_semaphore: Arc, /// Admission permits bounding verification workers and queued waiters. @@ -1902,13 +1945,21 @@ impl ReplicationEngine { let (monetized_pin_tx, monetized_pin_rx) = mpsc::channel(config::FIRST_AUDIT_INGRESS_CAPACITY); + let source_limit = + fetch_source_limit(config.close_group_size, config.paid_list_close_group_size); + let engine = Self { config: Arc::clone(&config), p2p_node, storage, paid_list, payment_verifier, - queues: Arc::new(RwLock::new(ReplicationQueues::new())), + queues: Arc::new(RwLock::new( + ReplicationQueues::new() + .with_fetch_source_limit(source_limit) + .with_fetch_request_timeout(config.fetch_request_timeout), + )), + fetch_concurrency: max_parallel_fetch(), sync_state: Arc::new(RwLock::new(initial_neighbors)), sync_history: Arc::new(RwLock::new(HashMap::new())), audit_on_gossip_cooldown: Arc::new(RwLock::new(HashMap::new())), @@ -1940,6 +1991,8 @@ impl ReplicationEngine { FETCH_RESPONDER_MAX_OUTSTANDING, )), fetch_responder_inflight: Arc::new(RwLock::new(HashMap::new())), + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc::new(FetchServeGate::default()), verification_responder_worker_semaphore: Arc::new(Semaphore::new( VERIFICATION_RESPONDER_WORKER_LIMIT, )), @@ -2194,6 +2247,35 @@ impl ReplicationEngine { self.queues.read().await.fetch_queued_or_in_flight(key) } + /// Test-only: run `concurrency` fetches at once instead of one per hardware + /// thread. Call before [`Self::start`], which reads it. Per engine, so it + /// affects no other node or test. + #[cfg(any(test, feature = "test-utils"))] + pub fn set_fetch_concurrency_for_test(&mut self, concurrency: usize) { + self.fetch_concurrency = concurrency; + } + + /// Test-only: hold this engine's admitted fetch requests before they are + /// served, until [`Self::open_fetch_serve_gate_for_test`]. + #[cfg(any(test, feature = "test-utils"))] + pub fn close_fetch_serve_gate_for_test(&self) { + self.fetch_serve_gate.close(); + } + + /// Test-only: release this engine's held fetch requests and stop holding + /// new ones. + #[cfg(any(test, feature = "test-utils"))] + pub fn open_fetch_serve_gate_for_test(&self) { + self.fetch_serve_gate.open(); + } + + /// Test-only: every key queued for fetch and every in-flight fetch with + /// the source it was sent to, read under one lock. + #[cfg(any(test, feature = "test-utils"))] + pub async fn fetch_pipeline_snapshot_for_test(&self) -> FetchPipelineSnapshot { + self.queues.read().await.fetch_pipeline_snapshot() + } + /// Test-only: place `key` into pending verification as though `hinter` had /// just advertised it with a replica hint. Returns whether it was admitted. /// @@ -2434,6 +2516,12 @@ impl ReplicationEngine { } } + // A test may shut down with the fetch serve gate still closed. The + // requests it holds wait on the gate before they reach a worker pool, + // so release them, or the drain below never finishes. + #[cfg(any(test, feature = "test-utils"))] + self.fetch_serve_gate.open(); + // Close the responder worker pools. Detached tasks still queued for a // worker take the `Err` arm of `acquire_owned()` and exit immediately // rather than waiting for a slot that will never free up — without @@ -3063,6 +3151,8 @@ impl ReplicationEngine { let fetch_responder_admission_semaphore = Arc::clone(&self.fetch_responder_admission_semaphore); let fetch_responder_inflight = Arc::clone(&self.fetch_responder_inflight); + #[cfg(any(test, feature = "test-utils"))] + let fetch_serve_gate = Arc::clone(&self.fetch_serve_gate); let verification_responder_worker_semaphore = Arc::clone(&self.verification_responder_worker_semaphore); let verification_responder_admission_semaphore = @@ -3119,6 +3209,8 @@ impl ReplicationEngine { fetch_responder_worker_semaphore, fetch_responder_admission_semaphore, fetch_responder_inflight, + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate, verification_responder_worker_semaphore, verification_responder_admission_semaphore, verification_responder_inflight, @@ -3881,72 +3973,91 @@ impl ReplicationEngine { let is_bootstrapping = Arc::clone(&self.is_bootstrapping); let bootstrap_complete_notify = Arc::clone(&self.bootstrap_complete_notify); let detached_task_tracker = self.detached_task_tracker.clone(); - let concurrency = max_parallel_fetch(); + let concurrency = self.fetch_concurrency; info!("Fetch worker concurrency set to {concurrency} (hardware threads)"); let handle = tokio::spawn(async move { - // Each in-flight future yields (key, Option) so we - // always recover the key — even if the inner task panics. + // Each in-flight future yields (key, Option, when it + // finished), so we always recover the key, even if the inner task + // panics. let mut in_flight = FuturesUnordered::::new(); + let mut dispatch_window = FetchDispatchWindow::new(); + // Each answer as it arrives, ahead of its fetch's outcome. + let (answers, mut early_answers) = mpsc::unbounded_channel(); loop { - // Fill up to `concurrency` slots from the queue. - { - let mut q = queues.write().await; - while in_flight.len() < concurrency { - let Some(candidate) = q.dequeue_fetch() else { - break; - }; - let fetch_key = candidate.key; - let Some(&source) = candidate.sources.first() else { - warn!( - "Fetch candidate {} has no sources; requeueing for verification", - hex::encode(fetch_key) - ); - let _ = q.requeue_candidate_for_verification( - candidate, - config.verification_request_timeout, - ); - continue; - }; - q.start_dequeued_fetch(candidate, source); - - let p2p = Arc::clone(&p2p); - let storage = Arc::clone(&storage); - let config = Arc::clone(&config); - let token = shutdown.clone(); - let tracker = detached_task_tracker.clone(); - in_flight.push(Box::pin(async move { - // Tracked so shutdown() still awaits the task if - // this awaiter is dropped (e.g. the worker is - // aborted): it holds Arc and must - // not outlive the engine. - let handle = tracker.spawn(async move { - // Cancel-aware: abort when the engine shuts down. - tokio::select! { - () = token.cancelled() => FetchOutcome { - key: fetch_key, - result: FetchResult::SourceFailed, - }, - outcome = execute_single_fetch( - p2p, storage, config, fetch_key, source, - ) => outcome, - } - }); - match handle.await { - Ok(outcome) => (outcome.key, Some(outcome)), - Err(e) => { - error!( - "Fetch task for {} panicked: {e}", - hex::encode(fetch_key) - ); - (fetch_key, None) - } + // Fill free slots from the queue, never sending more than + // `FETCH_MAX_IN_FLIGHT_PER_SOURCE` of this node's fetches to any + // one answering source at once. + let lock_requested = Instant::now(); + let mut q = queues.write().await; + let lock_acquired = Instant::now(); + record_early_answers(&mut q, &mut early_answers); + let free_slots = concurrency.saturating_sub(in_flight.len()); + let dispatch = q.dispatch_fetches(free_slots, FETCH_MAX_IN_FLIGHT_PER_SOURCE); + let parked = dispatch.parked; + for candidate in dispatch.sourceless { + warn!( + "Fetch candidate {} has no untried sources; requeueing for verification", + hex::encode(candidate.key) + ); + let _ = q.requeue_candidate_for_verification( + candidate, + config.verification_request_timeout, + ); + } + for (fetch_key, source) in dispatch.started { + let p2p = Arc::clone(&p2p); + let storage = Arc::clone(&storage); + let config = Arc::clone(&config); + let answers = answers.clone(); + let token = shutdown.clone(); + let tracker = detached_task_tracker.clone(); + in_flight.push(Box::pin(async move { + // Tracked so shutdown() still awaits the task if + // this awaiter is dropped (e.g. the worker is + // aborted): it holds Arc and must + // not outlive the engine. + let handle = tracker.spawn(async move { + // Cancel-aware: abort when the engine shuts down. + tokio::select! { + () = token.cancelled() => FetchOutcome { + key: fetch_key, + source, + result: FetchResult::SourceFailed, + contact: Contact::Unknown, + }, + outcome = execute_single_fetch( + p2p, storage, config, fetch_key, source, answers, + ) => outcome, } - })); - } - } // release queues write lock + }); + let joined = handle.await; + let completed_at = Instant::now(); + match joined { + Ok(outcome) => (outcome.key, Some(outcome), completed_at), + Err(e) => { + error!("Fetch task for {} panicked: {e}", hex::encode(fetch_key)); + (fetch_key, None, completed_at) + } + } + })); + } + drop(q); + dispatch_window.parks += parked; + dispatch_window.record_fill( + lock_acquired.duration_since(lock_requested), + lock_acquired.elapsed(), + ); + if dispatch_window.age() >= FETCH_DISPATCH_SUMMARY_INTERVAL { + let stats_requested = Instant::now(); + let stats = queues + .write() + .await + .take_fetch_dispatch_stats(FETCH_MAX_IN_FLIGHT_PER_SOURCE); + dispatch_window.log_and_reset(&stats, stats_requested.elapsed()); + } if in_flight.is_empty() { // No work — wait for new items or shutdown. @@ -3961,22 +4072,32 @@ impl ReplicationEngine { // Wait for the next fetch to complete and process the result. tokio::select! { () = shutdown.cancelled() => break, - Some((key, maybe_outcome)) = in_flight.next() => { + Some((key, maybe_outcome, completed_at)) = in_flight.next() => { let mut q = queues.write().await; + record_early_answers(&mut q, &mut early_answers); + let attempt = attempt_latency(q.in_flight_started_at(&key), completed_at); + dispatch_window.record_outcome(maybe_outcome.as_ref(), attempt); let terminal = if let Some(outcome) = maybe_outcome { + q.record_fetch_contact(outcome.source, outcome.contact); match apply_fetch_result( &mut q, &key, &outcome.result, config.verification_request_timeout, + xor_distance(&key, p2p.peer_id().as_bytes()), ) { FetchFollowUp::Terminal => true, FetchFollowUp::RequeuedForVerification => false, + FetchFollowUp::Parked => { + dispatch_window.retry_parks += 1; + false + } FetchFollowUp::RetryFrom(next_peer) => { // Spawn a new fetch task for the next source. let p2p = Arc::clone(&p2p); let storage = Arc::clone(&storage); let config = Arc::clone(&config); + let answers = answers.clone(); let token = shutdown.clone(); let tracker = detached_task_tracker.clone(); let fetch_key = key; @@ -3987,21 +4108,32 @@ impl ReplicationEngine { tokio::select! { () = token.cancelled() => FetchOutcome { key: fetch_key, + source: next_peer, result: FetchResult::SourceFailed, + contact: Contact::Unknown, }, outcome = execute_single_fetch( - p2p, storage, config, fetch_key, next_peer, + p2p, + storage, + config, + fetch_key, + next_peer, + answers, ) => outcome, } }); - match handle.await { - Ok(outcome) => (outcome.key, Some(outcome)), + let joined = handle.await; + let completed_at = Instant::now(); + match joined { + Ok(outcome) => { + (outcome.key, Some(outcome), completed_at) + } Err(e) => { error!( "Fetch task for {} panicked: {e}", hex::encode(fetch_key) ); - (fetch_key, None) + (fetch_key, None, completed_at) } } })); @@ -4043,7 +4175,22 @@ impl ReplicationEngine { // Cancel and drain remaining in-flight fetches on shutdown. // The CancellationToken is already cancelled by this point, so // spawned tasks will see cancellation via their select! branches. - while in_flight.next().await.is_some() {} + // Their outcomes are counted but not applied, so the final + // summary's in-flight count still includes them. + let mut drained = Vec::new(); + while let Some(resolved) = in_flight.next().await { + drained.push(resolved); + } + let stats_requested = Instant::now(); + let mut q = queues.write().await; + for (key, maybe_outcome, completed_at) in &drained { + let attempt = attempt_latency(q.in_flight_started_at(key), *completed_at); + dispatch_window.record_outcome(maybe_outcome.as_ref(), attempt); + } + let stats = q.take_fetch_dispatch_stats(FETCH_MAX_IN_FLIGHT_PER_SOURCE); + drop(q); + dispatch_window.shutdown = true; + dispatch_window.log_and_reset(&stats, stats_requested.elapsed()); debug!("Fetch worker shut down"); }); self.task_handles.push(handle); @@ -4589,6 +4736,9 @@ struct ReplicationMessageHandlerContext { fetch_responder_worker_semaphore: Arc, fetch_responder_admission_semaphore: Arc, fetch_responder_inflight: Arc>>, + /// Test-only hold on serving admitted fetch requests. + #[cfg(any(test, feature = "test-utils"))] + fetch_serve_gate: Arc, verification_responder_worker_semaphore: Arc, verification_responder_admission_semaphore: Arc, verification_responder_inflight: Arc>>, @@ -7112,6 +7262,45 @@ pub fn verification_requests_for_key_from_for_test(requester: &PeerId, key: &Xor }) } +/// Test-only: holds an engine's admitted fetch requests before they are served. +/// +/// A held request keeps its admission slot and its requester's per-peer share, +/// exactly as a request waiting for a worker does, so a test can read what a +/// requester has outstanding without racing the speed at which the host serves +/// a chunk. One per engine, so closing it affects no other node or test. +#[cfg(any(test, feature = "test-utils"))] +#[derive(Debug, Default)] +struct FetchServeGate { + closed: AtomicBool, + opened: Notify, +} + +#[cfg(any(test, feature = "test-utils"))] +impl FetchServeGate { + fn close(&self) { + self.closed.store(true, Ordering::Release); + } + + fn open(&self) { + self.closed.store(false, Ordering::Release); + self.opened.notify_waiters(); + } + + /// Wait while the gate is closed. The wakeup is registered before the flag + /// is read, so an open that lands between the two is not missed. + async fn wait_open(&self) { + loop { + let opened = self.opened.notified(); + tokio::pin!(opened); + opened.as_mut().enable(); + if !self.closed.load(Ordering::Acquire) { + return; + } + opened.await; + } + } +} + async fn handle_verification_request( source: &PeerId, request: &protocol::VerificationRequest, @@ -7365,8 +7554,12 @@ async fn dispatch_fetch_request( let p2p_node = Arc::clone(&ctx.p2p_node); let request_timeout = ctx.config.fetch_request_timeout; let rr_message_id = rr_message_id.map(ToOwned::to_owned); + #[cfg(any(test, feature = "test-utils"))] + let serve_gate = Arc::clone(&ctx.fetch_serve_gate); ctx.detached_task_tracker.spawn(async move { let _guard = guard; + #[cfg(any(test, feature = "test-utils"))] + serve_gate.wait_open().await; let Ok(_worker_permit) = worker_semaphore.acquire_owned().await else { debug!( responder_class = "fetch", @@ -8931,7 +9124,10 @@ async fn complete_bootstrap( /// Result classification for a single fetch attempt. enum FetchResult { /// Data fetched, integrity-checked, and stored successfully. - Stored, + Stored { + /// Size of the stored chunk, for the per-source scheduler's summary. + bytes: usize, + }, /// Content-address integrity check failed — do not retry. IntegrityFailed, /// Source failed (network error or non-success response) — retryable. @@ -8979,7 +9175,159 @@ enum FetchResult { /// worker loop to update queue state. struct FetchOutcome { key: XorName, + /// The source this attempt was sent to. + source: PeerId, result: FetchResult, + /// Whether `source` answered, for the per-source cap's exemption of + /// sources that stopped answering. + contact: Contact, +} + +/// How often the fetch worker reports its dispatching. +const FETCH_DISPATCH_SUMMARY_INTERVAL: Duration = Duration::from_mins(5); + +/// How long an attempt that started at `started` took, up to when its task +/// finished rather than when the worker got to its outcome. +fn attempt_latency(started: Option, completed_at: Instant) -> Option { + started.map(|started| completed_at.saturating_duration_since(started)) +} + +/// What the fetch worker observed about its own dispatching since the last +/// summary. +#[derive(Default)] +struct FetchDispatchWindow { + /// When this window began. + started: Option, + /// Longest wait for the queues write lock before a fill pass. + fill_lock_wait_max: Duration, + /// Longest hold of the queues write lock by a fill pass, spawning included. + fill_lock_hold_max: Duration, + /// Fill passes in this window. + fills: u64, + /// Queued keys parked by fill passes in this window. + parks: usize, + /// Failed fetches parked because every untried source was full. + retry_parks: u64, + /// Attempts by outcome. + stored: u64, + already_held: u64, + no_longer_responsible: u64, + source_failed: u64, + integrity_failed: u64, + local_write_failed: u64, + panicked: u64, + /// Attempts cancelled by shutdown before they resolved. + cancelled: u64, + /// Bytes of chunks stored by attempts in this window. + stored_bytes: u64, + /// Attempt latency: longest, and total over `timed_attempts`. + attempt_max: Duration, + attempt_total: Duration, + timed_attempts: u64, + /// Set for the summary logged at shutdown, whose in-flight count includes + /// attempts drained without being applied. + shutdown: bool, +} + +impl FetchDispatchWindow { + fn new() -> Self { + Self { + started: Some(Instant::now()), + ..Self::default() + } + } + + /// Time since this window began. + fn age(&self) -> Duration { + self.started + .map_or(Duration::ZERO, |started| started.elapsed()) + } + + fn record_fill(&mut self, wait: Duration, hold: Duration) { + self.fill_lock_wait_max = self.fill_lock_wait_max.max(wait); + self.fill_lock_hold_max = self.fill_lock_hold_max.max(hold); + self.fills += 1; + } + + /// Count one resolved attempt: `None` when its task panicked. A cancelled + /// attempt is counted as cancelled whatever result it carries, and its + /// cut-short latency is left out. + fn record_outcome(&mut self, outcome: Option<&FetchOutcome>, attempt: Option) { + if outcome.is_some_and(|outcome| outcome.contact == Contact::Unknown) { + self.cancelled += 1; + return; + } + match outcome.map(|outcome| &outcome.result) { + Some(FetchResult::Stored { bytes }) => { + self.stored += 1; + self.stored_bytes = self + .stored_bytes + .saturating_add(u64::try_from(*bytes).unwrap_or(u64::MAX)); + } + Some(FetchResult::AlreadyHeld) => self.already_held += 1, + Some(FetchResult::NoLongerResponsible) => self.no_longer_responsible += 1, + Some(FetchResult::SourceFailed) => self.source_failed += 1, + Some(FetchResult::IntegrityFailed) => self.integrity_failed += 1, + Some(FetchResult::LocalWriteFailed) => self.local_write_failed += 1, + None => self.panicked += 1, + } + if let Some(attempt) = attempt { + self.attempt_max = self.attempt_max.max(attempt); + self.attempt_total = self.attempt_total.saturating_add(attempt); + self.timed_attempts += 1; + } + } + + /// Log this window with the queues' state, then start a new window. + /// `stats_lock` is how long reading that state took, lock wait included. + fn log_and_reset(&mut self, stats: &FetchDispatchStats, stats_lock: Duration) { + let attempt_mean_ms = u32::try_from(self.timed_attempts) + .ok() + .filter(|count| *count > 0) + .map_or(0, |count| (self.attempt_total / count).as_millis()); + info!( + per_source_cap = FETCH_MAX_IN_FLIGHT_PER_SOURCE, + window_secs = self.age().as_secs(), + parked = stats.parked, + parked_associations = stats.parked_associations, + oldest_parked_secs = stats.oldest_parked.map_or(0, |age| age.as_secs()), + queued = stats.queued, + in_flight = stats.in_flight, + saturated_sources = stats.saturated_sources, + fills = self.fills, + parks = self.parks, + retry_parks = self.retry_parks, + stored = self.stored, + stored_bytes = self.stored_bytes, + already_held = self.already_held, + no_longer_responsible = self.no_longer_responsible, + source_failed = self.source_failed, + integrity_failed = self.integrity_failed, + local_write_failed = self.local_write_failed, + panicked = self.panicked, + cancelled = self.cancelled, + shutdown = self.shutdown, + attempt_max_ms = self.attempt_max.as_millis(), + attempt_mean_ms, + fill_lock_wait_max_ms = self.fill_lock_wait_max.as_millis(), + fill_lock_hold_max_ms = self.fill_lock_hold_max.as_millis(), + parked_maintenance_max_ms = stats.parked_maintenance_max.as_millis(), + tracked_sources = stats.tracked_sources, + silent_sources = stats.silent_sources, + health_overflow_active = stats.overflow_active, + silent_peak = stats.health.silent_peak, + silent_starts = stats.health.silent_starts, + unreachable = stats.health.unreachable, + silent_marks = stats.health.marks, + silent_clears = stats.health.clears, + silent_expired = stats.health.expired_marks, + silent_ignored_stale = stats.health.ignored_stale, + health_overflows = stats.health.overflows, + stats_lock_ms = stats_lock.as_millis(), + "replication fetch dispatch summary" + ); + *self = Self::new(); + } } /// What the fetch worker must do next for a key whose fetch attempt resolved. @@ -8993,6 +9341,11 @@ enum FetchFollowUp { RequeuedForVerification, /// Retry immediately from this next untried verified source. RetryFrom(PeerId), + /// Every untried source already has this node's per-source share in + /// flight, so the key went back to the fetch queue, parked, keeping its + /// tried sources and retry reservation. Not terminal: the key is still in + /// the pipeline, and the next fill pass starts it once a share frees. + Parked, } /// Apply a resolved [`FetchResult`] to the queues and report the follow-up. @@ -9013,19 +9366,26 @@ fn apply_fetch_result( key: &XorName, result: &FetchResult, verification_retry_after: Duration, + distance: XorName, ) -> FetchFollowUp { match result { - FetchResult::Stored | FetchResult::NoLongerResponsible | FetchResult::AlreadyHeld => { + FetchResult::Stored { .. } + | FetchResult::NoLongerResponsible + | FetchResult::AlreadyHeld => { q.complete_fetch(key); FetchFollowUp::Terminal } FetchResult::IntegrityFailed | FetchResult::SourceFailed => { - if let Some(next_peer) = q.retry_fetch(key) { - FetchFollowUp::RetryFrom(next_peer) - } else if q.requeue_fetch_for_verification(key, verification_retry_after) { - FetchFollowUp::RequeuedForVerification - } else { - FetchFollowUp::Terminal + match q.retry_or_park_fetch(key, distance, FETCH_MAX_IN_FLIGHT_PER_SOURCE) { + FetchRetry::Next(next_peer) => FetchFollowUp::RetryFrom(next_peer), + FetchRetry::Parked => FetchFollowUp::Parked, + FetchRetry::Exhausted => { + if q.requeue_fetch_for_verification(key, verification_retry_after) { + FetchFollowUp::RequeuedForVerification + } else { + FetchFollowUp::Terminal + } + } } } // No `retry_fetch`: the local write is what failed, so every other @@ -9072,7 +9432,6 @@ async fn is_storage_admitted( .await } -#[allow(clippy::too_many_lines)] /// Execute a single fetch request against `source` for `key`. /// /// Handles encoding, network I/O, integrity checking, storage, and trust @@ -9092,17 +9451,64 @@ async fn execute_single_fetch( config: Arc, key: XorName, source: PeerId, + answers: mpsc::UnboundedSender<(PeerId, Contact)>, ) -> FetchOutcome { + let mut contact = Contact::NotAttempted; + let result = fetch_attempt( + p2p_node, + storage, + config, + key, + source, + &mut contact, + &answers, + ) + .await; + FetchOutcome { + key, + source, + result, + contact, + } +} + +/// Record every answer a fetch has reported since the last call. +/// +/// A fetch reports its source's answer as soon as it arrives, before checking +/// and storing the chunk, so the per-source cap never treats an answering +/// source as silent while that work is still under way. The same answer comes +/// again with the fetch's outcome, which records it a second time to no +/// effect. Each is reported while its fetch is in flight, and the worker calls +/// this before applying any outcome, so it is recorded before its fetch ends. +fn record_early_answers( + queues: &mut ReplicationQueues, + early_answers: &mut mpsc::UnboundedReceiver<(PeerId, Contact)>, +) { + while let Ok((source, contact)) = early_answers.try_recv() { + queues.record_fetch_contact(source, contact); + } +} + +/// The body of [`execute_single_fetch`]: the attempt's result, with `contact` +/// set to whether the source answered, stamped when the request returned. An +/// answer is also sent on `answers` at once. +#[allow(clippy::too_many_lines)] +async fn fetch_attempt( + p2p_node: Arc, + storage: Arc, + config: Arc, + key: XorName, + source: PeerId, + contact: &mut Contact, + answers: &mpsc::UnboundedSender<(PeerId, Contact)>, +) -> FetchResult { let self_id = *p2p_node.peer_id(); if !is_storage_admitted(&self_id, &key, &p2p_node, &config).await { debug!( "Skipping fetch for {}: no longer in the storage-admission group", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::NoLongerResponsible, - }; + return FetchResult::NoLongerResponsible; } // Possession, then capacity — both before the dial, and in that order. @@ -9116,10 +9522,7 @@ async fn execute_single_fetch( "Skipping fetch for {}: already held locally", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::AlreadyHeld, - }; + return FetchResult::AlreadyHeld; } // Capacity is checked before the dial rather than after the bytes arrive. @@ -9141,10 +9544,7 @@ async fn execute_single_fetch( "Skipping fetch for {}: local storage cannot accept a write: {e}", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::LocalWriteFailed, - }; + return FetchResult::LocalWriteFailed; } let request = protocol::FetchRequest { key }; @@ -9157,13 +9557,11 @@ async fn execute_single_fetch( Ok(data) => data, Err(e) => { warn!("Failed to encode fetch request: {e}"); - return FetchOutcome { - key, - result: FetchResult::SourceFailed, - }; + return FetchResult::SourceFailed; } }; + let sent = next_fetch_contact(); let result = p2p_node .send_request( &source, @@ -9172,6 +9570,18 @@ async fn execute_single_fetch( config.fetch_request_timeout, ) .await; + *contact = if result.is_ok() { + Contact::Answered { + order: next_fetch_contact(), + } + } else { + Contact::Unreachable { sent } + }; + if matches!(contact, Contact::Answered { .. }) { + // Fails only once the fetch worker has stopped, when no scheduling + // decision is left to inform. + let _ = answers.send((source, *contact)); + } match result { Ok(response) => { @@ -9182,10 +9592,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::SourceFailed, - }; + return FetchResult::SourceFailed; }; match resp_msg.body { @@ -9209,10 +9616,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::IntegrityFailed, - }; + return FetchResult::IntegrityFailed; } // Enforce chunk size invariant on fetched data. @@ -9231,10 +9635,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::IntegrityFailed, - }; + return FetchResult::IntegrityFailed; } // Content-address integrity check. @@ -9251,10 +9652,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - return FetchOutcome { - key, - result: FetchResult::IntegrityFailed, - }; + return FetchResult::IntegrityFailed; } // Responsibility can lapse during the network round-trip. @@ -9268,10 +9666,7 @@ async fn execute_single_fetch( "Fetched {} but responsibility lapsed in transit; not storing", hex::encode(key) ); - return FetchOutcome { - key, - result: FetchResult::NoLongerResponsible, - }; + return FetchResult::NoLongerResponsible; } if let Err(e) = storage.put(&resp_key, &data).await { @@ -9285,16 +9680,10 @@ async fn execute_single_fetch( "Failed to store fetched record {}: {e}", hex::encode(resp_key) ); - return FetchOutcome { - key, - result: FetchResult::LocalWriteFailed, - }; + return FetchResult::LocalWriteFailed; } - FetchOutcome { - key, - result: FetchResult::Stored, - } + FetchResult::Stored { bytes: data.len() } } ReplicationMessageBody::FetchResponse( ref response @ (protocol::FetchResponse::NotFound { .. } @@ -9320,10 +9709,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - FetchOutcome { - key, - result: FetchResult::SourceFailed, - } + FetchResult::SourceFailed } _ => { // Unexpected message type — treat as malformed. @@ -9333,10 +9719,7 @@ async fn execute_single_fetch( TrustEvent::ApplicationFailure(REPLICATION_TRUST_WEIGHT), ) .await; - FetchOutcome { - key, - result: FetchResult::SourceFailed, - } + FetchResult::SourceFailed } } } @@ -9344,10 +9727,7 @@ async fn execute_single_fetch( debug!("Fetch request to {source} failed: {e}"); // No ApplicationFailure here — P2PNode::send_request() already // reports ConnectionTimeout / ConnectionFailed to the TrustEngine. - FetchOutcome { - key, - result: FetchResult::SourceFailed, - } + FetchResult::SourceFailed } } } @@ -10537,6 +10917,7 @@ mod tests { use crate::replication::audit::AuditTickResult; use crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL; use crate::replication::recent_provers::RecentProvers; + use crate::replication::scheduling::MAX_FETCH_QUEUE; use crate::replication::types::{AuditFailureReason, FailureEvidence}; use lru::LruCache; use saorsa_core::identity::PeerId; @@ -13779,8 +14160,13 @@ mod tests { drive_key_in_flight(&mut q, key, vec![test_peer(0x01), test_peer(0x02)]); assert_eq!(q.retry_reserved_slot_count(), 1); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::NoLongerResponsible, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::NoLongerResponsible, + RETRY_AFTER, + key, + ); assert_eq!( follow_up, @@ -13810,12 +14196,15 @@ mod tests { // Both variants must walk the identical terminal gate so the // battle-tested Stored accounting (retry-slot release + bootstrap // pending-set shrink in the caller) covers lapsed responsibility too. - for result in [FetchResult::Stored, FetchResult::NoLongerResponsible] { + for result in [ + FetchResult::Stored { bytes: 0 }, + FetchResult::NoLongerResponsible, + ] { let mut q = ReplicationQueues::new(); let key = test_key(0xCD); drive_key_in_flight(&mut q, key, vec![test_peer(0x03)]); - let follow_up = apply_fetch_result(&mut q, &key, &result, RETRY_AFTER); + let follow_up = apply_fetch_result(&mut q, &key, &result, RETRY_AFTER, key); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); @@ -13833,7 +14222,8 @@ mod tests { let second = test_peer(0x05); drive_key_in_flight(&mut q, key, vec![first, second]); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); assert_eq!( follow_up, FetchFollowUp::RetryFrom(second), @@ -13841,7 +14231,8 @@ mod tests { ); assert_eq!(q.in_flight_count(), 1, "retry keeps the key in flight"); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); assert_eq!( follow_up, FetchFollowUp::RequeuedForVerification, @@ -13855,6 +14246,199 @@ mod tests { ); } + /// Put two unrelated fetches in flight to `source`, filling its share. + fn saturate_source(q: &mut ReplicationQueues, source: PeerId, first_key_byte: u8) { + for offset in 0..2u8 { + let key = test_key(first_key_byte + offset); + q.start_fetch(key, source, vec![source]); + } + } + + #[test] + fn source_failure_with_every_untried_source_full_parks_the_key() { + const RETRY_AFTER: Duration = Duration::from_mins(1); + + let mut q = ReplicationQueues::new(); + let key = test_key(0x2A); + let (first, second) = (test_peer(0x01), test_peer(0x02)); + assert_eq!(drive_key_in_flight(&mut q, key, vec![first, second]), first); + saturate_source(&mut q, second, 0xE0); + let reserved = q.retry_reserved_slot_count(); + + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); + + assert_eq!( + follow_up, + FetchFollowUp::Parked, + "the only untried source is at its share, so the key must wait rather \ + than be sent a request it will drop" + ); + assert!( + q.contains_key(&key), + "a parked key is still in the pipeline" + ); + assert!(q.fetch_queued_or_in_flight(&key)); + assert_eq!( + q.in_flight_count(), + 2, + "only the two saturating fetches remain" + ); + assert_eq!( + q.pending_count(), + 0, + "parking is not a verification requeue" + ); + assert_eq!(q.retry_reserved_slot_count(), reserved); + } + + #[test] + fn an_answer_reported_before_its_fetch_ends_keeps_a_timed_out_source_capped() { + const RETRY_AFTER: Duration = Duration::from_mins(1); + const SLOTS: usize = 8; + + let source = test_peer(0x0A); + let (dropped, answered) = (test_key(0x50), test_key(0x51)); + let waiting: Vec = (0x60..0x64).map(test_key).collect(); + // Both of `source`'s shares are in flight and four keys only it holds + // wait behind them. + let setup = || { + let mut q = ReplicationQueues::new(); + q.start_fetch(dropped, source, vec![source]); + q.start_fetch(answered, source, vec![source]); + for key in &waiting { + assert!(q.enqueue_fetch(*key, *key, vec![source])); + } + q + }; + let in_flight_to_source = |q: &ReplicationQueues| { + q.fetch_pipeline_snapshot() + .in_flight + .iter() + .filter(|(_, peer)| *peer == source) + .count() + }; + // The first request is dropped and times out. The second, sent after + // it, has been answered and is still checking and storing its chunk. + let dropped_sent = next_fetch_contact(); + let _answered_sent = next_fetch_contact(); + let answer = Contact::Answered { + order: next_fetch_contact(), + }; + let timeout = Contact::Unreachable { sent: dropped_sent }; + + // Known only from the second fetch's outcome, the answer comes too + // late: the timeout marks `source` silent and the next pass sends it + // every waiting key, past its share. + let mut late = setup(); + late.record_fetch_contact(source, timeout); + apply_fetch_result( + &mut late, + &dropped, + &FetchResult::SourceFailed, + RETRY_AFTER, + dropped, + ); + late.dispatch_fetches(SLOTS, FETCH_MAX_IN_FLIGHT_PER_SOURCE); + assert_eq!(in_flight_to_source(&late), 1 + waiting.len()); + + // Reported when it arrives, the answer is recorded before the timeout, + // which it outdates, so `source` stays within its share. + let (answers, mut early_answers) = mpsc::unbounded_channel(); + let mut q = setup(); + assert!(answers.send((source, answer)).is_ok()); + record_early_answers(&mut q, &mut early_answers); + q.record_fetch_contact(source, timeout); + apply_fetch_result( + &mut q, + &dropped, + &FetchResult::SourceFailed, + RETRY_AFTER, + dropped, + ); + record_early_answers(&mut q, &mut early_answers); + let dispatch = q.dispatch_fetches(SLOTS, FETCH_MAX_IN_FLIGHT_PER_SOURCE); + assert_eq!(dispatch.started.len(), 1, "one share is free"); + assert_eq!(in_flight_to_source(&q), FETCH_MAX_IN_FLIGHT_PER_SOURCE); + + // The same answer again, with the second fetch's outcome, changes + // nothing. + q.record_fetch_contact(source, answer); + assert_eq!( + q.dispatch_fetches(SLOTS, FETCH_MAX_IN_FLIGHT_PER_SOURCE) + .started + .len(), + 0 + ); + assert_eq!(in_flight_to_source(&q), FETCH_MAX_IN_FLIGHT_PER_SOURCE); + } + + #[test] + fn the_dispatch_window_times_resolved_attempts_and_counts_cancelled_ones() { + let (key, source) = (test_key(0x31), test_peer(0x01)); + let outcome = |result: FetchResult, contact: Contact| FetchOutcome { + key, + source, + result, + contact, + }; + let started = Instant::now(); + let at = |millis: u64| started + Duration::from_millis(millis); + let mut window = FetchDispatchWindow::new(); + + // Two attempts resolve at different times; the worker reaches both + // later, which must not count. + let stored = outcome( + FetchResult::Stored { bytes: 10 }, + Contact::Answered { order: 2 }, + ); + window.record_outcome(Some(&stored), attempt_latency(Some(started), at(100))); + let failed = outcome(FetchResult::SourceFailed, Contact::Unreachable { sent: 1 }); + window.record_outcome(Some(&failed), attempt_latency(Some(started), at(300))); + // A cancelled attempt is counted as cancelled, not by its placeholder + // result, and its cut-short time is left out. + let cancelled = outcome(FetchResult::SourceFailed, Contact::Unknown); + window.record_outcome(Some(&cancelled), attempt_latency(Some(started), at(5_000))); + + assert_eq!((window.stored, window.source_failed), (1, 1)); + assert_eq!((window.cancelled, window.timed_attempts), (1, 2)); + assert_eq!(window.attempt_max, Duration::from_millis(300)); + assert_eq!(window.attempt_total, Duration::from_millis(400)); + assert_eq!(window.stored_bytes, 10); + assert_eq!(attempt_latency(None, at(100)), None); + } + + #[test] + fn source_failure_with_a_full_fetch_queue_requeues_for_verification() { + const RETRY_AFTER: Duration = Duration::from_mins(1); + + let mut q = ReplicationQueues::new(); + let key = test_key(0x2B); + let (first, second) = (test_peer(0x01), test_peer(0x02)); + drive_key_in_flight(&mut q, key, vec![first, second]); + saturate_source(&mut q, second, 0xE0); + let filler_source = test_peer(0x09); + let mut filler = 0u32; + while q.fetch_queue_count() < MAX_FETCH_QUEUE { + let mut filler_key = [0xC0u8; 32]; + filler_key[..4].copy_from_slice(&filler.to_be_bytes()); + assert!(q.enqueue_fetch(filler_key, filler_key, vec![filler_source])); + filler += 1; + } + + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); + + assert_eq!( + follow_up, + FetchFollowUp::RequeuedForVerification, + "with no room to park, the key returns to verification on its own \ + reserved slot instead of overfilling the fetch queue" + ); + assert_eq!(q.fetch_queue_count(), MAX_FETCH_QUEUE); + assert_eq!(q.pending_count(), 1); + } + #[test] fn source_failure_without_retry_metadata_is_terminal() { const RETRY_AFTER: Duration = Duration::from_mins(1); @@ -13868,7 +14452,8 @@ mod tests { let candidate = q.dequeue_fetch().expect("enqueued key must dequeue"); q.start_dequeued_fetch(candidate, source); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::SourceFailed, RETRY_AFTER, key); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); @@ -13883,7 +14468,8 @@ mod tests { let source = test_peer(0x0B); drive_key_in_flight(&mut q, key, vec![source, test_peer(0x0C)]); - let follow_up = apply_fetch_result(&mut q, &key, &FetchResult::AlreadyHeld, RETRY_AFTER); + let follow_up = + apply_fetch_result(&mut q, &key, &FetchResult::AlreadyHeld, RETRY_AFTER, key); assert_eq!( follow_up, @@ -13912,8 +14498,13 @@ mod tests { let third = test_peer(0x09); drive_key_in_flight(&mut q, key, vec![first, second, third]); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::LocalWriteFailed, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::LocalWriteFailed, + RETRY_AFTER, + key, + ); assert_eq!( follow_up, @@ -13946,8 +14537,13 @@ mod tests { let candidate = q.dequeue_fetch().expect("enqueued key must dequeue"); q.start_dequeued_fetch(candidate, source); - let follow_up = - apply_fetch_result(&mut q, &key, &FetchResult::LocalWriteFailed, RETRY_AFTER); + let follow_up = apply_fetch_result( + &mut q, + &key, + &FetchResult::LocalWriteFailed, + RETRY_AFTER, + key, + ); assert_eq!(follow_up, FetchFollowUp::Terminal); assert!(!q.contains_key(&key)); diff --git a/src/replication/scheduling.rs b/src/replication/scheduling.rs index e296606e..95c6b168 100644 --- a/src/replication/scheduling.rs +++ b/src/replication/scheduling.rs @@ -5,13 +5,15 @@ //! time, with strict dedup across all three stages. use std::cmp::Ordering; -use std::collections::{BTreeMap, BinaryHeap, HashMap, HashSet, VecDeque}; +use std::collections::{BTreeMap, BTreeSet, BinaryHeap, HashMap, HashSet, VecDeque}; use std::time::{Duration, Instant}; use crate::logging::debug; -use crate::ant_protocol::XorName; -use crate::replication::config::VERIFICATION_RETRY_BACKOFF_MAX; +use crate::ant_protocol::{XorName, CLOSE_GROUP_SIZE}; +use crate::replication::config::{ + FETCH_REQUEST_TIMEOUT, PAID_LIST_CLOSE_GROUP_SIZE, VERIFICATION_RETRY_BACKOFF_MAX, +}; use crate::replication::types::{ FetchCandidate, FetchOrder, FetchPayload, VerificationEntry, VerificationState, }; @@ -94,6 +96,32 @@ pub const MAX_FETCH_QUEUE: usize = 131_072; /// decides *which* sources survive the cap. pub const MAX_HINT_SOURCES_PER_KEY: usize = 8; +/// Maximum distinct source peers one key may be associated with while it is +/// queued or in flight for fetch, counting the sources it was already fetched +/// from this cycle, for the given close-group widths. +/// +/// A verification round names at most its quorum targets and its paid-list +/// targets as holders, and the replica-hint fallbacks added to them come from a +/// pending entry already bounded by [`MAX_HINT_SOURCES_PER_KEY`], so this never +/// truncates a source list a round produced. It bounds what later +/// advertisements can merge into a queued or in-flight key, and with it the +/// parked-fetch index, which holds one entry per untried source of each parked +/// key. +#[must_use] +pub(crate) const fn fetch_source_limit( + close_group_size: usize, + paid_list_close_group_size: usize, +) -> usize { + close_group_size + .saturating_add(paid_list_close_group_size) + .saturating_add(MAX_HINT_SOURCES_PER_KEY) +} + +/// [`fetch_source_limit`] at the production close-group widths, and the limit +/// [`ReplicationQueues::new`] starts with. +pub(crate) const MAX_FETCH_SOURCES_PER_KEY: usize = + fetch_source_limit(CLOSE_GROUP_SIZE, PAID_LIST_CLOSE_GROUP_SIZE); + /// Outcome of [`ReplicationQueues::add_pending_verify`]. /// /// Distinguishes "the key is already being handled" from "the key was @@ -174,9 +202,13 @@ pub struct InFlightEntry { pub key: XorName, /// The peer we are currently fetching from. pub source: PeerId, - /// When the fetch started. + /// When the current attempt started: the first, or under the per-source + /// scheduler the latest retry. pub started_at: Instant, - /// All verified sources for this key. + /// Verified sources for this key not yet tried before this episode's + /// current attempt. A key that was parked between attempts keeps the + /// sources it already tried only in `tried`, so everything the key is + /// associated with is the union of the two. pub all_sources: Vec, /// Sources already attempted (failed or in progress). pub tried: HashSet, @@ -184,6 +216,409 @@ pub struct InFlightEntry { pub retry_verification: Option, } +// --------------------------------------------------------------------------- +// Per-source fetch dispatch +// --------------------------------------------------------------------------- + +/// A queued key none of whose untried sources had a free share when it was +/// last considered for dispatch. +/// +/// Its payload stays in `fetch_payloads`; only its place in line moves here +/// from the fetch heap. +#[derive(Debug, Clone)] +struct ParkedFetch { + /// XOR distance from self to the key, its priority once startable again. + distance: XorName, + /// Untried sources whose next free share could start the key, in the + /// payload's source-preference order. Each one indexes the key in + /// `parked_by_source`. + wake_sources: Vec, + /// When the key was parked, for the dispatch summary's oldest-parked age. + parked_at: Instant, +} + +/// Fetches started, and candidates found with no untried source, by one +/// [`ReplicationQueues::dispatch_fetches`] pass. +#[derive(Debug, Default)] +pub(crate) struct FetchDispatch { + /// Each started key with the source it was sent to. Already in flight. + pub(crate) started: Vec<(XorName, PeerId)>, + /// Dequeued candidates with no untried source left. The caller decides + /// what to do with them, exactly as for a sourceless dequeue. + pub(crate) sourceless: Vec, + /// Heap keys this pass parked because every untried source was full. + pub(crate) parked: usize, +} + +/// The per-source scheduler's queue state, for its periodic summary. +/// +/// Read only by that summary's log line, which a build without logging +/// compiles out. +#[derive(Debug, Clone, Copy)] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub(crate) struct FetchDispatchStats { + /// Keys parked behind full sources. + pub(crate) parked: usize, + /// How long the longest-parked key has waited, if any is parked. + pub(crate) oldest_parked: Option, + /// Keys queued for fetch, parked ones included. + pub(crate) queued: usize, + /// Fetches in flight. + pub(crate) in_flight: usize, + /// Sources holding at least the per-source cap of this node's fetches. + pub(crate) saturated_sources: usize, + /// Entries in the parked index: one per untried source of each parked key. + pub(crate) parked_associations: usize, + /// Longest parked-index maintenance after a departed source, since the last + /// read. + pub(crate) parked_maintenance_max: Duration, + /// Sources in the health table before the summary's purge. + pub(crate) tracked_sources: usize, + /// Tracked sources silent now. + pub(crate) silent_sources: usize, + /// Whether the health table is overflowing, so every untracked source + /// also counts as silent. + pub(crate) overflow_active: bool, + /// Source-health events since the last read. + pub(crate) health: SourceHealthCounters, +} + +/// What [`ReplicationQueues::retry_or_park_fetch`] did with a failed fetch. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum FetchRetry { + /// Retry now from this untried source, which has a free share. + Next(PeerId), + /// Every untried source is at its share: the key went back to the fetch + /// queue, parked, keeping its tried sources and retry reservation. + Parked, + /// No untried source is left, or the fetch queue is full. The key stays + /// in flight, with its failed source now marked tried, for the caller's + /// verification requeue. + Exhausted, +} + +// --------------------------------------------------------------------------- +// Fetch source health +// --------------------------------------------------------------------------- + +/// How many fetch timeouts a source stays silent after this node last learned +/// of an unanswered request to it. +const SILENCE_LIFETIME_TIMEOUTS: u32 = 2; + +/// Shortest time a mark lasts, whatever the fetch timeout, so a zero or +/// tiny configured timeout cannot turn silent sources off. +const MIN_SILENCE_LIFETIME: Duration = Duration::from_secs(1); + +/// Most fetch sources whose answering this node tracks at once. +/// +/// Far above the sources one node fetches from within a silence lifetime. At +/// the bound, entries that neither hold a mark nor guard an in-flight fetch +/// are dropped. If that frees nothing, a source not in the table counts as +/// silent for one silence lifetime: fetched from as it was before the +/// per-source cap, never more slowly. +const MAX_TRACKED_FETCH_SOURCES: usize = 8_192; + +/// Whether a fetch attempt reached its source. +/// +/// Requests and answers carry their place in one process-wide sequence, taken +/// as the request is sent and as its answer returns, so the table can tell +/// which came first however the fetch worker orders their outcomes and +/// whatever the clock reads. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum Contact { + /// The attempt ended before a request was sent. + NotAttempted, + /// The source answered, whatever the answer then turned out to be. + Answered { + /// Where the answer's arrival falls in the sequence. + order: u64, + }, + /// The request got no answer: a timeout or a connection failure. + Unreachable { + /// Where the request's sending falls in the sequence. + sent: u64, + }, + /// The attempt was cancelled at shutdown, so nothing is known. + Unknown, +} + +/// What this node has observed of one source's answering. +#[derive(Debug, Clone, Copy, Default)] +struct SourceHealth { + /// Sequence place of the latest answer from this source. + last_answer: Option, + /// Sequence place of the latest-sent request to this source that got no + /// answer. + last_unanswered: Option, + /// When this node learned of that request's failure, while the request is + /// newer than the latest answer and the mark has not expired. + marked_since: Option, +} + +impl SourceHealth { + /// Whether a request sent after this source's latest answer went + /// unanswered. It depends only on the two latest events, so it is the same + /// whatever order the outcomes were applied in. + fn unanswered_since_answer(&self) -> bool { + self.last_unanswered + .is_some_and(|sent| self.last_answer.is_none_or(|answer| sent > answer)) + } +} + +/// Source-health events since the dispatch summary last read them. +/// +/// Read only by that summary's log line, which a build without logging +/// compiles out. +#[derive(Debug, Clone, Copy, Default)] +#[cfg_attr(not(feature = "logging"), allow(dead_code))] +pub(crate) struct SourceHealthCounters { + /// Attempts whose request got no answer. + pub(crate) unreachable: u64, + /// Sources that became silent. + pub(crate) marks: u64, + /// Marks cleared by an answer. + pub(crate) clears: u64, + /// Marks that ran out their lifetime. + pub(crate) expired_marks: u64, + /// Unanswered requests sent before their source's latest answer. + pub(crate) ignored_stale: u64, + /// Fetches started or retried on a silent source, outside the cap. + pub(crate) silent_starts: u64, + /// Times the table was full of marks and in-flight guards. + pub(crate) overflows: u64, + /// Most tracked sources silent at once. + pub(crate) silent_peak: usize, +} + +/// What this node has observed of each fetch source's answering. +/// +/// The per-source cap keeps this node within the admission share a responder +/// grants it, and only a responder that answers has a share to keep within. A +/// source is silent while the latest-sent request to it that went unanswered +/// was sent after its latest answer: it is fetched from without the cap, as +/// before the cap existed. Any answer received after that request was sent +/// ends it. So does learning of no newer unanswered request for one silence +/// lifetime, so a source this node has stopped trying is probed again under +/// the cap. +#[derive(Debug)] +struct SourceHealthTable { + entries: HashMap, + /// Every mark as `(marked_since, source)`, so marks expire in order. + marks_by_since: BTreeSet<(Instant, PeerId)>, + /// How long a mark lasts without newer evidence. + silence_lifetime: Duration, + /// When the table last overflowed. For one silence lifetime after, a + /// source not in the table counts as silent. + overflow_since: Option, + counters: SourceHealthCounters, +} + +impl SourceHealthTable { + fn new(fetch_request_timeout: Duration) -> Self { + Self { + entries: HashMap::new(), + marks_by_since: BTreeSet::new(), + silence_lifetime: fetch_request_timeout + .saturating_mul(SILENCE_LIFETIME_TIMEOUTS) + .max(MIN_SILENCE_LIFETIME), + overflow_since: None, + counters: SourceHealthCounters::default(), + } + } + + /// Whether something that began at `since` is still within one silence + /// lifetime at `now`. + fn within_lifetime(&self, since: Instant, now: Instant) -> bool { + now.saturating_duration_since(since) < self.silence_lifetime + } + + fn overflowing(&self, now: Instant) -> bool { + self.overflow_since + .is_some_and(|since| self.within_lifetime(since, now)) + } + + /// Whether `source` counts as silent at `now`. + fn is_silent(&self, source: &PeerId, now: Instant) -> bool { + self.entries.get(source).map_or_else( + || self.overflowing(now), + |health| { + health + .marked_since + .is_some_and(|since| self.within_lifetime(since, now)) + }, + ) + } + + /// Drop the marks that ran out their lifetime by `now`. + fn expire(&mut self, now: Instant) { + while let Some(&(since, source)) = self.marks_by_since.first() { + if self.within_lifetime(since, now) { + break; + } + self.marks_by_since.pop_first(); + if let Some(health) = self.entries.get_mut(&source) { + if health.marked_since == Some(since) { + health.marked_since = None; + } + } + self.counters.expired_marks += 1; + } + } + + /// Drop every entry that neither holds a mark nor guards an in-flight + /// fetch to its source. Every contact, an answer reported ahead of its + /// fetch's outcome included, is applied while its own fetch is still in + /// flight, so none can arrive for a dropped source's past requests: + /// anything later is newer than all of its history. Call after + /// [`Self::expire`]. + fn purge(&mut self, in_flight: &HashMap) { + let guarded: HashSet = in_flight.values().map(|entry| entry.source).collect(); + self.entries + .retain(|source, health| health.marked_since.is_some() || guarded.contains(source)); + } + + /// The entry for `source`, purging the table at the bound. `None` when the + /// table cannot take it, which starts an overflow. + fn entry( + &mut self, + source: PeerId, + now: Instant, + in_flight: &HashMap, + ) -> Option<&mut SourceHealth> { + if !self.entries.contains_key(&source) && self.entries.len() >= MAX_TRACKED_FETCH_SOURCES { + if self.overflowing(now) { + return None; + } + self.purge(in_flight); + if self.entries.len() >= MAX_TRACKED_FETCH_SOURCES { + self.overflow_since = Some(now); + self.counters.overflows += 1; + return None; + } + } + Some(self.entries.entry(source).or_default()) + } + + /// Apply, at `now`, one attempt's contact with `source`. + fn record( + &mut self, + source: PeerId, + contact: Contact, + now: Instant, + in_flight: &HashMap, + ) { + self.expire(now); + match contact { + Contact::NotAttempted | Contact::Unknown => {} + Contact::Answered { order } => { + let Some(health) = self.entry(source, now, in_flight) else { + return; + }; + health.last_answer = health.last_answer.max(Some(order)); + let cleared = if health.unanswered_since_answer() { + None + } else { + health.marked_since.take() + }; + if let Some(since) = cleared { + self.marks_by_since.remove(&(since, source)); + self.counters.clears += 1; + } + } + Contact::Unreachable { sent } => { + self.counters.unreachable += 1; + let Some(health) = self.entry(source, now, in_flight) else { + return; + }; + let stale = health.last_answer.is_some_and(|answer| sent < answer); + let newest = health.last_unanswered.is_none_or(|latest| sent > latest); + if newest { + health.last_unanswered = Some(sent); + } + // The newest unanswered request, if newer than every answer, + // marks the source from now: from when this node learned of it. + let (replaced, marked) = if newest && health.unanswered_since_answer() { + (health.marked_since.replace(now), true) + } else { + (None, false) + }; + if stale { + self.counters.ignored_stale += 1; + } + if marked { + if let Some(since) = replaced { + self.marks_by_since.remove(&(since, source)); + } else { + self.counters.marks += 1; + } + self.marks_by_since.insert((now, source)); + self.counters.silent_peak = + self.counters.silent_peak.max(self.marks_by_since.len()); + } + } + } + } + + /// The counters since the last call, and the table's size before the + /// purge that follows. + fn take_counters( + &mut self, + now: Instant, + in_flight: &HashMap, + ) -> (SourceHealthCounters, usize) { + self.expire(now); + let tracked = self.entries.len(); + self.purge(in_flight); + let counters = std::mem::take(&mut self.counters); + self.counters.silent_peak = self.marks_by_since.len(); + (counters, tracked) + } +} + +/// What lets a parked key start now: an untried source below the per-source +/// cap, or a silent one. +#[derive(Clone, Copy)] +struct Startable<'a> { + in_flight: &'a HashMap, + per_source_cap: usize, + now: Instant, +} + +/// The next queued key, taken off the heap or out of the parked set. +/// +/// Returned and matched at once, never stored, so the size gap between the +/// variants costs nothing; boxing would add an allocation per dispatched fetch. +#[allow(clippy::large_enum_variant)] +enum TakenFetch { + /// The key, rejoined with its payload. + Candidate(FetchCandidate), + /// An inconsistent entry was dropped instead; try again. + Skipped, + /// The heap is empty and no parked key was asked for. + Empty, +} + +/// How [`ReplicationQueues::dispatch_fetches`] may start one candidate. +enum SourceChoice { + /// This untried source has a free share. + Start(PeerId), + /// Untried sources exist, but every one is at its share. + AllBusy, + /// No untried source is left. + NoSource, +} + +/// Test-only: every key queued or parked for fetch, and every in-flight fetch +/// with its source, read under one lock. +#[cfg(any(test, feature = "test-utils"))] +#[derive(Debug, Clone, Default)] +pub struct FetchPipelineSnapshot { + /// Keys queued for fetch, on the heap or parked. + pub queued: Vec, + /// In-flight fetches with the source each one was sent to. + pub in_flight: Vec<(XorName, PeerId)>, +} + // --------------------------------------------------------------------------- // Central queue manager // --------------------------------------------------------------------------- @@ -205,10 +640,11 @@ pub struct ReplicationQueues { /// Capacity-bounded by [`MAX_PENDING_VERIFY`]. At capacity, max-min sender /// accounting reclaims borrowed entries before rejecting new work. pending_verify: HashMap, - /// Nearest-first priority order over the keys in `fetch_payloads`. + /// Nearest-first priority order over the queued keys that are not parked. /// /// Holds ordering data only: every entry has exactly one matching - /// `fetch_payloads` entry, and vice versa. Splitting the mutable half of a + /// `fetch_payloads` entry, and every `fetch_payloads` entry is represented + /// exactly once, here or in `fetch_parked`. Splitting the mutable half of a /// queued fetch out into `fetch_payloads` keeps this heap's ordering /// immutable for as long as a key is queued, so a source merge never has /// to rebuild it. @@ -221,8 +657,30 @@ pub struct ReplicationQueues { /// Capacity-bounded by [`MAX_FETCH_QUEUE`]: enqueues are dropped once /// full, preventing unbounded growth under a network hint flood. fetch_payloads: HashMap, + /// Queued keys taken off `fetch_queue` because every untried source was at + /// its per-source share. Every key in `fetch_payloads` is in exactly one of + /// `fetch_queue` and this map. + fetch_parked: HashMap, + /// Exact reverse index over `fetch_parked`: `(distance, key)` is in the set + /// for `source` if and only if `key` is parked with `source` among its + /// wake sources. Ordered so the nearest parked key per source is `first()`; + /// empty sets are removed. + parked_by_source: HashMap>, + /// Sources a queued key was already fetched from in the current + /// verification cycle. Present only for a key that went back to the queue + /// from flight, and carried into its next in-flight entry. + fetch_tried: HashMap>, /// Active downloads keyed by `XorName`. in_flight_fetch: HashMap, + /// Most distinct sources a queued or in-flight key may be associated with, + /// tried ones included. See [`fetch_source_limit`]. + max_fetch_sources: usize, + /// Longest parked-index maintenance after a departed source since the + /// dispatch summary last read it. + parked_maintenance_max: Duration, + /// Which fetch sources stopped answering, so the per-source cap can leave + /// them out. + source_health: SourceHealthTable, /// Reverse index for removing a departed peer from every pending hint /// without scanning the entire pending table. pending_keys_by_source: HashMap>, @@ -272,7 +730,13 @@ impl ReplicationQueues { pending_verify: HashMap::new(), fetch_queue: BinaryHeap::new(), fetch_payloads: HashMap::new(), + fetch_parked: HashMap::new(), + parked_by_source: HashMap::new(), + fetch_tried: HashMap::new(), in_flight_fetch: HashMap::new(), + max_fetch_sources: MAX_FETCH_SOURCES_PER_KEY, + parked_maintenance_max: Duration::ZERO, + source_health: SourceHealthTable::new(FETCH_REQUEST_TIMEOUT), pending_keys_by_source: HashMap::new(), capacity_owner_by_key: HashMap::new(), pending_keys_by_owner: HashMap::new(), @@ -286,6 +750,39 @@ impl ReplicationQueues { } } + /// Use `limit` as the most distinct sources a queued or in-flight key may + /// be associated with, for close-group widths other than the production + /// ones. See [`fetch_source_limit`]. + #[must_use] + pub(crate) fn with_fetch_source_limit(mut self, limit: usize) -> Self { + self.set_fetch_source_limit(limit); + self + } + + /// Change the source limit. Only for queues no key has entered yet, such + /// as an engine's before it starts. + pub(crate) fn set_fetch_source_limit(&mut self, limit: usize) { + self.max_fetch_sources = limit.max(1); + } + + /// Use `timeout`, the fetch request timeout, to time how long a source + /// that stopped answering stays silent. Only for queues no fetch has + /// resolved in yet. + #[must_use] + pub(crate) fn with_fetch_request_timeout(mut self, timeout: Duration) -> Self { + self.source_health = SourceHealthTable::new(timeout); + self + } + + /// Apply whether a fetch attempt's `source` answered. Called for every + /// resolved attempt, before the attempt's result moves its key on, and for + /// an answer as soon as it arrives, while its attempt is still in flight. + /// Applying the same answer twice changes nothing. + pub(crate) fn record_fetch_contact(&mut self, source: PeerId, contact: Contact) { + self.source_health + .record(source, contact, Instant::now(), &self.in_flight_fetch); + } + // ----------------------------------------------------------------------- // PendingVerify // ----------------------------------------------------------------------- @@ -302,6 +799,7 @@ impl ReplicationQueues { /// source-side retry) MUST treat this as still-outstanding work, not as /// "done". Without this distinction a bootstrap snapshot whose hints /// are capacity-rejected would silently mark itself drained. + #[allow(clippy::too_many_lines)] pub fn add_pending_verify( &mut self, key: XorName, @@ -373,32 +871,66 @@ impl ReplicationQueues { return AdmissionResult::AlreadyPresent; } // Merging a source touches only `FetchPayload`, which no ordering - // reads, so the fetch heap is left completely untouched here. + // reads, so the fetch heap is left completely untouched here. A source + // this key was already fetched from this cycle is not merged back: it + // would be asked twice. A parked key can start from a merged source as + // soon as that source has a free share, so it is indexed there too. if let Some(payload) = self.fetch_payloads.get_mut(&key) { + let tried = self.fetch_tried.get(&key); + let mut merged = Vec::new(); for source in &entry.replica_hint_sources { - if !payload.sources.contains(source) { + // Payload sources and tried sources are disjoint, so this is + // the size of everything the key has been associated with. + if payload.sources.len() + tried.map_or(0, HashSet::len) >= self.max_fetch_sources { + break; + } + if !payload.sources.contains(source) && !tried.is_some_and(|t| t.contains(source)) { payload.sources.push(*source); + merged.push(*source); + } + } + if let Some(parked) = self.fetch_parked.get_mut(&key) { + for source in merged { + parked.wake_sources.push(source); + self.parked_by_source + .entry(source) + .or_default() + .insert((parked.distance, key)); } } if let Some(retry) = &mut payload.retry_verification { - retry - .replica_hint_sources - .extend(entry.replica_hint_sources); - retry.hint_sources.extend(entry.hint_sources); + Self::merge_retry_hints( + retry, + &entry.replica_hint_sources, + &entry.hint_sources, + self.capacity_owner_by_key.get(&key).copied(), + ); } return AdmissionResult::AlreadyPresent; } if let Some(in_flight) = self.in_flight_fetch.get_mut(&key) { + let mut associated = in_flight.all_sources.len() + + in_flight + .tried + .iter() + .filter(|peer| !in_flight.all_sources.contains(peer)) + .count(); for source in &entry.replica_hint_sources { - if !in_flight.all_sources.contains(source) { + if associated >= self.max_fetch_sources { + break; + } + if !in_flight.all_sources.contains(source) && !in_flight.tried.contains(source) { in_flight.all_sources.push(*source); + associated += 1; } } if let Some(retry) = &mut in_flight.retry_verification { - retry - .replica_hint_sources - .extend(entry.replica_hint_sources); - retry.hint_sources.extend(entry.hint_sources); + Self::merge_retry_hints( + retry, + &entry.replica_hint_sources, + &entry.hint_sources, + self.capacity_owner_by_key.get(&key).copied(), + ); } return AdmissionResult::AlreadyPresent; } @@ -652,10 +1184,7 @@ impl ReplicationQueues { key: &XorName, protected_owner: Option, ) -> bool { - let victim = entry.hint_sources.iter().copied().find(|source| { - !entry.replica_hint_sources.contains(source) && protected_owner != Some(*source) - }); - let Some(victim) = victim else { + let Some(victim) = Self::paid_only_victim(entry, protected_owner) else { return false; }; entry.hint_sources.remove(&victim); @@ -668,6 +1197,53 @@ impl ReplicationQueues { true } + /// The paid-only advertiser [`Self::displace_paid_only_source`] would drop + /// from `entry`: never a replica claimant, never `protected_owner`. + fn paid_only_victim( + entry: &VerificationEntry, + protected_owner: Option, + ) -> Option { + entry.hint_sources.iter().copied().find(|source| { + !entry.replica_hint_sources.contains(source) && protected_owner != Some(*source) + }) + } + + /// Merge a re-advertisement into the verification entry a fetch-stage key + /// keeps for a later retry, under the bound and in the order + /// [`Self::add_pending_verify`] applies to a pending entry: replica + /// claimants first, displacing paid-only advertisers other than + /// `protected_owner` when the set is full, then paid-only advertisers + /// while it is below [`MAX_HINT_SOURCES_PER_KEY`]. A fetch-stage key is in + /// no pending index, so only the entry changes. + fn merge_retry_hints( + retry: &mut VerificationEntry, + replica_hint_sources: &HashSet, + hint_sources: &HashSet, + protected_owner: Option, + ) { + for source in replica_hint_sources { + if retry.replica_hint_sources.contains(source) { + continue; + } + if !retry.hint_sources.contains(source) + && retry.hint_sources.len() >= MAX_HINT_SOURCES_PER_KEY + { + let Some(victim) = Self::paid_only_victim(retry, protected_owner) else { + break; + }; + retry.hint_sources.remove(&victim); + } + retry.replica_hint_sources.insert(*source); + retry.hint_sources.insert(*source); + } + for source in hint_sources { + if retry.hint_sources.len() >= MAX_HINT_SOURCES_PER_KEY { + break; + } + retry.hint_sources.insert(*source); + } + } + /// Reserve a pending slot for `key`, charged to its capacity owner. /// /// Must be called while `capacity_owner_by_key` still holds `key`, which is @@ -951,8 +1527,9 @@ impl ReplicationQueues { .retain(|key, _| !orphaned_fetch_keys.contains(key)); self.fetch_queue .retain(|order| !orphaned_fetch_keys.contains(&order.key)); - orphaned.extend(orphaned_fetch_keys); } + self.drop_departed_source_from_parked(source, &orphaned_fetch_keys); + orphaned.extend(orphaned_fetch_keys); for entry in self.in_flight_fetch.values_mut() { entry.all_sources.retain(|peer| peer != source); @@ -1114,7 +1691,7 @@ impl ReplicationQueues { &mut self, key: XorName, distance: XorName, - sources: Vec, + mut sources: Vec, retry_verification: Option, ) -> bool { if self.pending_verify.contains_key(&key) @@ -1130,6 +1707,7 @@ impl ReplicationQueues { ); return false; } + sources.truncate(self.max_fetch_sources); self.fetch_payloads.insert( key, FetchPayload { @@ -1138,6 +1716,10 @@ impl ReplicationQueues { }, ); self.fetch_queue.push(FetchOrder { key, distance }); + // A fresh enqueue starts a new verification cycle's fetch, so nothing + // asked in an earlier cycle is excluded. Cleared only now, after the + // insert succeeded: a refused enqueue leaves an existing entry alone. + self.fetch_tried.remove(&key); true } @@ -1184,39 +1766,84 @@ impl ReplicationQueues { enqueued } - /// Dequeue the nearest fetch candidate. + /// Dequeue the nearest fetch candidate, whether it is on the fetch heap or + /// parked. /// /// Returns `None` when the queue is empty. Silently skips candidates /// that are somehow already in-flight. Concurrency is enforced by the /// fetch worker, not by this method. /// /// A returned candidate may carry a live verification retry-slot - /// reservation. Callers must consume it with - /// [`Self::start_dequeued_fetch`], [`Self::discard_fetch_candidate`], or + /// reservation, and the sources it was already fetched from this cycle. + /// Callers must consume it with [`Self::start_dequeued_fetch`], + /// [`Self::discard_fetch_candidate`], or /// [`Self::requeue_candidate_for_verification`] so that reservation is /// either transferred, released, or restored to `pending_verify`. pub fn dequeue_fetch(&mut self) -> Option { - while let Some(order) = self.fetch_queue.pop() { - let Some(payload) = self.fetch_payloads.remove(&order.key) else { - debug_assert!( - false, - "fetch order without payload for {}", - hex::encode(order.key) - ); - continue; - }; - let candidate = FetchCandidate { - key: order.key, - distance: order.distance, - sources: payload.sources, - retry_verification: payload.retry_verification, + loop { + let parked = self.nearest_parked(None); + let candidate = match self.take_queued(self.parked_if_nearer(parked)) { + TakenFetch::Candidate(candidate) => candidate, + TakenFetch::Skipped => continue, + TakenFetch::Empty => return None, }; if !self.in_flight_fetch.contains_key(&candidate.key) { return Some(candidate); } - self.release_retry_slot_for_candidate(&candidate); + self.drop_duplicate_of_in_flight(&candidate); } - None + } + + /// `parked`'s key when it is nearer than the heap head, or the heap is + /// empty; `None` to take the heap head. + fn parked_if_nearer(&self, parked: Option<(XorName, XorName)>) -> Option { + let heap_head = self + .fetch_queue + .peek() + .map(|order| (order.distance, order.key)); + match (heap_head, parked) { + (Some(head), Some(parked)) if parked < head => Some(parked.1), + (None, Some(parked)) => Some(parked.1), + _ => None, + } + } + + /// Remove a queued key from where it waits, the parked key `parked` or + /// otherwise the heap head, and rejoin it with its payload. + fn take_queued(&mut self, parked: Option) -> TakenFetch { + let (key, distance) = if let Some(key) = parked { + let Some(distance) = self.unpark(&key) else { + self.purge_parked_index_entry(&key); + return TakenFetch::Skipped; + }; + (key, distance) + } else { + let Some(order) = self.fetch_queue.pop() else { + return TakenFetch::Empty; + }; + (order.key, order.distance) + }; + let Some(payload) = self.fetch_payloads.remove(&key) else { + debug_assert!( + false, + "queued fetch without payload for {}", + hex::encode(key) + ); + return TakenFetch::Skipped; + }; + TakenFetch::Candidate(FetchCandidate { + key, + distance, + sources: payload.sources, + retry_verification: payload.retry_verification, + }) + } + + /// Discard a dequeued candidate whose key is somehow already in flight, + /// releasing what it carried. + fn drop_duplicate_of_in_flight(&mut self, candidate: &FetchCandidate) { + self.release_retry_slot_for_candidate(candidate); + self.fetch_tried.remove(&candidate.key); } /// Number of keys waiting in the fetch queue. @@ -1253,8 +1880,41 @@ impl ReplicationQueues { all_sources: Vec, retry_verification: Option, ) { - let mut tried = HashSet::new(); + self.start_fetch_after(key, source, all_sources, retry_verification, HashSet::new()); + } + + /// Start `key` from `source`, carrying the sources it was already fetched + /// from this cycle into the in-flight entry's tried set. + fn start_fetch_after( + &mut self, + key: XorName, + source: PeerId, + mut all_sources: Vec, + retry_verification: Option, + mut tried: HashSet, + ) { tried.insert(source); + // Keep the distinct union of tried and listed sources within the + // limit by dropping untried sources from the end of the list. Every + // dispatched start is already within it; this binds direct starts. + let union = all_sources.len() + + tried + .iter() + .filter(|peer| !all_sources.contains(peer)) + .count(); + let mut excess = union.saturating_sub(self.max_fetch_sources); + if excess > 0 { + let mut kept = Vec::with_capacity(all_sources.len()); + for peer in all_sources.into_iter().rev() { + if excess > 0 && !tried.contains(&peer) { + excess -= 1; + } else { + kept.push(peer); + } + } + kept.reverse(); + all_sources = kept; + } let replaced = self.in_flight_fetch.insert( key, InFlightEntry { @@ -1271,8 +1931,9 @@ impl ReplicationQueues { } } - /// Consume a dequeued fetch candidate and transfer its retry reservation - /// into the in-flight entry. + /// Consume a dequeued fetch candidate and transfer its retry reservation, + /// and the sources it was already fetched from this cycle, into the + /// in-flight entry. pub fn start_dequeued_fetch(&mut self, candidate: FetchCandidate, source: PeerId) { let FetchCandidate { key, @@ -1280,7 +1941,8 @@ impl ReplicationQueues { retry_verification, .. } = candidate; - self.start_fetch_with_retry(key, source, sources, retry_verification); + let tried = self.fetch_tried.remove(&key).unwrap_or_default(); + self.start_fetch_after(key, source, sources, retry_verification, tried); } /// Mark a fetch as completed (success or permanent failure). @@ -1300,6 +1962,7 @@ impl ReplicationQueues { retry_verification, .. } = candidate; + self.fetch_tried.remove(&key); if retry_verification.is_some() { self.release_retry_slot(&key); self.capacity_owner_by_key.remove(&key); @@ -1329,59 +1992,454 @@ impl ReplicationQueues { } } - /// Consume a dequeued candidate and restore its verification entry for a - /// later retry when retry metadata exists. - pub fn requeue_candidate_for_verification( + // ----------------------------------------------------------------------- + // Per-source dispatch + // ----------------------------------------------------------------------- + + /// Start up to `slots` queued fetches without putting more than + /// `per_source_cap` of this node's fetches in flight to any one source + /// that is answering. + /// + /// Each slot goes to the nearest queued key that can start now: one with + /// an untried source below the cap, or an untried silent source. It starts + /// on the first untried source below the cap that is not silent, in its + /// source-preference order, or else on its first untried silent source, + /// past the cap. A heap key that cannot start is parked under each of its + /// untried sources, so it leaves the heap once and a startable key behind + /// it is reached in the same pass; a parked key becomes a candidate again + /// as soon as one of those sources has a free share or falls silent, and + /// competes on distance with the heap. + /// + /// With `per_source_cap = usize::MAX` nothing parks and every key starts on + /// its first untried source that is not silent, in plain nearest-first + /// order. + pub(crate) fn dispatch_fetches( &mut self, - candidate: FetchCandidate, - retry_after: Duration, - ) -> bool { + slots: usize, + per_source_cap: usize, + ) -> FetchDispatch { + let now = Instant::now(); + self.source_health.expire(now); + let mut dispatch = FetchDispatch::default(); + let mut in_flight = self.in_flight_counts_by_source(None); + let mut best_parked = self.nearest_parked(Some(Startable { + in_flight: &in_flight, + per_source_cap, + now, + })); + while dispatch.started.len() < slots { + let from_parked = self.parked_if_nearer(best_parked); + let candidate = match self.take_queued(from_parked) { + TakenFetch::Candidate(candidate) => Some(candidate), + TakenFetch::Skipped => None, + TakenFetch::Empty => break, + }; + let mut started = false; + if let Some(candidate) = candidate { + if self.in_flight_fetch.contains_key(&candidate.key) { + self.drop_duplicate_of_in_flight(&candidate); + } else { + let startable = Startable { + in_flight: &in_flight, + per_source_cap, + now, + }; + match self.first_startable_source(&candidate, startable) { + SourceChoice::Start(source) => { + if self.source_health.is_silent(&source, now) { + self.source_health.counters.silent_starts += 1; + } + *in_flight.entry(source).or_default() += 1; + dispatch.started.push((candidate.key, source)); + self.start_dequeued_fetch(candidate, source); + started = true; + } + SourceChoice::NoSource => dispatch.sourceless.push(candidate), + SourceChoice::AllBusy => { + if from_parked.is_none() { + dispatch.parked += 1; + } + self.park_candidate(candidate); + } + } + } + } + // A start changes the per-source counts, and a key taken from the + // parked set changes the set itself. A heap key that parked is + // indexed only under full sources that are not silent, so it + // cannot change the answer. + if started || from_parked.is_some() { + best_parked = self.nearest_parked(Some(Startable { + in_flight: &in_flight, + per_source_cap, + now, + })); + } + } + dispatch + } + + /// Put a dequeued candidate back in the queue, parked under its untried + /// sources. + fn park_candidate(&mut self, candidate: FetchCandidate) { let FetchCandidate { key, + distance, + sources, retry_verification, - .. } = candidate; - let Some(mut verification) = retry_verification else { - return false; - }; - verification.state = VerificationState::PendingVerify; - verification.verified_sources.clear(); - verification.tried_sources.clear(); - verification.next_verify_at = Instant::now() + retry_after; + self.fetch_payloads.insert( + key, + FetchPayload { + sources, + retry_verification, + }, + ); + self.park(key, distance); + } - let owner = self - .capacity_owner_by_key - .get(&key) + /// Mark the current fetch attempt as failed and move `key` on, without + /// sending to a source that is answering and already has `per_source_cap` + /// of this node's fetches in flight. + /// + /// The next untried source with a free share that is not silent is + /// returned to retry from now, as [`Self::retry_fetch`] would, or else the + /// first untried silent source, past the cap. If every untried source is + /// answering and at its share, the key goes back to the fetch queue + /// parked, with only its untried sources and with its tried set kept, so + /// no source is asked twice this cycle; its retry reservation travels with + /// it. If no untried source is left, or the fetch queue is at + /// [`MAX_FETCH_QUEUE`], the in-flight entry is left for the caller's + /// verification requeue, which keeps `MAX_FETCH_QUEUE` a hard bound. + pub(crate) fn retry_or_park_fetch( + &mut self, + key: &XorName, + distance: XorName, + per_source_cap: usize, + ) -> FetchRetry { + let now = Instant::now(); + self.source_health.expire(now); + let in_flight = self.in_flight_counts_by_source(Some(key)); + let queue_full = self.fetch_payloads.len() >= MAX_FETCH_QUEUE; + let Some(entry) = self.in_flight_fetch.get_mut(key) else { + return FetchRetry::Exhausted; + }; + entry.tried.insert(entry.source); + let untried: Vec = entry + .all_sources + .iter() + .filter(|peer| !entry.tried.contains(peer)) .copied() - .or_else(|| self.least_loaded_source(&verification.hint_sources)); - let Some(owner) = owner else { - self.release_retry_slot(&key); - return false; + .collect(); + if untried.is_empty() { + return FetchRetry::Exhausted; + } + let health = &self.source_health; + let free = untried + .iter() + .find(|peer| { + !health.is_silent(peer, now) + && in_flight.get(peer).copied().unwrap_or(0) < per_source_cap + }) + .copied(); + let silent = || { + untried + .iter() + .find(|peer| health.is_silent(peer, now)) + .copied() }; - // Release before re-inserting: the reservation is charged to the owner - // recorded now, which `insert_pending_owned_unchecked` may replace. - self.release_retry_slot(&key); - self.insert_pending_owned_unchecked(key, verification, owner, true); - true + let next = free.or_else(silent); + if let Some(next) = next { + if free.is_none() { + self.source_health.counters.silent_starts += 1; + } + entry.source = next; + entry.tried.insert(next); + entry.started_at = now; + return FetchRetry::Next(next); + } + if queue_full { + return FetchRetry::Exhausted; + } + let Some(entry) = self.in_flight_fetch.remove(key) else { + return FetchRetry::Exhausted; + }; + self.fetch_payloads.insert( + *key, + FetchPayload { + sources: untried, + retry_verification: entry.retry_verification, + }, + ); + self.fetch_tried.insert(*key, entry.tried); + self.park(*key, distance); + FetchRetry::Parked } - /// Complete an exhausted fetch and restore its verification entry for a - /// later retry when retry metadata exists. - pub fn requeue_fetch_for_verification(&mut self, key: &XorName, retry_after: Duration) -> bool { - let Some(mut entry) = self.in_flight_fetch.remove(key) else { - return false; - }; - let Some(mut verification) = entry.retry_verification.take() else { - self.capacity_owner_by_key.remove(key); - return false; - }; - verification.state = VerificationState::PendingVerify; - verification.verified_sources.clear(); - verification.tried_sources.clear(); - verification.next_verify_at = Instant::now() + retry_after; + /// In-flight fetches per source, optionally leaving out `exclude`'s own + /// entry. + fn in_flight_counts_by_source(&self, exclude: Option<&XorName>) -> HashMap { + let mut counts = HashMap::new(); + for (key, entry) in &self.in_flight_fetch { + if exclude != Some(key) { + *counts.entry(entry.source).or_default() += 1; + } + } + counts + } - let owner = self - .capacity_owner_by_key + /// The first untried source of `candidate`, in its source-preference + /// order, that is not silent and is below the cap, or else its first + /// untried silent source. + fn first_startable_source( + &self, + candidate: &FetchCandidate, + startable: Startable<'_>, + ) -> SourceChoice { + let tried = self.fetch_tried.get(&candidate.key); + let mut any_untried = false; + let mut first_silent = None; + for source in &candidate.sources { + if tried.is_some_and(|t| t.contains(source)) { + continue; + } + any_untried = true; + if self.source_health.is_silent(source, startable.now) { + first_silent = first_silent.or(Some(*source)); + } else if startable.in_flight.get(source).copied().unwrap_or(0) + < startable.per_source_cap + { + return SourceChoice::Start(*source); + } + } + match (first_silent, any_untried) { + (Some(source), _) => SourceChoice::Start(source), + (None, true) => SourceChoice::AllBusy, + (None, false) => SourceChoice::NoSource, + } + } + + /// Park a queued key whose payload is in `fetch_payloads`, indexed under + /// each of its untried sources. A key with no untried source goes back on + /// the heap instead, so it is never parked with nothing to wake it. + fn park(&mut self, key: XorName, distance: XorName) { + self.unpark(&key); + let tried = self.fetch_tried.get(&key); + let wake_sources: Vec = self + .fetch_payloads + .get(&key) + .map(|payload| { + payload + .sources + .iter() + .filter(|source| !tried.is_some_and(|t| t.contains(source))) + .copied() + .collect() + }) + .unwrap_or_default(); + if wake_sources.is_empty() { + self.fetch_queue.push(FetchOrder { key, distance }); + return; + } + for source in &wake_sources { + self.parked_by_source + .entry(*source) + .or_default() + .insert((distance, key)); + } + self.fetch_parked.insert( + key, + ParkedFetch { + distance, + wake_sources, + parked_at: Instant::now(), + }, + ); + } + + /// The per-source scheduler's state for its periodic summary, resetting + /// the maintenance maximum and source-health counters it reports and + /// purging the health table. Walks the parked set, the index's per-source + /// sets, the health table and the in-flight fetches once each. + pub(crate) fn take_fetch_dispatch_stats( + &mut self, + per_source_cap: usize, + ) -> FetchDispatchStats { + let now = Instant::now(); + let parked_maintenance_max = std::mem::take(&mut self.parked_maintenance_max); + let (health, tracked_sources) = + self.source_health.take_counters(now, &self.in_flight_fetch); + FetchDispatchStats { + tracked_sources, + silent_sources: self.source_health.marks_by_since.len(), + overflow_active: self.source_health.overflowing(now), + health, + parked_associations: self.parked_by_source.values().map(BTreeSet::len).sum(), + parked_maintenance_max, + parked: self.fetch_parked.len(), + oldest_parked: self + .fetch_parked + .values() + .map(|parked| now.saturating_duration_since(parked.parked_at)) + .max(), + queued: self.fetch_payloads.len(), + in_flight: self.in_flight_fetch.len(), + saturated_sources: self + .in_flight_counts_by_source(None) + .values() + .filter(|count| **count >= per_source_cap) + .count(), + } + } + + /// The parked-state part of [`Self::remove_hint_source`]: orphaned keys + /// leave it entirely, and a parked key can no longer start from the + /// departed source. One left with no source to wake it goes back on the + /// heap, where dispatch hands it out as sourceless, exactly as a queued key + /// that lost its last source. Timed for the dispatch summary. + fn drop_departed_source_from_parked( + &mut self, + source: &PeerId, + orphaned_fetch_keys: &HashSet, + ) { + if self.fetch_parked.is_empty() && self.fetch_tried.is_empty() { + return; + } + let started = Instant::now(); + for key in orphaned_fetch_keys { + self.unpark(key); + self.fetch_tried.remove(key); + } + if let Some(parked_here) = self.parked_by_source.remove(source) { + for (distance, key) in parked_here { + let Some(parked) = self.fetch_parked.get_mut(&key) else { + continue; + }; + parked.wake_sources.retain(|peer| peer != source); + if parked.wake_sources.is_empty() { + self.fetch_parked.remove(&key); + self.fetch_queue.push(FetchOrder { key, distance }); + } + } + } + self.parked_maintenance_max = self.parked_maintenance_max.max(started.elapsed()); + } + + /// Take `key` out of the parked state and the index, returning its + /// distance. The payload is left in `fetch_payloads`. + fn unpark(&mut self, key: &XorName) -> Option { + let parked = self.fetch_parked.remove(key)?; + for source in &parked.wake_sources { + if let Some(keys) = self.parked_by_source.get_mut(source) { + keys.remove(&(parked.distance, *key)); + if keys.is_empty() { + self.parked_by_source.remove(source); + } + } + } + Some(parked.distance) + } + + /// The nearest parked key, as `(distance, key)`. With `startable`, only + /// keys indexed under a source below the cap or silent count, so the + /// result is the nearest parked key that could start now. + fn nearest_parked(&self, startable: Option>) -> Option<(XorName, XorName)> { + self.parked_by_source + .iter() + .filter(|(source, _)| { + startable.is_none_or(|startable| { + startable.in_flight.get(source).copied().unwrap_or(0) < startable.per_source_cap + || self.source_health.is_silent(source, startable.now) + }) + }) + .filter_map(|(_, keys)| keys.first().copied()) + .min() + } + + /// Defensive: drop every index entry for `key` when the index names a key + /// that is not parked. Unreachable while the index stays exact; without it + /// such an entry would be offered forever. + fn purge_parked_index_entry(&mut self, key: &XorName) { + debug_assert!( + false, + "parked index names an unparked key {}", + hex::encode(key) + ); + self.parked_by_source.retain(|_, keys| { + keys.retain(|(_, parked)| parked != key); + !keys.is_empty() + }); + } + + /// Test-only: every queued or parked key, and every in-flight fetch with + /// its source, read together. + #[cfg(any(test, feature = "test-utils"))] + #[must_use] + pub fn fetch_pipeline_snapshot(&self) -> FetchPipelineSnapshot { + FetchPipelineSnapshot { + queued: self.fetch_payloads.keys().copied().collect(), + in_flight: self + .in_flight_fetch + .iter() + .map(|(key, entry)| (*key, entry.source)) + .collect(), + } + } + + /// Consume a dequeued candidate and restore its verification entry for a + /// later retry when retry metadata exists. + pub fn requeue_candidate_for_verification( + &mut self, + candidate: FetchCandidate, + retry_after: Duration, + ) -> bool { + let FetchCandidate { + key, + retry_verification, + .. + } = candidate; + self.fetch_tried.remove(&key); + let Some(mut verification) = retry_verification else { + return false; + }; + verification.state = VerificationState::PendingVerify; + verification.verified_sources.clear(); + verification.tried_sources.clear(); + verification.next_verify_at = Instant::now() + retry_after; + + let owner = self + .capacity_owner_by_key + .get(&key) + .copied() + .or_else(|| self.least_loaded_source(&verification.hint_sources)); + let Some(owner) = owner else { + self.release_retry_slot(&key); + return false; + }; + // Release before re-inserting: the reservation is charged to the owner + // recorded now, which `insert_pending_owned_unchecked` may replace. + self.release_retry_slot(&key); + self.insert_pending_owned_unchecked(key, verification, owner, true); + true + } + + /// Complete an exhausted fetch and restore its verification entry for a + /// later retry when retry metadata exists. + pub fn requeue_fetch_for_verification(&mut self, key: &XorName, retry_after: Duration) -> bool { + let Some(mut entry) = self.in_flight_fetch.remove(key) else { + return false; + }; + let Some(mut verification) = entry.retry_verification.take() else { + self.capacity_owner_by_key.remove(key); + return false; + }; + verification.state = VerificationState::PendingVerify; + verification.verified_sources.clear(); + verification.tried_sources.clear(); + verification.next_verify_at = Instant::now() + retry_after; + + let owner = self + .capacity_owner_by_key .get(key) .copied() .or_else(|| self.least_loaded_source(&verification.hint_sources)); @@ -1396,6 +2454,11 @@ impl ReplicationQueues { true } + /// When `key`'s current fetch attempt started, if it is in flight. + pub(crate) fn in_flight_started_at(&self, key: &XorName) -> Option { + self.in_flight_fetch.get(key).map(|entry| entry.started_at) + } + /// Number of in-flight fetches. #[must_use] pub fn in_flight_count(&self) -> usize { @@ -1516,6 +2579,8 @@ fn max_min_allocations( mod tests { use super::*; use crate::replication::config::CAPACITY_BLOCKED_RETRY; + use rand::rngs::StdRng; + use rand::{Rng, SeedableRng}; /// Build a `PeerId` from a single byte (zero-padded to 32 bytes). fn peer_id_from_byte(b: u8) -> PeerId { @@ -3026,4 +4091,1493 @@ mod tests { ); assert_eq!(queues.pending_count(), 1); } + + // -- per-source fetch dispatch --------------------------------------------- + + const CAP: usize = 2; + + fn peer(b: u8) -> PeerId { + peer_id_from_byte(b) + } + + /// A key whose byte order matches the numeric order of `distance`. + fn key_at(distance: u32) -> XorName { + let mut name = [0u8; 32]; + name[..4].copy_from_slice(&distance.to_be_bytes()); + name + } + + /// Queue `key` at `distance` with `sources`, keyed at its own distance so + /// tests can reason about nearest-first order directly. + fn queue(queues: &mut ReplicationQueues, distance: u32, sources: &[PeerId]) -> XorName { + let key = key_at(distance); + assert!(queues.enqueue_fetch(key, key, sources.to_vec())); + key + } + + /// Queue `key` through pending verification and promotion, so it carries a + /// verification retry reservation owned by `hinter`. + fn queue_promoted( + queues: &mut ReplicationQueues, + distance: u32, + hinter: u8, + sources: &[PeerId], + ) -> XorName { + let key = key_at(distance); + assert!(queues + .add_pending_verify(key, test_entry(hinter)) + .admitted()); + assert!(queues.promote_pending_to_fetch(key, key, sources.to_vec())); + key + } + + fn in_flight_to(queues: &ReplicationQueues, source: &PeerId) -> usize { + queues + .in_flight_fetch + .values() + .filter(|entry| entry.source == *source) + .count() + } + + fn heap_keys(queues: &ReplicationQueues) -> HashSet { + queues.fetch_queue.iter().map(|order| order.key).collect() + } + + /// I1 and I2: every queued payload is on the heap or parked, never both, + /// and the reverse index is exactly what `fetch_parked` implies. + fn assert_queue_invariants(queues: &ReplicationQueues) { + let heap = heap_keys(queues); + assert_eq!(heap.len(), queues.fetch_queue.len(), "duplicate heap entry"); + for key in queues.fetch_payloads.keys() { + let on_heap = heap.contains(key); + let parked = queues.fetch_parked.contains_key(key); + assert!( + on_heap != parked, + "queued key must be on exactly one of heap/parked" + ); + } + for key in heap.iter().chain(queues.fetch_parked.keys()) { + assert!( + queues.fetch_payloads.contains_key(key), + "order without payload" + ); + } + let mut expected: HashMap> = HashMap::new(); + for (key, parked) in &queues.fetch_parked { + assert!( + !parked.wake_sources.is_empty(), + "parked key with nothing to wake it" + ); + for source in &parked.wake_sources { + expected + .entry(*source) + .or_default() + .insert((parked.distance, *key)); + } + } + assert_eq!( + queues.parked_by_source, expected, + "parked index is not exact" + ); + for key in queues.fetch_tried.keys() { + assert!( + queues.fetch_payloads.contains_key(key), + "tried sources kept for a key that is not queued" + ); + } + } + + #[test] + fn dispatch_spreads_fetches_across_sources_within_the_cap() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let keys: Vec = (0..8).map(|d| queue(&mut queues, d, &[a, b, c])).collect(); + + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!(dispatch.started.len(), 6, "two per source, three sources"); + assert!(dispatch.sourceless.is_empty()); + for source in [a, b, c] { + assert_eq!(in_flight_to(&queues, &source), CAP); + } + // Preference order is kept: the nearest keys take the first source. + let expected: Vec<(XorName, PeerId)> = keys + .iter() + .take(6) + .zip([a, a, b, b, c, c]) + .map(|(key, source)| (*key, source)) + .collect(); + assert_eq!(dispatch.started, expected); + assert_eq!(queues.fetch_parked.len(), 2); + assert_queue_invariants(&queues); + } + + #[test] + fn uncapped_dispatch_keeps_the_first_source_order() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + for d in 0..5 { + queue(&mut queues, d, &[a, b]); + } + + let dispatch = queues.dispatch_fetches(8, usize::MAX); + + assert_eq!(dispatch.started.len(), 5); + assert!(dispatch.started.iter().all(|(_, source)| *source == a)); + assert!(queues.fetch_parked.is_empty()); + assert_queue_invariants(&queues); + } + + #[test] + fn a_startable_key_behind_blocked_keys_starts_in_the_same_pass() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + assert_eq!(queues.dispatch_fetches(8, CAP).started.len(), 2); + for d in 10..310 { + queue(&mut queues, d, &[a]); + } + let deeper = queue(&mut queues, 1_000, &[b]); + + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!(dispatch.started, vec![(deeper, b)]); + assert_eq!( + queues.fetch_parked.len(), + 300, + "every blocked key left the heap" + ); + assert!(queues.fetch_queue.is_empty()); + assert_eq!( + queues.fetch_queue_count(), + 300, + "parked keys are still queued" + ); + assert_queue_invariants(&queues); + } + + #[test] + fn a_freed_share_starts_the_nearest_parked_key_for_that_source() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let first = queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + for d in [40, 20, 30] { + queue(&mut queues, d, &[a]); + } + assert_eq!(queues.dispatch_fetches(8, CAP).started.len(), 2); + assert_eq!(queues.fetch_parked.len(), 3); + + queues.complete_fetch(&first); + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!(dispatch.started, vec![(key_at(20), a)]); + assert_eq!(in_flight_to(&queues, &a), CAP); + assert_queue_invariants(&queues); + } + + #[test] + fn the_nearest_startable_parked_key_wins_whatever_its_source() { + let mut queues = ReplicationQueues::new(); + let sources: Vec = (1..=5).map(peer).collect(); + // Saturate every source with two blockers, then park one key per source. + let mut blockers = Vec::new(); + for (index, source) in sources.iter().enumerate() { + let base = u32::try_from(index).unwrap() * 2; + blockers.push(queue(&mut queues, base, &[*source])); + blockers.push(queue(&mut queues, base + 1, &[*source])); + } + assert_eq!(queues.dispatch_fetches(10, CAP).started.len(), 10); + let parked_distances = [500, 100, 400, 200, 300]; + for (source, distance) in sources.iter().zip(parked_distances) { + queue(&mut queues, distance, &[*source]); + } + assert_eq!(queues.dispatch_fetches(5, CAP).started, Vec::new()); + assert_eq!(queues.fetch_parked.len(), 5); + + // Free one share on every source: the two slots go to the two nearest. + for pair in blockers.chunks(2) { + queues.complete_fetch(&pair[0]); + } + let dispatch = queues.dispatch_fetches(2, CAP); + + assert_eq!( + dispatch.started, + vec![(key_at(100), sources[1]), (key_at(200), sources[3])] + ); + assert_queue_invariants(&queues); + } + + #[test] + fn a_parked_key_starts_on_its_preferred_free_source() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + let blockers: Vec = (0..4) + .map(|d| queue(&mut queues, d, &[if d < 2 { a } else { b }])) + .collect(); + assert_eq!(queues.dispatch_fetches(4, CAP).started.len(), 4); + let key = queue(&mut queues, 50, &[a, b]); + assert_eq!(queues.dispatch_fetches(1, CAP).started, Vec::new()); + + // Free b first, then a: both have room when dispatch runs. + queues.complete_fetch(&blockers[2]); + queues.complete_fetch(&blockers[0]); + let dispatch = queues.dispatch_fetches(1, CAP); + + assert_eq!(dispatch.started, vec![(key, a)]); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_moves_to_a_free_untried_source() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + queue(&mut queues, 10, &[b]); + queue(&mut queues, 11, &[b]); + let key = queue(&mut queues, 0, &[a, b, c]); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 3); + assert_eq!(queues.in_flight_fetch[&key].source, a); + + let retry = queues.retry_or_park_fetch(&key, key, CAP); + + assert_eq!(retry, FetchRetry::Next(c), "b is at its share"); + let entry = &queues.in_flight_fetch[&key]; + assert_eq!(entry.source, c); + assert_eq!(entry.tried, HashSet::from([a, c])); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_with_every_untried_source_full_parks_and_keeps_its_history() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = queue_promoted(&mut queues, 0, 9, &[a, b, c]); + let blockers: Vec = (10..14) + .map(|d| queue(&mut queues, d, &[if d < 12 { b } else { c }])) + .collect(); + assert_eq!(queues.dispatch_fetches(5, CAP).started.len(), 5); + assert_eq!(queues.in_flight_fetch[&key].source, a); + let reserved = queues.retry_reserved_slot_count(); + let owner = queues.capacity_owner_by_key.get(&key).copied(); + let reserved_by_owner = queues.retry_reserved_by_owner.clone(); + assert_eq!(reserved_by_owner.get(&peer(9)), Some(&1)); + + let retry = queues.retry_or_park_fetch(&key, key, CAP); + + assert_eq!(retry, FetchRetry::Parked); + assert!(!queues.in_flight_fetch.contains_key(&key)); + assert_eq!(queues.fetch_payloads[&key].sources, vec![b, c]); + assert!(queues.fetch_payloads[&key].retry_verification.is_some()); + assert_eq!(queues.fetch_tried[&key], HashSet::from([a])); + assert_eq!(queues.fetch_parked[&key].wake_sources, vec![b, c]); + assert_eq!(queues.retry_reserved_slot_count(), reserved); + assert_eq!(queues.capacity_owner_by_key.get(&key).copied(), owner); + assert_eq!(queues.retry_reserved_by_owner, reserved_by_owner); + assert!(queues.contains_key(&key)); + assert_queue_invariants(&queues); + + // A share on b frees: the key restarts there, still remembering a. + queues.complete_fetch(&blockers[0]); + let dispatch = queues.dispatch_fetches(1, CAP); + assert_eq!(dispatch.started, vec![(key, b)]); + assert_eq!(queues.in_flight_fetch[&key].tried, HashSet::from([a, b])); + assert!(!queues.fetch_tried.contains_key(&key)); + assert_eq!(queues.retry_reserved_slot_count(), reserved); + assert_eq!(queues.capacity_owner_by_key.get(&key).copied(), owner); + assert_eq!(queues.retry_reserved_by_owner, reserved_by_owner); + assert_queue_invariants(&queues); + } + + #[test] + fn a_parked_key_wakes_through_any_of_its_sources_beyond_the_eighth() { + let mut queues = ReplicationQueues::new(); + let sources: Vec = (1..=9).map(peer).collect(); + let mut blockers = Vec::new(); + for (index, source) in sources.iter().enumerate() { + let base = u32::try_from(index).unwrap() * 2; + blockers.push(queue(&mut queues, base, &[*source])); + blockers.push(queue(&mut queues, base + 1, &[*source])); + } + assert_eq!(queues.dispatch_fetches(18, CAP).started.len(), 18); + let key = queue(&mut queues, 100, &sources); + assert_eq!(queues.dispatch_fetches(1, CAP).started, Vec::new()); + assert_eq!(queues.fetch_parked[&key].wake_sources, sources); + + // Only the ninth source frees a share. + queues.complete_fetch(&blockers[16]); + let dispatch = queues.dispatch_fetches(1, CAP); + + assert_eq!(dispatch.started, vec![(key, sources[8])]); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_with_no_untried_source_is_exhausted_and_left_in_flight() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let key = queue(&mut queues, 0, &[a]); + assert_eq!(queues.dispatch_fetches(1, CAP).started.len(), 1); + + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Exhausted + ); + assert!(queues.in_flight_fetch.contains_key(&key)); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_with_a_full_fetch_queue_is_exhausted_rather_than_overfilling_it() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + let key = queue(&mut queues, 0, &[a, b]); + queue(&mut queues, 1, &[b]); + queue(&mut queues, 2, &[b]); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 3); + let mut filler = 1_000_u32; + while queues.fetch_queue_count() < MAX_FETCH_QUEUE { + queue(&mut queues, filler, &[peer(7)]); + filler += 1; + } + + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Exhausted + ); + assert!(queues.in_flight_fetch.contains_key(&key)); + assert_eq!(queues.fetch_queue_count(), MAX_FETCH_QUEUE); + } + + /// Park `key` (sources a, b, c) with a already tried and b, c full. + fn park_after_trying_a( + queues: &mut ReplicationQueues, + a: PeerId, + b: PeerId, + c: PeerId, + ) -> XorName { + let key = queue(queues, 0, &[a, b, c]); + for d in 10..14 { + queue(queues, d, &[if d < 12 { b } else { c }]); + } + assert_eq!(queues.dispatch_fetches(5, CAP).started.len(), 5); + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Parked + ); + key + } + + #[test] + fn a_tried_source_readvertised_for_a_parked_key_is_not_merged_back() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + + let result = queues.add_pending_verify(key, test_entry(1)); + + assert_eq!(result, AdmissionResult::AlreadyPresent); + assert_eq!(queues.fetch_payloads[&key].sources, vec![b, c]); + assert_eq!(queues.fetch_parked[&key].wake_sources, vec![b, c]); + assert_queue_invariants(&queues); + } + + #[test] + fn a_new_source_merged_into_a_parked_key_can_start_it() { + let mut queues = ReplicationQueues::new(); + let (a, b, c, d) = (peer(1), peer(2), peer(3), peer(4)); + let key = park_after_trying_a(&mut queues, a, b, c); + + assert_eq!( + queues.add_pending_verify(key, test_entry(4)), + AdmissionResult::AlreadyPresent + ); + assert_eq!(queues.fetch_parked[&key].wake_sources, vec![b, c, d]); + assert_queue_invariants(&queues); + + let dispatch = queues.dispatch_fetches(1, CAP); + assert_eq!(dispatch.started, vec![(key, d)]); + assert_eq!(queues.in_flight_fetch[&key].tried, HashSet::from([a, d])); + assert_queue_invariants(&queues); + } + + #[test] + fn an_in_flight_merge_skips_tried_sources_and_respects_the_limit() { + let mut queues = ReplicationQueues::new().with_fetch_source_limit(4); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + // Restart it on b, carrying a as tried: associated sources are a, b, c. + let blocker = queues + .in_flight_fetch + .iter() + .find(|(_, entry)| entry.source == b) + .map(|(key, _)| *key) + .unwrap(); + queues.complete_fetch(&blocker); + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, b)]); + + // a was tried: not merged. 5 is new and fits (4 associated). 6 does not. + for hinter in [1, 5, 6] { + queues.add_pending_verify(key, test_entry(hinter)); + } + + assert_eq!( + queues.in_flight_fetch[&key].all_sources, + vec![b, c, peer(5)] + ); + assert_queue_invariants(&queues); + } + + /// A re-advertisement from `sender` that claims no replica. + fn paid_only_entry(sender: u8) -> VerificationEntry { + let mut entry = test_entry(sender); + entry.replica_hint_sources.clear(); + entry + } + + /// Every fetch-stage key's retry entry is within the hint bound, as a + /// pending entry is, and still holds the key's capacity owner. + fn assert_retry_entries_bounded(queues: &ReplicationQueues) { + let retries = queues + .fetch_payloads + .iter() + .filter_map(|(key, payload)| Some((key, payload.retry_verification.as_ref()?))) + .chain( + queues + .in_flight_fetch + .iter() + .filter_map(|(key, entry)| Some((key, entry.retry_verification.as_ref()?))), + ); + for (key, retry) in retries { + assert!( + retry.hint_sources.len() <= MAX_HINT_SOURCES_PER_KEY, + "{} retry entry over the hint bound", + hex::encode(key) + ); + assert!(retry.replica_hint_sources.is_subset(&retry.hint_sources)); + if let Some(owner) = queues.capacity_owner_by_key.get(key) { + assert!( + retry.hint_sources.contains(owner), + "{} retry entry lost its capacity owner", + hex::encode(key) + ); + } + } + } + + #[test] + fn readvertisements_keep_a_fetch_stage_retry_entry_within_the_hint_bound() { + let mut queues = ReplicationQueues::new(); + let (owner, holder) = (peer(1), peer(2)); + let key = key_at(0); + // The owner advertised the key without claiming a replica, so only + // the capacity accounting keeps it in the set. + assert!(queues + .add_pending_verify(key, paid_only_entry(1)) + .admitted()); + assert!(queues.promote_pending_to_fetch(key, key, vec![holder])); + + // Queued: paid-only advertisers fill the set to the bound, then replica + // claimants displace every one of them except the owner. + for sender in 20..40 { + queues.add_pending_verify(key, paid_only_entry(sender)); + } + for sender in 40..60 { + queues.add_pending_verify(key, test_entry(sender)); + } + assert_retry_entries_bounded(&queues); + let retry_sizes = queues + .fetch_payloads + .get(&key) + .and_then(|payload| payload.retry_verification.as_ref()) + .map(|retry| (retry.hint_sources.len(), retry.replica_hint_sources.len())); + assert_eq!( + retry_sizes, + Some((MAX_HINT_SOURCES_PER_KEY, MAX_HINT_SOURCES_PER_KEY - 1)), + "a full set keeps the owner and fills the rest with replica claimants" + ); + + // In flight: the same bound. + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, holder)]); + for sender in 60..80 { + queues.add_pending_verify(key, test_entry(sender)); + } + assert_retry_entries_bounded(&queues); + + // Back in pending verification, the restored entry keeps the bound and + // the owner its reservation was charged to. + assert!(queues.requeue_fetch_for_verification(&key, Duration::ZERO)); + let pending = queues.get_pending(&key).map(|entry| { + ( + entry.hint_sources.len(), + entry.hint_sources.contains(&owner), + ) + }); + assert_eq!(pending, Some((MAX_HINT_SOURCES_PER_KEY, true))); + assert_eq!(queues.capacity_owner_by_key.get(&key), Some(&owner)); + } + + #[test] + fn source_lists_are_capped_at_the_limit_including_tried_sources() { + let mut queues = ReplicationQueues::new().with_fetch_source_limit(3); + let sources: Vec = (1..=6).map(peer).collect(); + let key = queue(&mut queues, 0, &sources); + assert_eq!(queues.fetch_payloads[&key].sources, sources[..3].to_vec()); + + // Park it with one tried source: payload b, c plus tried a is the limit. + let mut queues = ReplicationQueues::new().with_fetch_source_limit(3); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + queues.add_pending_verify(key, test_entry(4)); + assert_eq!(queues.fetch_payloads[&key].sources, vec![b, c]); + assert_queue_invariants(&queues); + + // A direct start with an oversized list is cut to the limit too. + let mut queues = ReplicationQueues::new().with_fetch_source_limit(2); + let direct = key_at(9); + queues.start_fetch(direct, sources[0], sources.clone()); + assert_eq!( + queues.in_flight_fetch[&direct].all_sources, + sources[..2].to_vec() + ); + + // A start from a source outside the list counts that source too. + let mut queues = ReplicationQueues::new().with_fetch_source_limit(3); + let outside = peer(9); + queues.start_fetch(direct, outside, sources[..4].to_vec()); + let entry = &queues.in_flight_fetch[&direct]; + assert_eq!(entry.all_sources, sources[..2].to_vec()); + let union: HashSet<&PeerId> = entry.all_sources.iter().chain(&entry.tried).collect(); + assert_eq!(union.len(), 3); + } + + #[test] + fn a_parked_key_that_loses_its_last_wake_source_returns_to_the_heap() { + let mut queues = ReplicationQueues::new(); + let (b, c) = (peer(2), peer(3)); + queue(&mut queues, 0, &[b]); + queue(&mut queues, 1, &[b]); + queue(&mut queues, 2, &[c]); + // Hinted by 9, so the retry metadata survives b leaving. + let key = queue_promoted(&mut queues, 10, 9, &[b]); + let started = queues.dispatch_fetches(4, CAP).started; + assert!(started.iter().all(|(started_key, _)| *started_key != key)); + assert!(queues.fetch_parked.contains_key(&key)); + + let orphaned = queues.remove_hint_source(&b); + + assert!(!orphaned.contains(&key)); + assert!(!queues.fetch_parked.contains_key(&key)); + assert!(heap_keys(&queues).contains(&key)); + assert_queue_invariants(&queues); + let dispatch = queues.dispatch_fetches(4, CAP); + assert!(dispatch + .sourceless + .iter() + .any(|candidate| candidate.key == key)); + } + + #[test] + fn an_orphaned_parked_key_leaves_every_structure() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + queues.remove_hint_source(&b); + assert!( + queues.fetch_parked.contains_key(&key), + "c can still wake it" + ); + + let orphaned = queues.remove_hint_source(&c); + + assert!(orphaned.contains(&key)); + assert!(!queues.fetch_payloads.contains_key(&key)); + assert!(!queues.fetch_parked.contains_key(&key)); + assert!(!queues.fetch_tried.contains_key(&key)); + assert!(!queues.contains_key(&key)); + assert_queue_invariants(&queues); + } + + #[test] + fn tried_sources_survive_dequeue_and_move_into_the_next_start() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + + let candidate = queues.dequeue_fetch().unwrap(); + assert_eq!( + candidate.key, key, + "the parked key is the nearest queued key" + ); + assert!(queues.fetch_tried.contains_key(&key)); + queues.start_dequeued_fetch(candidate, c); + + assert_eq!(queues.in_flight_fetch[&key].tried, HashSet::from([a, c])); + assert!(!queues.fetch_tried.contains_key(&key)); + assert_queue_invariants(&queues); + } + + #[test] + fn tried_sources_are_dropped_with_a_discarded_or_requeued_candidate() { + for requeue in [false, true] { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + let candidate = queues.dequeue_fetch().unwrap(); + assert_eq!(candidate.key, key); + if requeue { + queues.requeue_candidate_for_verification(candidate, Duration::from_secs(1)); + } else { + queues.discard_fetch_candidate(candidate); + } + assert!(!queues.fetch_tried.contains_key(&key)); + assert_queue_invariants(&queues); + } + } + + #[test] + fn tried_sources_are_kept_by_a_refused_enqueue_and_cleared_by_a_fresh_one() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = park_after_trying_a(&mut queues, a, b, c); + assert!(!queues.enqueue_fetch(key, key, vec![a]), "already queued"); + assert_eq!(queues.fetch_tried[&key], HashSet::from([a])); + + let stray = key_at(77); + queues.fetch_tried.insert(stray, HashSet::from([a])); + assert!(queues.enqueue_fetch(stray, stray, vec![a])); + assert!(!queues.fetch_tried.contains_key(&stray)); + } + + #[test] + fn a_duplicate_of_an_in_flight_key_is_dropped_with_its_tried_sources() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let key = key_at(3); + queues.start_fetch(key, a, vec![a]); + // Force the impossible state the defensive skip exists for. + queues.fetch_payloads.insert( + key, + FetchPayload { + sources: vec![a], + retry_verification: None, + }, + ); + queues.fetch_queue.push(FetchOrder { key, distance: key }); + queues.fetch_tried.insert(key, HashSet::from([a])); + + assert!(queues.dequeue_fetch().is_none()); + assert!(!queues.fetch_tried.contains_key(&key)); + assert!(queues.in_flight_fetch.contains_key(&key)); + } + + #[test] + fn dequeue_returns_a_parked_key_nearer_than_the_heap_head() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + let parked = queue(&mut queues, 5, &[a]); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 2); + let farther = queue(&mut queues, 9, &[peer(2)]); + + assert_eq!(queues.dequeue_fetch().map(|c| c.key), Some(parked)); + assert_eq!(queues.dequeue_fetch().map(|c| c.key), Some(farther)); + } + + #[test] + fn a_parked_key_holds_bootstrap_drain_open() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + queue(&mut queues, 0, &[a]); + queue(&mut queues, 1, &[a]); + let parked = queue(&mut queues, 2, &[a]); + queues.dispatch_fetches(3, CAP); + assert!(queues.fetch_parked.contains_key(&parked)); + + assert!(!queues.is_bootstrap_work_empty(&HashSet::from([parked]))); + } + + // -- silent sources --------------------------------------------------------- + + /// A fetch request timeout for the table tests: a mark lasts twice this. + const TEST_TIMEOUT: Duration = Duration::from_secs(10); + + /// A peer distinct from every `peer(b)` and every other index. + fn distinct_peer(index: u32) -> PeerId { + let mut bytes = [0xA5_u8; 32]; + bytes[..4].copy_from_slice(&index.to_be_bytes()); + PeerId::from_bytes(bytes) + } + + /// `base` moved forward by `millis`. + fn at(base: Instant, millis: u64) -> Instant { + base + Duration::from_millis(millis) + } + + fn unreachable(sent: u64) -> Contact { + Contact::Unreachable { sent } + } + + fn answered(order: u64) -> Contact { + Contact::Answered { order } + } + + /// Mark `source` silent from now with a request at sequence place `sent`. + fn silence(queues: &mut ReplicationQueues, source: PeerId, sent: u64) { + queues.record_fetch_contact(source, unreachable(sent)); + } + + fn in_flight_entry(key: XorName, source: PeerId, started_at: Instant) -> InFlightEntry { + InFlightEntry { + key, + source, + started_at, + all_sources: vec![source], + tried: HashSet::from([source]), + retry_verification: None, + } + } + + /// The health table's mark index matches its entries exactly, and only a + /// source with an unanswered request since its latest answer is marked. + fn assert_health_invariants(table: &SourceHealthTable) { + assert!(table.entries.len() <= MAX_TRACKED_FETCH_SOURCES); + let marked: BTreeSet<(Instant, PeerId)> = table + .entries + .iter() + .filter_map(|(source, health)| health.marked_since.map(|since| (since, *source))) + .collect(); + assert_eq!(table.marks_by_since, marked, "mark index is not exact"); + for health in table.entries.values() { + assert!( + health.marked_since.is_none() || health.unanswered_since_answer(), + "a mark outlived the evidence for it" + ); + } + } + + #[test] + fn an_unanswered_request_marks_its_source_until_the_mark_expires() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + table.record(a, unreachable(1), at(t0, 100), &none); + assert!(table.is_silent(&a, at(t0, 100))); + assert!(table.is_silent(&a, at(t0, 20_099))); + assert!( + !table.is_silent(&a, at(t0, 20_100)), + "a mark lasts two fetch timeouts from when the failure was learned" + ); + + table.expire(at(t0, 20_100)); + assert_eq!(table.counters.expired_marks, 1); + assert!(table.marks_by_since.is_empty()); + + // An older unanswered request learned after the expiry is not new + // evidence: the source stays under the cap for its probe. + table.record(a, unreachable(0), at(t0, 20_200), &none); + assert!(!table.is_silent(&a, at(t0, 20_200))); + + // A request sent later that also goes unanswered marks it again. + table.record(a, unreachable(2), at(t0, 30_200), &none); + assert!(table.is_silent(&a, at(t0, 30_200))); + assert_eq!(table.counters.marks, 2); + assert_eq!(table.counters.unreachable, 3); + assert_health_invariants(&table); + } + + #[test] + fn a_source_that_keeps_failing_stays_silent_past_one_lifetime() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + for (sent, millis) in [(1, 0), (2, 15_000), (3, 30_000)] { + table.record(a, unreachable(sent), at(t0, millis), &none); + } + + assert!(table.is_silent(&a, at(t0, 49_999))); + assert!(!table.is_silent(&a, at(t0, 50_000))); + assert_eq!( + table.counters.marks, 1, + "one mark, refreshed by newer failures" + ); + assert_eq!(table.marks_by_since.len(), 1); + assert_health_invariants(&table); + } + + #[test] + fn a_failure_learned_late_still_marks_for_a_whole_lifetime() { + let none = HashMap::new(); + let a = peer(1); + let learned = at(Instant::now(), 600_000); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + table.record(a, unreachable(1), learned, &none); + + assert!(table.is_silent(&a, learned)); + assert!(table.is_silent(&a, at(learned, 19_999))); + assert!(!table.is_silent(&a, at(learned, 20_000))); + } + + /// Every order of applying `events` to a fresh table, all learned at + /// `now`, so only the sequence places can decide. + fn apply_in_every_order(events: &[Contact], now: Instant) -> Vec { + let none = HashMap::new(); + let a = peer(1); + let mut orders: Vec> = vec![Vec::new()]; + for _ in events { + orders = orders + .into_iter() + .flat_map(|prefix| { + (0..events.len()) + .filter(|index| !prefix.contains(index)) + .map(|index| { + let mut next = prefix.clone(); + next.push(index); + next + }) + .collect::>() + }) + .collect(); + } + orders + .iter() + .map(|order| { + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + for event in order.iter().filter_map(|index| events.get(*index)) { + table.record(a, *event, now, &none); + } + assert_health_invariants(&table); + table + }) + .collect() + } + + #[test] + fn whether_a_source_is_silent_does_not_depend_on_the_order_outcomes_arrive() { + let a = peer(1); + let now = Instant::now(); + let lifetime_ms = + u64::try_from((TEST_TIMEOUT * SILENCE_LIFETIME_TIMEOUTS).as_millis()).unwrap(); + + // Unanswered at 1, an answer at 2, unanswered again at 3: silent, as + // the request sent after the answer went unanswered, for one lifetime. + let silent = [unreachable(1), answered(2), unreachable(3)]; + let tables = apply_in_every_order(&silent, now); + assert_eq!(tables.len(), 6); + for table in &tables { + assert!(table.is_silent(&a, now)); + assert!(table.is_silent(&a, at(now, lifetime_ms - 1))); + assert!(!table.is_silent(&a, at(now, lifetime_ms))); + } + + // Unanswered at 1 and 2, then an answer at 3: answering. + let answering = [unreachable(1), unreachable(2), answered(3)]; + for table in &apply_in_every_order(&answering, now) { + assert!(!table.is_silent(&a, now)); + } + } + + #[test] + fn an_answer_older_than_the_unanswered_request_neither_clears_nor_blocks_its_mark() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(a, unreachable(5), t0, &none); + table.record(a, answered(4), t0, &none); + assert!(table.is_silent(&a, t0)); + assert_eq!(table.counters.clears, 0); + + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(a, answered(4), t0, &none); + table.record(a, unreachable(5), t0, &none); + assert!(table.is_silent(&a, t0)); + assert_eq!(table.counters.ignored_stale, 0); + + // A request sent before the latest answer is stale and cannot mark. + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(a, answered(6), t0, &none); + table.record(a, unreachable(5), t0, &none); + assert!(!table.is_silent(&a, t0)); + assert_eq!((table.counters.marks, table.counters.ignored_stale), (0, 1)); + assert_health_invariants(&table); + } + + #[test] + fn an_expired_mark_is_kept_only_while_its_source_has_a_fetch_in_flight() { + let t0 = Instant::now(); + let (guarded, idle) = (peer(1), peer(2)); + let key = key_at(0); + let in_flight = HashMap::from([(key, in_flight_entry(key, guarded, t0))]); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + table.record(guarded, unreachable(1), t0, &in_flight); + table.record(idle, unreachable(2), t0, &in_flight); + + let later = at(t0, 20_000); + let (counters, tracked) = table.take_counters(later, &in_flight); + + assert_eq!((counters.marks, counters.expired_marks, tracked), (2, 2, 2)); + assert!(table.entries.contains_key(&guarded)); + assert!(!table.entries.contains_key(&idle)); + // The guarded source keeps its history: its older request is no new + // evidence, its newer one marks it again. + table.record(guarded, unreachable(0), later, &in_flight); + assert!(!table.is_silent(&guarded, later)); + table.record(guarded, unreachable(3), later, &in_flight); + assert!(table.is_silent(&guarded, later)); + assert_health_invariants(&table); + } + + #[test] + fn attempts_that_sent_nothing_or_were_cancelled_change_nothing() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + + table.record(a, Contact::NotAttempted, t0, &none); + table.record(a, Contact::Unknown, t0, &none); + + assert!(table.entries.is_empty()); + assert!(!table.is_silent(&a, t0)); + assert_eq!(table.counters.unreachable, 0); + } + + #[test] + fn a_zero_fetch_timeout_still_leaves_silent_sources_a_lifetime() { + let none = HashMap::new(); + let a = peer(1); + let t0 = Instant::now(); + let mut table = SourceHealthTable::new(Duration::ZERO); + + table.record(a, unreachable(1), t0, &none); + + assert!(table.is_silent(&a, t0)); + assert!(table.is_silent(&a, at(t0, 999))); + assert!(!table.is_silent(&a, at(t0, 1_000))); + } + + /// A table at its bound: `guarded` answered and has a fetch in flight, + /// `idle` answered and has none, and every other entry is marked. + fn full_table( + t0: Instant, + marked: usize, + ) -> ( + SourceHealthTable, + HashMap, + PeerId, + PeerId, + ) { + let mut table = SourceHealthTable::new(TEST_TIMEOUT); + let guarded = distinct_peer(0); + let idle = distinct_peer(1); + let key = key_at(0); + let in_flight = HashMap::from([(key, in_flight_entry(key, guarded, t0))]); + table.record(guarded, answered(1), t0, &in_flight); + if marked < MAX_TRACKED_FETCH_SOURCES - 1 { + table.record(idle, answered(1), t0, &in_flight); + } + for index in 0..marked { + let source = distinct_peer(u32::try_from(index).unwrap() + 2); + table.record(source, unreachable(1), t0, &in_flight); + } + assert_eq!(table.entries.len(), MAX_TRACKED_FETCH_SOURCES); + (table, in_flight, guarded, idle) + } + + #[test] + fn a_full_table_makes_room_by_dropping_answers_that_guard_nothing() { + let t0 = Instant::now(); + let (mut table, in_flight, guarded, idle) = full_table(t0, MAX_TRACKED_FETCH_SOURCES - 2); + let newcomer = distinct_peer(u32::MAX); + + table.record(newcomer, unreachable(1), t0, &in_flight); + + assert!(table.is_silent(&newcomer, t0)); + assert!( + table.entries.contains_key(&guarded), + "an in-flight guard stays" + ); + assert!(!table.entries.contains_key(&idle)); + assert_eq!(table.counters.overflows, 0); + assert_health_invariants(&table); + } + + #[test] + fn a_table_full_of_marks_and_guards_counts_untracked_sources_as_silent() { + let t0 = Instant::now(); + let (mut table, in_flight, guarded, _) = full_table(t0, MAX_TRACKED_FETCH_SOURCES - 1); + let newcomer = distinct_peer(u32::MAX); + let bystander = distinct_peer(u32::MAX - 1); + + table.record(newcomer, unreachable(1), t0, &in_flight); + + assert_eq!(table.counters.overflows, 1); + assert!(!table.entries.contains_key(&newcomer)); + assert!( + table.is_silent(&newcomer, t0), + "an untracked source that stopped answering is fetched from as before the cap" + ); + assert!(table.is_silent(&bystander, t0)); + assert!( + table.entries.contains_key(&guarded), + "an in-flight guard is never dropped" + ); + assert!( + !table.is_silent(&guarded, t0), + "a tracked source keeps its own state" + ); + let lifetime = TEST_TIMEOUT * SILENCE_LIFETIME_TIMEOUTS; + assert!( + !table.is_silent(&bystander, t0 + lifetime), + "the overflow lasts one silence lifetime" + ); + assert_health_invariants(&table); + } + + #[test] + fn a_silent_source_is_fetched_from_past_the_cap() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + for d in 0..6 { + queue(&mut queues, d, &[a]); + } + assert_eq!(queues.dispatch_fetches(8, CAP).started.len(), CAP); + assert_eq!(queues.fetch_parked.len(), 4); + + silence(&mut queues, a, 1); + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!( + dispatch.started.len(), + 4, + "every parked key starts on the silent source" + ); + assert_eq!(in_flight_to(&queues, &a), 6); + assert!(queues.fetch_parked.is_empty()); + assert_eq!(queues.source_health.counters.silent_starts, 4); + assert_queue_invariants(&queues); + } + + #[test] + fn a_sole_silent_holder_drains_one_timeout_after_it_is_noticed() { + // Sixteen keys held only by `a`, eight fetch slots, and every request + // to `a` unanswered: one wave of two at the share, then the rest past + // it, eight per wave, as many waves as before the cap plus one. + let mut queues = ReplicationQueues::new(); + let a = peer(1); + for d in 0..16 { + queue(&mut queues, d, &[a]); + } + let mut sequence = 0; + let mut waves = Vec::new(); + loop { + let started = queues.dispatch_fetches(8, CAP).started; + if started.is_empty() { + break; + } + waves.push(started.len()); + // Every request in the wave times out. + for (key, source) in started { + sequence += 1; + queues.record_fetch_contact(source, unreachable(sequence)); + assert_eq!( + queues.retry_or_park_fetch(&key, key, CAP), + FetchRetry::Exhausted + ); + queues.complete_fetch(&key); + } + assert_queue_invariants(&queues); + } + assert_eq!(waves, vec![2, 8, 6]); + assert_eq!(queues.fetch_queue_count(), 0); + } + + #[test] + fn an_answering_source_with_room_is_preferred_over_a_silent_one() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + silence(&mut queues, a, 1); + let keys: Vec = (0..3).map(|d| queue(&mut queues, d, &[a, b])).collect(); + + let dispatch = queues.dispatch_fetches(8, CAP); + + let expected: Vec<(XorName, PeerId)> = keys.iter().copied().zip([b, b, a]).collect(); + assert_eq!( + dispatch.started, expected, + "b until its share is full, then the silent a rather than parking" + ); + assert!(queues.fetch_parked.is_empty()); + assert_queue_invariants(&queues); + } + + #[test] + fn retry_prefers_an_answering_source_with_room_then_a_silent_one() { + let (a, b, c) = (peer(1), peer(2), peer(3)); + for c_full in [false, true] { + let mut queues = ReplicationQueues::new(); + let key = queue(&mut queues, 0, &[a, b, c]); + if c_full { + queue(&mut queues, 10, &[c]); + queue(&mut queues, 11, &[c]); + } + let started = if c_full { 3 } else { 1 }; + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), started); + assert_eq!( + queues.in_flight_fetch.get(&key).map(|entry| entry.source), + Some(a) + ); + silence(&mut queues, b, 1); + + let retry = queues.retry_or_park_fetch(&key, key, CAP); + + let expected = if c_full { b } else { c }; + assert_eq!(retry, FetchRetry::Next(expected)); + assert_eq!( + queues.source_health.counters.silent_starts, + u64::from(c_full) + ); + assert_queue_invariants(&queues); + } + } + + #[test] + fn parked_keys_start_on_a_source_that_falls_silent_while_the_others_stay_full() { + let mut queues = ReplicationQueues::new(); + let (a, b) = (peer(1), peer(2)); + for d in 0..4 { + queue(&mut queues, d, &[if d < 2 { a } else { b }]); + } + assert_eq!(queues.dispatch_fetches(4, CAP).started.len(), 4); + let parked: Vec = (10..20).map(|d| queue(&mut queues, d, &[a, b])).collect(); + assert_eq!(queues.dispatch_fetches(8, CAP).started, Vec::new()); + assert_eq!(queues.fetch_parked.len(), 10); + + silence(&mut queues, b, 1); + let dispatch = queues.dispatch_fetches(8, CAP); + + // a stays full and answering: every slot goes to b, and the pass ends. + let expected: Vec<(XorName, PeerId)> = parked.iter().take(8).map(|key| (*key, b)).collect(); + assert_eq!(dispatch.started, expected); + assert_eq!(queues.fetch_parked.len(), 2); + assert_queue_invariants(&queues); + } + + #[test] + fn an_answer_puts_a_silent_source_back_under_the_cap() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + for d in 0..6 { + queue(&mut queues, d, &[a]); + } + silence(&mut queues, a, 1); + assert_eq!(queues.dispatch_fetches(3, CAP).started.len(), 3); + + queues.record_fetch_contact(a, answered(2)); + let dispatch = queues.dispatch_fetches(8, CAP); + + assert_eq!( + dispatch.started, + Vec::new(), + "a answers again and already has three fetches in flight" + ); + assert_eq!(queues.fetch_parked.len(), 3); + assert_queue_invariants(&queues); + } + + #[test] + fn a_departure_keeps_what_was_observed_of_the_source() { + let mut queues = ReplicationQueues::new(); + let a = peer(1); + let key = queue(&mut queues, 0, &[a]); + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, a)]); + silence(&mut queues, a, 1); + + queues.remove_hint_source(&a); + + assert!(queues.source_health.is_silent(&a, Instant::now())); + // The attempt still in flight answers after the departure: applied + // like any other outcome. + queues.record_fetch_contact(a, answered(2)); + assert!(!queues.source_health.is_silent(&a, Instant::now())); + // A late unanswered request sent before that answer cannot mark it. + queues.record_fetch_contact(a, unreachable(1)); + assert!(!queues.source_health.is_silent(&a, Instant::now())); + assert_health_invariants(&queues.source_health); + } + + #[test] + fn the_summary_drops_answers_that_guard_nothing_and_keeps_marks() { + let mut queues = ReplicationQueues::new(); + let (a, b, c) = (peer(1), peer(2), peer(3)); + let key = queue(&mut queues, 0, &[a]); + assert_eq!(queues.dispatch_fetches(1, CAP).started, vec![(key, a)]); + queues.record_fetch_contact(a, answered(1)); + queues.record_fetch_contact(b, answered(2)); + silence(&mut queues, c, 3); + + let stats = queues.take_fetch_dispatch_stats(CAP); + + assert_eq!(stats.tracked_sources, 3); + assert_eq!(stats.silent_sources, 1); + assert!(!stats.overflow_active); + assert_eq!(stats.health.marks, 1); + assert_eq!(stats.health.silent_peak, 1); + let kept: HashSet = queues.source_health.entries.keys().copied().collect(); + assert_eq!( + kept, + HashSet::from([a, c]), + "a guards a fetch in flight and c is marked" + ); + assert_eq!(queues.source_health.counters.marks, 0, "counters restart"); + assert_eq!( + queues.source_health.counters.silent_peak, 1, + "the peak restarts at the sources silent now" + ); + assert_health_invariants(&queues.source_health); + } + + #[test] + #[allow(clippy::too_many_lines)] + fn queue_invariants_hold_under_random_operations() { + let mut rng = StdRng::seed_from_u64(0x1514); + let mut queues = ReplicationQueues::new().with_fetch_source_limit(5); + let sources: Vec = (1..=6).map(peer).collect(); + let mut sequence = 0_u64; + for step in 0..4_000_u32 { + match rng.gen_range(0..10) { + 0 | 1 => { + let count = rng.gen_range(1..=4); + let mut chosen: Vec = Vec::new(); + while chosen.len() < count { + let source = sources[rng.gen_range(0..sources.len())]; + if !chosen.contains(&source) { + chosen.push(source); + } + } + let key = key_at(rng.gen_range(0..400)); + if rng.gen_bool(0.25) { + // Through verification, so it carries a retry entry. + let hinter = rng.gen_range(1..=30); + if queues + .add_pending_verify(key, test_entry(hinter)) + .admitted() + { + queues.promote_pending_to_fetch(key, key, chosen); + } + } else { + queues.enqueue_fetch(key, key, chosen); + } + } + 2 => { + // As the fetch worker does, a candidate with no untried + // source goes back to verification. + let dispatch = queues.dispatch_fetches(rng.gen_range(0..6), CAP); + for candidate in dispatch.sourceless { + queues.requeue_candidate_for_verification(candidate, Duration::ZERO); + } + } + 3 => { + if let Some(key) = queues.in_flight_fetch.keys().min().copied() { + queues.complete_fetch(&key); + } + } + 4 => { + let mut in_flight: Vec = + queues.in_flight_fetch.keys().copied().collect(); + in_flight.sort_unstable(); + if !in_flight.is_empty() { + let key = in_flight[rng.gen_range(0..in_flight.len())]; + if queues.retry_or_park_fetch(&key, key, CAP) == FetchRetry::Exhausted { + queues.complete_fetch(&key); + } + } + } + 5 => { + let mut waiting: Vec = queues.fetch_payloads.keys().copied().collect(); + waiting.sort_unstable(); + if !waiting.is_empty() { + let key = waiting[rng.gen_range(0..waiting.len())]; + let hinter = rng.gen_range(1..=30); + let entry = if rng.gen_bool(0.5) { + test_entry(hinter) + } else { + paid_only_entry(hinter) + }; + queues.add_pending_verify(key, entry); + } + } + 6 => { + if rng.gen_range(0..10) == 0 { + queues.remove_hint_source(&sources[rng.gen_range(0..sources.len())]); + } + } + 7 => { + if let Some(candidate) = queues.dequeue_fetch() { + match candidate.sources.first().copied() { + Some(source) if rng.gen_bool(0.5) => { + queues.start_dequeued_fetch(candidate, source); + } + _ => queues.discard_fetch_candidate(candidate), + } + } + } + _ => { + // A source answers, or a request to it goes unanswered, + // applied a little out of sequence order. + if let Some(source) = sources.get(rng.gen_range(0..sources.len())).copied() { + sequence += 1; + let place = sequence.saturating_sub(rng.gen_range(0..4)); + let contact = if rng.gen_bool(0.5) { + Contact::Answered { order: place } + } else { + Contact::Unreachable { sent: place } + }; + queues.record_fetch_contact(source, contact); + } + } + } + assert_queue_invariants(&queues); + assert_health_invariants(&queues.source_health); + assert_retry_entries_bounded(&queues); + for (key, entry) in &queues.in_flight_fetch { + let associated: HashSet<&PeerId> = + entry.all_sources.iter().chain(entry.tried.iter()).collect(); + assert!( + associated.len() <= 5, + "step {step}: {} over the source limit", + hex::encode(key) + ); + } + for (key, payload) in &queues.fetch_payloads { + let tried = queues.fetch_tried.get(key).map_or(0, HashSet::len); + assert!( + payload.sources.len() + tried <= 5, + "step {step}: queued over the limit" + ); + } + } + } + + /// Scale benchmark for the per-source scheduler at the fetch queue's hard + /// bound. Runs only when `ANT_FETCH_SCALE_BENCH` names a sources-per-key + /// count (it allocates hundreds of MiB and takes seconds); otherwise it + /// returns at once. One count per process, so `/usr/bin/time -l` reports + /// that scenario's peak memory: + /// + /// `ANT_FETCH_SCALE_BENCH=8 cargo test --release --features test-utils --lib + /// scale_benchmark -- --nocapture` + /// + /// Thresholds, declared before the release runs: one pass parking every key + /// holds the queue for at most 1 s at up to 8 sources per key and at most + /// 5 s at the 35-source limit; waking, the stats read and a departure each + /// stay under 250 ms; the process's peak RSS is at most 1 GiB at 8 sources + /// per key and 3 GiB at 35. Dropping the queues was first held to 250 ms + /// too, but measured 274 to 315 ms at 35 sources per key, and 226 to 295 ms + /// with the scheduler before the silent-source handling. It happens once, + /// when the engine shuts down, outside any lock, so it is held to 1 s. + #[test] + fn scale_benchmark_at_the_fetch_queue_bound() { + let Some(sources_per_key) = std::env::var("ANT_FETCH_SCALE_BENCH") + .ok() + .and_then(|raw| raw.parse::().ok()) + else { + return; + }; + let park_limit = if sources_per_key <= 8 { + Duration::from_secs(1) + } else { + Duration::from_secs(5) + }; + { + let other_limit = Duration::from_millis(250); + let teardown_limit = Duration::from_secs(1); + let sources: Vec = (0..sources_per_key) + .map(|index| { + let mut bytes = [0u8; 32]; + bytes[..8].copy_from_slice(&(index as u64 + 1).to_be_bytes()); + PeerId::from_bytes(bytes) + }) + .collect(); + let mut queues = ReplicationQueues::new(); + // Saturate every source with this node's own fetches first. + let mut blockers = Vec::new(); + for (index, source) in sources.iter().enumerate() { + for offset in 0..CAP { + let key = key_at(u32::MAX - u32::try_from(index * CAP + offset).unwrap()); + queues.start_fetch(key, *source, vec![*source]); + blockers.push(key); + } + } + let mut distance = 0u32; + while queues.fetch_queue_count() < MAX_FETCH_QUEUE { + let key = key_at(distance); + assert!(queues.enqueue_fetch(key, key, sources.clone())); + distance += 1; + } + + let started = Instant::now(); + let dispatch = queues.dispatch_fetches(8, CAP); + let park_pass = started.elapsed(); + assert_eq!(dispatch.started, Vec::new()); + assert_eq!(queues.fetch_parked.len(), MAX_FETCH_QUEUE); + + let associations: usize = queues.parked_by_source.values().map(BTreeSet::len).sum(); + + let started = Instant::now(); + let stats = queues.take_fetch_dispatch_stats(CAP); + let stats_pass = started.elapsed(); + assert_eq!(stats.parked, MAX_FETCH_QUEUE); + + let started = Instant::now(); + queues.complete_fetch(&blockers[0]); + let wake = queues.dispatch_fetches(8, CAP); + let wake_pass = started.elapsed(); + assert_eq!(wake.started.len(), 1); + + let started = Instant::now(); + queues.remove_hint_source(&sources[0]); + let departure = started.elapsed(); + + let started = Instant::now(); + drop(queues); + let teardown = started.elapsed(); + + println!( + "FETCH-SCALE sources_per_key={sources_per_key} keys={MAX_FETCH_QUEUE} \ + associations={associations} park_pass_ms={} stats_ms={} wake_ms={} \ + departure_ms={} teardown_ms={}", + park_pass.as_millis(), + stats_pass.as_millis(), + wake_pass.as_millis(), + departure.as_millis(), + teardown.as_millis() + ); + assert!( + park_pass <= park_limit, + "park pass {park_pass:?} over {park_limit:?}" + ); + for (name, took) in [ + ("stats", stats_pass), + ("wake", wake_pass), + ("departure", departure), + ] { + assert!( + took <= other_limit, + "{name} took {took:?}, over {other_limit:?}" + ); + } + assert!( + teardown <= teardown_limit, + "teardown took {teardown:?}, over {teardown_limit:?}" + ); + } + } } diff --git a/tests/e2e/fetch_per_source_share.rs b/tests/e2e/fetch_per_source_share.rs new file mode 100644 index 00000000..7b9b87ef --- /dev/null +++ b/tests/e2e/fetch_per_source_share.rs @@ -0,0 +1,328 @@ +//! A fetching node keeps no more of its fetches in flight to one answering +//! holder than that holder will admit from it. +//! +//! A fetch responder admits two outstanding requests per requesting peer and +//! answers a third with nothing at all, so a requester that sends more than two +//! at once to one holder loses the rest to its own fetch timeout. The fetch +//! worker used to do exactly that: it ran one fetch per slot and sent every one +//! to the first verified holder of its key, so a node whose queued keys shared +//! one holder had all of them outstanding there at once. +//! +//! The observation is made on the requester, which is what the scheduler +//! governs, and made deterministic by holding the holder's work in place: the +//! holder's test-only serve gate holds every fetch request it admits before +//! serving it, so whatever the requester sends stays outstanding for as long as +//! the test looks at it, however fast the host serves a chunk. Every node runs +//! eight fetch slots, more than one holder's share on any host. With the +//! per-source limit removed the requester holds all eight in flight to the one +//! holder and this test fails. Both hooks are per engine, so no other node or +//! test is affected. +//! +//! The limit is a share of a holder that answers. A holder whose requests go +//! unanswered has no share to keep within, and keeping its keys at two per +//! fetch timeout would drain them four times more slowly than before the limit. +//! The second test closes the holder's gate for good, with a short fetch +//! timeout, and checks that once the first requests time out the requester +//! sends past the share and the keys leave its fetch pipeline in a bounded time. +//! +//! Both tests tear the network down with the holder's gate still closed and +//! requests held at it, which the holder's engine shutdown must release. + +#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] + +use super::testnet::TestNetworkConfig; +use super::verify_storage_gate::find_key_for_target; +use super::TestHarness; +use ant_node::client::XorName; +use ant_node::replication::config::storage_admission_width; +use ant_node::{ReplicationConfig, ReplicationEngine}; +use saorsa_core::identity::PeerId; +use serial_test::serial; +use std::collections::HashSet; +use std::time::{Duration, Instant}; + +/// Above `storage_admission_width` (9), so the network has a real close group. +const NODE_COUNT: usize = 12; +/// Production close-group size. +const CLOSE_GROUP_SIZE: usize = 7; +/// Node whose fetch worker is observed. +const REQUESTER_INDEX: usize = 5; +/// The one node hosting every chunk. +const HOLDER_INDEX: usize = 6; +/// More keys than fetch slots, all held by the one holder. +const KEY_COUNT: usize = 12; +/// Fetch slots per node, above the per-source share on every host. +const FETCH_CONCURRENCY: usize = 8; +/// The responder's per-peer fetch share. +const PER_SOURCE_SHARE: usize = 2; +/// How long the requester's fetch pipeline must stay empty before the test +/// starts, so earlier traffic cannot occupy its slots. +const QUIET_FOR: Duration = Duration::from_secs(1); +/// Upper bound on waiting for that quiet period. +const QUIET_DEADLINE: Duration = Duration::from_secs(30); +/// How long the requester is observed with the gate closed. Far below the 30 s +/// fetch timeout, so no held request times out and frees its slot early. +const OBSERVATION_WINDOW: Duration = Duration::from_secs(3); +const OBSERVATION_POLL: Duration = Duration::from_millis(50); +/// The silent-holder test's fetch timeout, short so silence is seen in seconds. +const SILENT_FETCH_TIMEOUT: Duration = Duration::from_secs(5); +/// Keys held only by the silent holder: two fetch slots' worth. +const SILENT_KEY_COUNT: usize = 16; +/// The longest the silent holder's keys may take to leave the requester's +/// fetch pipeline: four fetch timeouts. This scheduler takes three, one more +/// than fetching as before the limit, to see the silence; keeping the holder +/// at its share would take eight (sixteen keys, two per timeout). The exact +/// three waves are checked without timing in the scheduler's unit tests. +const SILENT_DRAIN_LIMIT: Duration = Duration::from_secs(20); +/// The longest tearing the network down may take with requests still held at +/// the holder's closed serve gate. Without the release at engine shutdown it +/// waits for them for ever. +const TEARDOWN_LIMIT: Duration = Duration::from_secs(120); + +#[tokio::test] +#[serial] +async fn requester_keeps_at_most_its_share_in_flight_to_one_holder() { + let max_in_flight = max_in_flight_to_one_holder().await; + assert!( + max_in_flight <= PER_SOURCE_SHARE, + "the requester had {max_in_flight} fetches outstanding at one holder, which \ + admits {PER_SOURCE_SHARE} from it and answers the rest with nothing" + ); + assert_eq!( + max_in_flight, PER_SOURCE_SHARE, + "the requester never filled the holder's share, so the run says nothing \ + about the limit: the worker did not dispatch" + ); +} + +#[tokio::test] +#[serial] +async fn a_holder_that_stops_answering_is_fetched_from_past_its_share() { + let harness = start_network(SILENT_FETCH_TIMEOUT).await; + let chunks = chunks_for_requester(&harness, "silent-holder", SILENT_KEY_COUNT).await; + let keys: HashSet = chunks.iter().map(|(_, key)| *key).collect(); + + let requester = harness.test_node(REQUESTER_INDEX).expect("requester"); + let requester_engine = requester + .replication_engine + .as_ref() + .expect("requester engine"); + let holder = harness.test_node(HOLDER_INDEX).expect("holder"); + let holder_peer = *holder.p2p_node.as_ref().expect("holder p2p").peer_id(); + let holder_engine = holder.replication_engine.as_ref().expect("holder engine"); + + wait_until_fetch_pipeline_is_quiet(requester_engine).await; + + // The holder answers none of these fetches, ever. + holder_engine.close_fetch_serve_gate_for_test(); + let started = Instant::now(); + for key in &keys { + assert!( + requester_engine + .enqueue_fetch_for_test(*key, vec![holder_peer]) + .await, + "every key must be queued for the silent holder alone" + ); + } + + let mut max_in_flight_to_holder = 0; + let drained_after = loop { + let snapshot = requester_engine.fetch_pipeline_snapshot_for_test().await; + let in_flight_to_holder = snapshot + .in_flight + .iter() + .filter(|(key, source)| keys.contains(key) && *source == holder_peer) + .count(); + max_in_flight_to_holder = max_in_flight_to_holder.max(in_flight_to_holder); + let pending = snapshot + .queued + .iter() + .chain(snapshot.in_flight.iter().map(|(key, _)| key)) + .filter(|key| keys.contains(*key)) + .count(); + if pending == 0 { + break started.elapsed(); + } + assert!( + started.elapsed() < SILENT_DRAIN_LIMIT, + "{pending} of {SILENT_KEY_COUNT} keys held only by a silent holder were still \ + in the fetch pipeline after {SILENT_DRAIN_LIMIT:?}" + ); + tokio::time::sleep(OBSERVATION_POLL).await; + }; + println!( + "SILENT-HOLDER-RESULT drained_after={drained_after:?} \ + max_in_flight_to_holder={max_in_flight_to_holder} timeout={SILENT_FETCH_TIMEOUT:?}" + ); + + teardown_with_the_gate_closed(harness).await; + assert!( + drained_after < SILENT_DRAIN_LIMIT, + "the silent holder's keys took {drained_after:?} to leave the fetch pipeline" + ); + assert!( + max_in_flight_to_holder > PER_SOURCE_SHARE, + "the requester kept a holder that answers nothing at its share of \ + {PER_SOURCE_SHARE}, which drains that holder's keys at two per timeout" + ); +} + +/// Build the network with `fetch_request_timeout` and warm its DHT. +async fn start_network(fetch_request_timeout: Duration) -> TestHarness { + let config = TestNetworkConfig { + node_count: NODE_COUNT, + replication_config: Some(ReplicationConfig { + close_group_size: CLOSE_GROUP_SIZE, + fetch_request_timeout, + ..ReplicationConfig::default() + }), + fetch_concurrency: Some(FETCH_CONCURRENCY), + ..TestNetworkConfig::default() + }; + let harness = TestHarness::setup_with_config(config) + .await + .expect("setup per-source share network"); + harness.warmup_dht().await.expect("warmup"); + harness +} + +/// `count` distinct chunks the requester is responsible for storing. +async fn chunks_for_requester( + harness: &TestHarness, + label: &str, + count: usize, +) -> Vec<(Vec, XorName)> { + let width = storage_admission_width(CLOSE_GROUP_SIZE); + let mut chunks = Vec::with_capacity(count); + for index in 0..count { + let label = format!("{label}-{index}"); + chunks.push(find_key_for_target(harness, REQUESTER_INDEX, width, true, &label).await); + } + let keys: HashSet<&XorName> = chunks.iter().map(|(_, key)| key).collect(); + assert_eq!(keys.len(), count, "every key must be distinct"); + chunks +} + +/// Build the network, direct twelve keys at one holder while that holder holds +/// what it admits, and return the most fetches the requester had in flight to +/// it. +async fn max_in_flight_to_one_holder() -> usize { + let harness = start_network(ReplicationConfig::default().fetch_request_timeout).await; + let chunks = chunks_for_requester(&harness, "per-source-share", KEY_COUNT).await; + let keys: HashSet = chunks.iter().map(|(_, key)| *key).collect(); + + let requester = harness.test_node(REQUESTER_INDEX).expect("requester"); + let requester_engine = requester + .replication_engine + .as_ref() + .expect("requester engine"); + let holder = harness.test_node(HOLDER_INDEX).expect("holder"); + let holder_peer = *holder.p2p_node.as_ref().expect("holder p2p").peer_id(); + let holder_engine = holder.replication_engine.as_ref().expect("holder engine"); + let holder_storage = holder + .ant_protocol + .as_ref() + .expect("holder protocol") + .storage(); + + // Start from an idle requester, so its fetch slots are all free. + wait_until_fetch_pipeline_is_quiet(requester_engine).await; + + // Hold every fetch the holder admits, then make the chunks available. + holder_engine.close_fetch_serve_gate_for_test(); + for (content, key) in &chunks { + holder_storage.put(key, content).await.expect("host chunk"); + } + let before = requester_engine.fetch_pipeline_snapshot_for_test().await; + let unrelated: Vec = before + .queued + .iter() + .chain(before.in_flight.iter().map(|(key, _)| key)) + .filter(|key| !keys.contains(*key)) + .copied() + .collect(); + assert!( + unrelated.is_empty(), + "unrelated fetch work could occupy the requester's slots: {unrelated:?}" + ); + for key in &keys { + // A replica hint may already have queued the key; either way it is + // headed to the one holder. + let _ = requester_engine + .enqueue_fetch_for_test(*key, vec![holder_peer]) + .await; + } + + let max_in_flight_to_holder = + observe_max_in_flight_to(requester_engine, &keys, &holder_peer).await; + println!( + "PER-SOURCE-SHARE-RESULT max_in_flight_to_holder={max_in_flight_to_holder} \ + concurrency={FETCH_CONCURRENCY} keys={KEY_COUNT}" + ); + + teardown_with_the_gate_closed(harness).await; + max_in_flight_to_holder +} + +/// Tear the network down while the holder's serve gate is still closed. +async fn teardown_with_the_gate_closed(harness: TestHarness) { + let torn_down = tokio::time::timeout(TEARDOWN_LIMIT, harness.teardown()).await; + assert!( + matches!(torn_down, Ok(Ok(()))), + "teardown with requests held at a closed serve gate did not finish within \ + {TEARDOWN_LIMIT:?}: {torn_down:?}" + ); +} + +/// Wait until `engine` has nothing queued or in flight for fetch, continuously +/// for [`QUIET_FOR`]. +async fn wait_until_fetch_pipeline_is_quiet(engine: &ReplicationEngine) { + let deadline = tokio::time::Instant::now() + QUIET_DEADLINE; + let mut quiet_since = tokio::time::Instant::now(); + loop { + let snapshot = engine.fetch_pipeline_snapshot_for_test().await; + let now = tokio::time::Instant::now(); + if !snapshot.queued.is_empty() || !snapshot.in_flight.is_empty() { + quiet_since = now; + } else if now.duration_since(quiet_since) >= QUIET_FOR { + return; + } + assert!( + now < deadline, + "the requester's fetch pipeline never went quiet: {snapshot:?}" + ); + tokio::time::sleep(OBSERVATION_POLL).await; + } +} + +/// The most of `keys` that `engine` had in flight to `holder` at once over +/// [`OBSERVATION_WINDOW`], failing if any other fetch work appears. +async fn observe_max_in_flight_to( + engine: &ReplicationEngine, + keys: &HashSet, + holder: &PeerId, +) -> usize { + let deadline = tokio::time::Instant::now() + OBSERVATION_WINDOW; + let mut max_in_flight = 0; + while tokio::time::Instant::now() < deadline { + let snapshot = engine.fetch_pipeline_snapshot_for_test().await; + let unrelated: Vec<&XorName> = snapshot + .queued + .iter() + .chain(snapshot.in_flight.iter().map(|(key, _)| key)) + .filter(|key| !keys.contains(*key)) + .collect(); + assert!( + unrelated.is_empty(), + "unrelated fetch work appeared during the observation: {unrelated:?}" + ); + let to_holder = snapshot + .in_flight + .iter() + .filter(|(key, source)| keys.contains(key) && source == holder) + .count(); + max_in_flight = max_in_flight.max(to_holder); + tokio::time::sleep(OBSERVATION_POLL).await; + } + max_in_flight +} diff --git a/tests/e2e/mod.rs b/tests/e2e/mod.rs index ec2a74c7..295b022d 100644 --- a/tests/e2e/mod.rs +++ b/tests/e2e/mod.rs @@ -75,6 +75,9 @@ mod fetch_local_write_guard; #[cfg(test)] mod fetch_responsibility_recheck; +#[cfg(test)] +mod fetch_per_source_share; + #[cfg(test)] mod first_audit_ab; diff --git a/tests/e2e/testnet.rs b/tests/e2e/testnet.rs index ca33a901..051e42b4 100644 --- a/tests/e2e/testnet.rs +++ b/tests/e2e/testnet.rs @@ -230,6 +230,10 @@ pub struct TestNetworkConfig { /// use this to shorten timers — e.g. the ADR-0003 possession-check delay. pub replication_config: Option, + /// Fetches each node's worker runs at once. `None` uses one per hardware + /// thread. + pub fetch_concurrency: Option, + /// Optional per-node storage disk-reserve overrides. /// /// Tests can set a node's reserve above available disk space to make its @@ -283,6 +287,7 @@ impl Default for TestNetworkConfig { payment_enforcement: false, evm_network: None, replication_config: None, + fetch_concurrency: None, storage_disk_reserve_overrides: HashMap::new(), } } @@ -1388,6 +1393,9 @@ impl TestNetwork { .await { Ok(mut engine) => { + if let Some(concurrency) = self.config.fetch_concurrency { + engine.set_fetch_concurrency_for_test(concurrency); + } // Pointers replicate through the same engine (ADR-0016), // which the service also asks before a final state // (ADR-0018).