diff --git a/.depot/actions/setup/action.yml b/.depot/actions/setup/action.yml index defe94e6e0..3c426c6eb0 100644 --- a/.depot/actions/setup/action.yml +++ b/.depot/actions/setup/action.yml @@ -19,7 +19,7 @@ inputs: default: "false" sp1-version: description: SP1 release tag used when sp1 is enabled (matches etc/just/succinct.just) - default: "v6.8.1" + default: "v6.10.0" native-deps: description: Install native build dependencies (clang, llvm, etc.) default: "false" @@ -98,8 +98,8 @@ runs: shell: bash env: GITHUB_TOKEN: ${{ github.token }} - # Commit for succinctlabs/sp1 v6.8.1; update alongside SP1_VERSION. - SP1UP_REF: c84ada1ed5911f28c4d3c9d0ed2f9e6cd7edb824 + # Commit for succinctlabs/sp1 v6.10.0; update alongside SP1_VERSION. + SP1UP_REF: d83ed91f96701209613720708f36ffce0e98ed33 SP1_VERSION: ${{ inputs.sp1-version }} run: | if [ -x "$HOME/.sp1/bin/cargo-prove" ]; then diff --git a/.github/actions/setup/action.yml b/.github/actions/setup/action.yml index 984df7d219..f5ae616606 100644 --- a/.github/actions/setup/action.yml +++ b/.github/actions/setup/action.yml @@ -19,7 +19,7 @@ inputs: default: "false" sp1-version: description: SP1 release tag used when sp1 is enabled (matches etc/just/succinct.just) - default: "v6.8.1" + default: "v6.10.0" native-deps: description: Install native build dependencies (clang, llvm, etc.) default: "false" @@ -161,8 +161,8 @@ runs: shell: bash env: GITHUB_TOKEN: ${{ github.token }} - # Commit for succinctlabs/sp1 v6.8.1; update alongside SP1_VERSION. - SP1UP_REF: c84ada1ed5911f28c4d3c9d0ed2f9e6cd7edb824 + # Commit for succinctlabs/sp1 v6.10.0; update alongside SP1_VERSION. + SP1UP_REF: d83ed91f96701209613720708f36ffce0e98ed33 SP1_VERSION: ${{ inputs.sp1-version }} run: | if [ -x "$HOME/.sp1/bin/cargo-prove" ]; then diff --git a/ACTIVATIONS.md b/ACTIVATIONS.md index efa57776e1..8f548fa19b 100644 --- a/ACTIVATIONS.md +++ b/ACTIVATIONS.md @@ -7,7 +7,9 @@ and refresh the statuses when an activation is announced. Denim and Everest are contract-backed: an L1 upgrade signal (`crates/utilities/upgrade-signal`) can schedule them on a network with no new node release. Zenith can only be set in genesis. -Statuses checked 2026-10-28 against Base `v1.5.1` (`crates/common/chains/src/config.rs`). +Statuses checked 2026-11-06 against Base `v1.5.4` (`crates/common/chains/src/config.rs`). On both +L1s, `getSchedule()` on the configured `protocol_versions_address` reverts, so no upgrade is +scheduled through the signal; the schedules below are the ones in the code. | Upgrade | Base mainnet | Base Sepolia | Effect on the Firehose block | |---------|--------------|--------------|------------------------------| @@ -15,7 +17,7 @@ Statuses checked 2026-10-28 against Base `v1.5.1` (`crates/common/chains/src/con | Azul | Active 2026-05-28 18:00 UTC | Active 2026-04-20 18:00 UTC | Already in the stream. | | Beryl | Active 2026-06-25 18:00 UTC | Active 2026-06-18 18:00 UTC | Already in the stream. | | Cobalt | Active 2026-09-30 18:00 UTC | Active 2026-09-23 18:00 UTC | B-20 V2 precompiles. Covered by `etc/systems/tests/firehose_cobalt_b20.rs`. | -| Denim | Scheduled 2026-11-05 18:00 UTC (`1_793_901_600`) from `v1.5.1`; first Denim block is 53_556_127 at `1_793_901_601` | Active 2026-10-21 18:00 UTC (`1_792_605_600`) | See [Denim](#denim). **Mainnet nodes must run `v1.5.1-fh3.1` or later before activation.** | +| Denim | Active 2026-11-05 18:00 UTC (`1_793_901_600`); first Denim block is 53_556_127 at `1_793_901_601` | Active 2026-10-21 18:00 UTC (`1_792_605_600`) | See [Denim](#denim). Mainnet nodes need `v1.5.1-fh3.1` or later. | | Everest | Not scheduled | Not scheduled | See [Everest](#everest). Traced from `v1.5.0-fh3.1-2` as `TRX_TYPE_ACCOUNT_ABSTRACTION`. **Earlier versions stop on the first EIP-8130 transaction.** | | Zenith | Never (genesis only) | Never (genesis only) | Gates the EIP-8130 keystore (account configuration changes, non-k1 authenticators). Unreachable on real networks. Traced from `v1.5.0-fh3.1-2`: create and config change entries in `account_changes`, non-k1 sender authentication in `sender_auth`. | diff --git a/CHANGELOG.sf.md b/CHANGELOG.sf.md index 2d5a15ed98..83ac0228d5 100644 --- a/CHANGELOG.sf.md +++ b/CHANGELOG.sf.md @@ -1,3 +1,16 @@ +## v1.5.4-fh3.1 + +### Changed + +* Rebased on upstream Base `v1.5.4` (with `v1.5.2` and `v1.5.3`), recommended for synchronization + and block propagation at 200 ms block times. Uses `streamingfast/reth` `base-v2.5.2.11-fh3.1`; + `firehose-tracer` stays on 5.7.0 and the Firehose block model is unchanged. +* Upstream raises the engine persistence defaults of `base-reth-node` to 70 blocks + (`--engine.persistence-threshold`, `--engine.memory-block-buffer-target`) and 160 blocks + (`--engine.persistence-backpressure-threshold`). After an unclean shutdown the node executes + the blocks it held in memory again and emits their FIRE BLOCK lines again: up to about 14 s of + chain at 200 ms blocks, instead of about 1 s. + ## v1.5.1-fh3.1 ### Changed diff --git a/Cargo.lock b/Cargo.lock index 8bbe7310b5..2d1024310a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -658,7 +658,7 @@ dependencies = [ "either", "futures", "futures-utils-wasm", - "lru 0.18.2", + "lru 0.18.5", "parking_lot", "pin-project", "reqwest 0.13.4", @@ -2046,7 +2046,7 @@ dependencies = [ [[package]] name = "audit-archiver" -version = "1.5.1" +version = "1.5.4" dependencies = [ "anyhow", "audit-archiver-lib", @@ -2063,7 +2063,7 @@ dependencies = [ [[package]] name = "audit-archiver-lib" -version = "1.5.1" +version = "1.5.4" dependencies = [ "anyhow", "async-trait", @@ -2720,40 +2720,6 @@ dependencies = [ "tower-service", ] -[[package]] -name = "axum" -version = "0.7.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "edca88bc138befd0323b20752846e6587272d3b03b0343c8ea28a6f819e6e71f" -dependencies = [ - "async-trait", - "axum-core 0.4.5", - "bytes", - "futures-util", - "http 1.4.2", - "http-body 1.1.0", - "http-body-util", - "hyper 1.11.0", - "hyper-util", - "itoa", - "matchit 0.7.3", - "memchr", - "mime", - "percent-encoding", - "pin-project-lite", - "rustversion", - "serde", - "serde_json", - "serde_path_to_error", - "serde_urlencoded", - "sync_wrapper 1.0.2", - "tokio", - "tower 0.5.3", - "tower-layer", - "tower-service", - "tracing", -] - [[package]] name = "axum" version = "0.8.9" @@ -2806,27 +2772,6 @@ dependencies = [ "tower-service", ] -[[package]] -name = "axum-core" -version = "0.4.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09f2bd6146b97ae3359fa0cc6d6b376d9539582c7b4220f041a33ec24c226199" -dependencies = [ - "async-trait", - "bytes", - "futures-util", - "http 1.4.2", - "http-body 1.1.0", - "http-body-util", - "mime", - "pin-project-lite", - "rustversion", - "sync_wrapper 1.0.2", - "tower-layer", - "tower-service", - "tracing", -] - [[package]] name = "axum-core" version = "0.5.6" @@ -2888,7 +2833,7 @@ dependencies = [ [[package]] name = "base" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-chains", "axum 0.8.9", @@ -2932,7 +2877,7 @@ dependencies = [ [[package]] name = "base-acceptance" -version = "1.5.1" +version = "1.5.4" dependencies = [ "chrono", "clap", @@ -2948,7 +2893,7 @@ dependencies = [ [[package]] name = "base-acceptance-cli" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-acceptance", "clap", @@ -2957,7 +2902,7 @@ dependencies = [ [[package]] name = "base-action-harness" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -3024,7 +2969,7 @@ dependencies = [ [[package]] name = "base-balance-monitor" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-network 2.4.1", "alloy-node-bindings", @@ -3037,7 +2982,7 @@ dependencies = [ [[package]] name = "base-batcher-admin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-batcher-core", "base-runtime", @@ -3051,7 +2996,7 @@ dependencies = [ [[package]] name = "base-batcher-cli" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "alloy-signer-local 2.4.1", @@ -3073,7 +3018,7 @@ dependencies = [ [[package]] name = "base-batcher-core" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -3098,7 +3043,7 @@ dependencies = [ [[package]] name = "base-batcher-encoder" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -3124,7 +3069,7 @@ dependencies = [ [[package]] name = "base-batcher-service" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-node-bindings", "alloy-primitives", @@ -3166,7 +3111,7 @@ dependencies = [ [[package]] name = "base-batcher-source" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -3184,7 +3129,7 @@ dependencies = [ [[package]] name = "base-blobs" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-eips 2.4.1", "alloy-primitives", @@ -3195,7 +3140,7 @@ dependencies = [ [[package]] name = "base-builder-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-builder-cli", "base-builder-core", @@ -3214,7 +3159,7 @@ dependencies = [ [[package]] name = "base-builder-cli" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "base-builder-core", @@ -3225,13 +3170,16 @@ dependencies = [ "base-observability-events", "clap", "eyre", + "metrics", + "metrics-exporter-prometheus", "rstest", + "tempfile", "tracing", ] [[package]] name = "base-builder-core" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-contract", @@ -3361,7 +3309,7 @@ dependencies = [ [[package]] name = "base-builder-metering" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "base-builder-core", @@ -3374,7 +3322,7 @@ dependencies = [ [[package]] name = "base-builder-multiplex" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -3417,7 +3365,7 @@ dependencies = [ [[package]] name = "base-builder-publish" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-metrics", "base-ring-buffer", @@ -3437,7 +3385,7 @@ dependencies = [ [[package]] name = "base-bundles" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -3453,7 +3401,7 @@ dependencies = [ [[package]] name = "base-challenger" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -3498,7 +3446,7 @@ dependencies = [ [[package]] name = "base-challenger-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-challenger", "base-cli-utils", @@ -3509,7 +3457,7 @@ dependencies = [ [[package]] name = "base-challenger-e2e" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-node-bindings", @@ -3537,7 +3485,7 @@ dependencies = [ [[package]] name = "base-challenger-e2e-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-challenger-e2e", "tokio", @@ -3547,7 +3495,7 @@ dependencies = [ [[package]] name = "base-cli-utils" -version = "1.5.1" +version = "1.5.4" dependencies = [ "backtrace", "base-metrics", @@ -3574,7 +3522,7 @@ dependencies = [ [[package]] name = "base-common-chains" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-chains", "alloy-eips 2.4.1", @@ -3589,7 +3537,7 @@ dependencies = [ [[package]] name = "base-common-consensus" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -3626,7 +3574,7 @@ dependencies = [ [[package]] name = "base-common-eip8130" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "alloy-sol-types", @@ -3636,7 +3584,7 @@ dependencies = [ [[package]] name = "base-common-evm" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -3667,7 +3615,7 @@ dependencies = [ [[package]] name = "base-common-evm2" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -3686,7 +3634,7 @@ dependencies = [ [[package]] name = "base-common-flashblocks" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "alloy-rpc-types-engine", @@ -3704,7 +3652,7 @@ dependencies = [ [[package]] name = "base-common-flz" -version = "1.5.1" +version = "1.5.4" dependencies = [ "criterion", "hex-literal 1.1.0", @@ -3714,7 +3662,7 @@ dependencies = [ [[package]] name = "base-common-genesis" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-chains", "alloy-consensus 2.4.1", @@ -3736,7 +3684,7 @@ dependencies = [ [[package]] name = "base-common-l1-fees" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "base-common-flz", @@ -3745,7 +3693,7 @@ dependencies = [ [[package]] name = "base-common-network" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-network 2.4.1", @@ -3762,7 +3710,7 @@ dependencies = [ [[package]] name = "base-common-precompiles" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-evm", "alloy-primitives", @@ -3780,7 +3728,7 @@ dependencies = [ [[package]] name = "base-common-rpc-types" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -3807,7 +3755,7 @@ dependencies = [ [[package]] name = "base-common-rpc-types-engine" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -3830,7 +3778,7 @@ dependencies = [ [[package]] name = "base-common-signer" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -3850,7 +3798,7 @@ dependencies = [ [[package]] name = "base-comp" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-common-genesis", "base-protocol", @@ -3861,7 +3809,7 @@ dependencies = [ [[package]] name = "base-consensus" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-cli-utils", "base-consensus-cli", @@ -3870,7 +3818,7 @@ dependencies = [ [[package]] name = "base-consensus-cli" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-chains", "alloy-genesis", @@ -3919,7 +3867,7 @@ dependencies = [ [[package]] name = "base-consensus-derive" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -3950,7 +3898,7 @@ dependencies = [ [[package]] name = "base-consensus-disc" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-rlp", "backon", @@ -3973,7 +3921,7 @@ dependencies = [ [[package]] name = "base-consensus-engine" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -4020,7 +3968,7 @@ dependencies = [ [[package]] name = "base-consensus-gossip" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-chains", "alloy-consensus 2.4.1", @@ -4060,7 +4008,7 @@ dependencies = [ [[package]] name = "base-consensus-node" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-chains", "alloy-consensus 2.4.1", @@ -4137,7 +4085,7 @@ dependencies = [ [[package]] name = "base-consensus-peers" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "alloy-rlp", @@ -4162,7 +4110,7 @@ dependencies = [ [[package]] name = "base-consensus-providers" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -4200,7 +4148,7 @@ dependencies = [ [[package]] name = "base-consensus-rpc" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-eips 2.4.1", "alloy-primitives", @@ -4229,7 +4177,7 @@ dependencies = [ [[package]] name = "base-consensus-safedb" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-eips 2.4.1", "alloy-primitives", @@ -4245,7 +4193,7 @@ dependencies = [ [[package]] name = "base-consensus-sources" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "alloy-rpc-client", @@ -4267,7 +4215,7 @@ dependencies = [ [[package]] name = "base-consensus-upgrades" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-eips 2.4.1", "alloy-primitives", @@ -4279,7 +4227,7 @@ dependencies = [ [[package]] name = "base-execution-chainspec" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-chains", "alloy-consensus 2.4.1", @@ -4304,7 +4252,7 @@ dependencies = [ [[package]] name = "base-execution-cli" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -4383,7 +4331,7 @@ dependencies = [ [[package]] name = "base-execution-consensus" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-chains", "alloy-consensus 2.4.1", @@ -4416,7 +4364,7 @@ dependencies = [ [[package]] name = "base-execution-eip8130" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "alloy-sol-types", @@ -4435,7 +4383,7 @@ dependencies = [ [[package]] name = "base-execution-eip8130-rpc" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -4464,7 +4412,7 @@ dependencies = [ [[package]] name = "base-execution-eip8130-rpc-node" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-eips 2.4.1", "alloy-genesis", @@ -4489,7 +4437,7 @@ dependencies = [ [[package]] name = "base-execution-evm" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -4521,7 +4469,7 @@ dependencies = [ [[package]] name = "base-execution-exex" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -4547,7 +4495,7 @@ dependencies = [ [[package]] name = "base-execution-firehose" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-evm", @@ -4566,7 +4514,7 @@ dependencies = [ [[package]] name = "base-execution-payload-builder" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -4587,10 +4535,13 @@ dependencies = [ "base-execution-txpool", "base-metrics", "base-observability-events", + "clap", + "criterion", "derive_more 2.1.1", "metrics", "metrics-exporter-prometheus", "moka", + "parking_lot", "reth-basic-payload-builder", "reth-chainspec", "reth-engine-tree", @@ -4607,6 +4558,7 @@ dependencies = [ "reth-provider", "reth-revm", "reth-storage-api", + "reth-tasks", "reth-transaction-pool", "reth-trie-common", "reth-trie-parallel", @@ -4619,7 +4571,7 @@ dependencies = [ [[package]] name = "base-execution-rpc" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -4696,7 +4648,7 @@ dependencies = [ [[package]] name = "base-execution-trie" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -4750,7 +4702,7 @@ dependencies = [ [[package]] name = "base-execution-txpool" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -4804,7 +4756,7 @@ dependencies = [ [[package]] name = "base-firehose-flashblocks" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -4849,7 +4801,7 @@ dependencies = [ [[package]] name = "base-firehose-prestate" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-genesis", "base-common-chains", @@ -4867,7 +4819,7 @@ dependencies = [ [[package]] name = "base-firehose-tests" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -4898,7 +4850,7 @@ dependencies = [ [[package]] name = "base-flashblocks" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -4956,7 +4908,7 @@ dependencies = [ [[package]] name = "base-flashblocks-node" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-contract", @@ -4977,11 +4929,13 @@ dependencies = [ "base-common-network", "base-common-rpc-types", "base-execution-chainspec", + "base-execution-txpool", "base-flashblocks", "base-flashblocks-node", "base-node-core", "base-node-runner", "base-test-utils", + "base-txpool-rpc", "criterion", "derive_more 2.1.1", "eyre", @@ -5011,7 +4965,7 @@ dependencies = [ [[package]] name = "base-health" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-transport-http", "async-trait", @@ -5033,7 +4987,7 @@ dependencies = [ [[package]] name = "base-jwt" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "alloy-provider", @@ -5049,7 +5003,7 @@ dependencies = [ [[package]] name = "base-l1-head" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "alloy-provider", @@ -5063,7 +5017,7 @@ dependencies = [ [[package]] name = "base-load-tester-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-network 2.4.1", "alloy-primitives", @@ -5084,7 +5038,7 @@ dependencies = [ [[package]] name = "base-load-tests" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -5127,7 +5081,7 @@ dependencies = [ [[package]] name = "base-metering" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -5173,7 +5127,7 @@ dependencies = [ [[package]] name = "base-metrics" -version = "1.5.1" +version = "1.5.4" dependencies = [ "ctor", "metrics", @@ -5183,7 +5137,7 @@ dependencies = [ [[package]] name = "base-node-core" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -5260,7 +5214,7 @@ dependencies = [ [[package]] name = "base-node-runner" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-eips 2.4.1", "alloy-genesis", @@ -5312,7 +5266,7 @@ dependencies = [ [[package]] name = "base-observability-events" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "base-metrics", @@ -5331,7 +5285,7 @@ dependencies = [ [[package]] name = "base-optimism-rpc" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-eips 2.4.1", "alloy-network 2.4.1", @@ -5344,7 +5298,7 @@ dependencies = [ [[package]] name = "base-precompile-macros" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "proc-macro2", @@ -5354,7 +5308,7 @@ dependencies = [ [[package]] name = "base-precompile-storage" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-evm", "alloy-primitives", @@ -5370,7 +5324,7 @@ dependencies = [ [[package]] name = "base-proof" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -5408,7 +5362,7 @@ dependencies = [ [[package]] name = "base-proof-client" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-evm", @@ -5431,7 +5385,7 @@ dependencies = [ [[package]] name = "base-proof-contracts" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-contract", "alloy-primitives", @@ -5449,7 +5403,7 @@ dependencies = [ [[package]] name = "base-proof-driver" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-evm", @@ -5469,7 +5423,7 @@ dependencies = [ [[package]] name = "base-proof-executor" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -5503,7 +5457,7 @@ dependencies = [ [[package]] name = "base-proof-host" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -5546,7 +5500,7 @@ dependencies = [ [[package]] name = "base-proof-mpt" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -5567,7 +5521,7 @@ dependencies = [ [[package]] name = "base-proof-preimage" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "async-channel", @@ -5581,7 +5535,7 @@ dependencies = [ [[package]] name = "base-proof-primitives" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-chains", "alloy-eips 2.4.1", @@ -5599,7 +5553,7 @@ dependencies = [ [[package]] name = "base-proof-rpc" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-eips 2.4.1", "alloy-network 2.4.1", @@ -5628,7 +5582,7 @@ dependencies = [ [[package]] name = "base-proof-submission" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -5644,11 +5598,11 @@ dependencies = [ [[package]] name = "base-proof-succinct-elfs" -version = "1.5.1" +version = "1.5.4" [[package]] name = "base-proof-succinct-scripts" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "anyhow", @@ -5664,7 +5618,7 @@ dependencies = [ [[package]] name = "base-proof-tee-nitro-enclave" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-chains", "alloy-eips 2.4.1", @@ -5697,7 +5651,7 @@ dependencies = [ [[package]] name = "base-proof-tee-nitro-host" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-genesis", "alloy-primitives", @@ -5729,7 +5683,7 @@ dependencies = [ [[package]] name = "base-proof-tee-nitro-verifier" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "alloy-sol-types", @@ -5746,7 +5700,7 @@ dependencies = [ [[package]] name = "base-proof-tee-registrar" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -5788,7 +5742,7 @@ dependencies = [ [[package]] name = "base-proof-tee-registrar-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "alloy-signer-local 2.4.1", @@ -5806,7 +5760,7 @@ dependencies = [ [[package]] name = "base-proof-worker" -version = "1.5.1" +version = "1.5.4" dependencies = [ "async-trait", "backon", @@ -5822,7 +5776,7 @@ dependencies = [ [[package]] name = "base-proof-zk-backend" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -5857,7 +5811,7 @@ dependencies = [ [[package]] name = "base-proof-zk-host" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "async-trait", @@ -5874,7 +5828,7 @@ dependencies = [ [[package]] name = "base-proof-zk-utils" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -5911,7 +5865,7 @@ dependencies = [ [[package]] name = "base-proof-zk-witness" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -5946,7 +5900,7 @@ dependencies = [ [[package]] name = "base-proofs-extension" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-execution-exex", "base-execution-rpc", @@ -5963,7 +5917,7 @@ dependencies = [ [[package]] name = "base-proposer" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -6005,7 +5959,7 @@ dependencies = [ [[package]] name = "base-proposer-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-cli-utils", "base-proposer", @@ -6017,7 +5971,7 @@ dependencies = [ [[package]] name = "base-protocol" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloc-no-stdlib", "alloy-consensus 2.4.1", @@ -6053,7 +6007,7 @@ dependencies = [ [[package]] name = "base-prover-nitro-enclave" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-proof-tee-nitro-enclave", "eyre", @@ -6062,7 +6016,7 @@ dependencies = [ [[package]] name = "base-prover-nitro-host" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "base-cli-utils", @@ -6084,7 +6038,7 @@ dependencies = [ [[package]] name = "base-prover-service" -version = "1.5.1" +version = "1.5.4" dependencies = [ "anyhow", "base-prover-service-db", @@ -6102,7 +6056,7 @@ dependencies = [ [[package]] name = "base-prover-service-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-cli-utils", "base-prover-service", @@ -6119,7 +6073,7 @@ dependencies = [ [[package]] name = "base-prover-service-client" -version = "1.5.1" +version = "1.5.4" dependencies = [ "async-trait", "backon", @@ -6136,7 +6090,7 @@ dependencies = [ [[package]] name = "base-prover-service-db" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "anyhow", @@ -6155,7 +6109,7 @@ dependencies = [ [[package]] name = "base-prover-service-protocol" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "base-proof-primitives", @@ -6168,7 +6122,7 @@ dependencies = [ [[package]] name = "base-prover-zk-host" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-cli-utils", "base-proof-worker", @@ -6188,7 +6142,7 @@ dependencies = [ [[package]] name = "base-reth-cli" -version = "1.5.1" +version = "1.5.4" dependencies = [ "anyhow", "base-common-chains", @@ -6207,7 +6161,7 @@ dependencies = [ [[package]] name = "base-reth-node" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-cli-utils", "base-common-genesis", @@ -6231,18 +6185,18 @@ dependencies = [ [[package]] name = "base-retry" -version = "1.5.1" +version = "1.5.4" dependencies = [ "backon", ] [[package]] name = "base-ring-buffer" -version = "1.5.1" +version = "1.5.4" [[package]] name = "base-roxy" -version = "1.5.1" +version = "1.5.4" dependencies = [ "anyhow", "axum 0.8.9", @@ -6256,7 +6210,7 @@ dependencies = [ [[package]] name = "base-roxy-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "anyhow", "base-cli-utils", @@ -6269,7 +6223,7 @@ dependencies = [ [[package]] name = "base-runtime" -version = "1.5.1" +version = "1.5.4" dependencies = [ "futures", "rand 0.9.5", @@ -6281,7 +6235,7 @@ dependencies = [ [[package]] name = "base-shadow-indexer" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -6315,7 +6269,7 @@ dependencies = [ [[package]] name = "base-shadow-indexer-db" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -6332,7 +6286,7 @@ dependencies = [ [[package]] name = "base-shadow-metrics" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -6351,7 +6305,7 @@ dependencies = [ [[package]] name = "base-shadow-metrics-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "anyhow", "axum 0.8.9", @@ -6366,7 +6320,7 @@ dependencies = [ [[package]] name = "base-sidecrush" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-provider", "alloy-rpc-types-eth 2.4.1", @@ -6379,7 +6333,7 @@ dependencies = [ [[package]] name = "base-sidecrush-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-sidecrush", "cadence", @@ -6391,7 +6345,7 @@ dependencies = [ [[package]] name = "base-snapshotter" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-eips 2.4.1", "alloy-provider", @@ -6418,7 +6372,7 @@ dependencies = [ [[package]] name = "base-snapshotter-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "anyhow", "aws-config", @@ -6435,7 +6389,7 @@ dependencies = [ [[package]] name = "base-snark-e2e" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "alloy-provider", @@ -6458,7 +6412,7 @@ dependencies = [ [[package]] name = "base-snark-e2e-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-snark-e2e", "tokio", @@ -6468,7 +6422,7 @@ dependencies = [ [[package]] name = "base-system-tests" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -6574,7 +6528,7 @@ dependencies = [ [[package]] name = "base-telemetry-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "anyhow", "base-cli-utils", @@ -6588,7 +6542,7 @@ dependencies = [ [[package]] name = "base-telemetry-service" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "anyhow", @@ -6618,7 +6572,7 @@ dependencies = [ [[package]] name = "base-test-utils" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-contract", @@ -6638,7 +6592,7 @@ dependencies = [ [[package]] name = "base-trusted-proxy" -version = "1.5.1" +version = "1.5.4" dependencies = [ "http 1.4.2", "ipnet", @@ -6648,7 +6602,7 @@ dependencies = [ [[package]] name = "base-tx-forwarding" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -6684,7 +6638,7 @@ dependencies = [ [[package]] name = "base-tx-manager" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -6719,7 +6673,7 @@ dependencies = [ [[package]] name = "base-txpool-rpc" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -6755,13 +6709,14 @@ dependencies = [ [[package]] name = "base-txpool-tracing" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", "alloy-primitives", "alloy-rpc-types-engine", "base-common-consensus", + "base-execution-txpool", "base-flashblocks", "base-flashblocks-node", "base-metrics", @@ -6774,12 +6729,14 @@ dependencies = [ "futures", "lru 0.16.4", "metrics", + "metrics-exporter-prometheus", "reth-node-api", "reth-primitives-traits", "reth-provider", "reth-tracing", "reth-transaction-pool", "revm-database", + "rstest", "serde_json", "tokio", "tokio-stream", @@ -6788,7 +6745,7 @@ dependencies = [ [[package]] name = "base-upgrade-signal" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-json-rpc 2.4.1", "alloy-primitives", @@ -6820,7 +6777,7 @@ dependencies = [ [[package]] name = "base-witness-diff" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-primitives", "alloy-provider", @@ -6846,7 +6803,7 @@ checksum = "4cbbc9d0964165b47557570cce6c952866c2678457aca742aafc9fb771d30270" [[package]] name = "base-zk-benchmarks" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-eips 2.4.1", "alloy-primitives", @@ -6864,7 +6821,7 @@ dependencies = [ [[package]] name = "base-zk-benchmarks-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-cli-utils", "base-load-tests", @@ -6877,7 +6834,7 @@ dependencies = [ [[package]] name = "base-zk-fork-dispute" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-node-bindings", "alloy-primitives", @@ -6905,7 +6862,7 @@ dependencies = [ [[package]] name = "base-zk-fork-dispute-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "base-cli-utils", "base-zk-fork-dispute", @@ -6972,7 +6929,7 @@ checksum = "8c3c1a368f70d6cf7302d78f8f7093da241fb8e8807c05cc9e51a125895a6d5b" [[package]] name = "basectl" -version = "1.5.1" +version = "1.5.4" dependencies = [ "anyhow", "basectl-cli", @@ -6984,7 +6941,7 @@ dependencies = [ [[package]] name = "basectl-cli" -version = "1.5.1" +version = "1.5.4" dependencies = [ "alloy-consensus 2.4.1", "alloy-contract", @@ -7495,7 +7452,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "85a885520bf6249ab931a764ffdb87b0ceef48e6e7d807cfdb21b751e086e1ad" dependencies = [ "prost 0.14.4", - "prost-types 0.14.4", + "prost-types", "tonic 0.14.6", "tonic-prost", "ureq", @@ -9954,7 +9911,7 @@ dependencies = [ "k256 0.13.4", "num_enum 0.7.6", "prost 0.14.4", - "prost-types 0.14.4", + "prost-types", "serde", "serde_json", "tracing", @@ -11297,7 +11254,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0b1e4910d3a9137442723dfb772c32dc10674c4181ca078d2fd227cd5dce9db0" dependencies = [ "bincode 1.3.3", - "derive_more 1.0.0", + "derive_more 2.1.1", "iai-callgrind-macros", "iai-callgrind-runner", ] @@ -11308,7 +11265,7 @@ version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4d03775318d3f9f01b39ac6612b01464006dc397a654a89dd57df2fd34fb68c3" dependencies = [ - "derive_more 1.0.0", + "derive_more 2.1.1", "proc-macro-error2", "proc-macro2", "quote", @@ -12272,9 +12229,9 @@ dependencies = [ [[package]] name = "kzg-rs" -version = "0.2.8" +version = "0.2.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee8b4f55c3dedcfaa8668de1dfc8469e7a32d441c28edf225ed1f566fb32977d" +checksum = "433994161b1b86020801f5486deebcd90c1e654d5713e323ef7ef636445e9b9d" dependencies = [ "ff 0.13.1", "hex", @@ -12842,15 +12799,6 @@ dependencies = [ "tracing-subscriber", ] -[[package]] -name = "lru" -version = "0.12.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "234cf4f4a04dc1f57e24b96cc0cd600cf2af460d4161ac5ecdd0af8e1f3b2a38" -dependencies = [ - "hashbrown 0.15.5", -] - [[package]] name = "lru" version = "0.16.4" @@ -12862,9 +12810,9 @@ dependencies = [ [[package]] name = "lru" -version = "0.18.2" +version = "0.18.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d2f2f9b4ba7e6b24d95e7e899329d35be83bcded72c8540cdd5368932d1d90a" +checksum = "ef9ac18847474e638e3702b76c65d4eb93428471a74778ef0f1be711717f89b5" dependencies = [ "hashbrown 0.17.1", ] @@ -13898,11 +13846,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ff76201f031d8863c38aa7f905eca4f53abbfa15f609db4277d44cd8938f33fe" dependencies = [ "crc32fast", - "flate2", "hashbrown 0.15.5", "indexmap 2.14.0", "memchr", - "ruzstd", ] [[package]] @@ -14810,11 +14756,12 @@ dependencies = [ [[package]] name = "petgraph" -version = "0.7.1" +version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772" +checksum = "8701b58ea97060d5e5b155d383a69952a60943f0e6dfe30b04c287beb0b27455" dependencies = [ "fixedbitset 0.5.7", + "hashbrown 0.15.5", "indexmap 2.14.0", ] @@ -15497,16 +15444,6 @@ dependencies = [ "prost-derive 0.12.6", ] -[[package]] -name = "prost" -version = "0.13.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2796faa41db3ec313a31f7624d9286acf277b52de526150b7e69f3debf891ee5" -dependencies = [ - "bytes", - "prost-derive 0.13.5", -] - [[package]] name = "prost" version = "0.14.4" @@ -15519,19 +15456,20 @@ dependencies = [ [[package]] name = "prost-build" -version = "0.13.5" +version = "0.14.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf" +checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042" dependencies = [ "heck", "itertools 0.14.0", "log", "multimap", - "once_cell", "petgraph", "prettyplease", - "prost 0.13.5", - "prost-types 0.13.5", + "prost 0.14.4", + "prost-types", + "pulldown-cmark", + "pulldown-cmark-to-cmark", "regex", "syn 2.0.119", "tempfile", @@ -15550,19 +15488,6 @@ dependencies = [ "syn 2.0.119", ] -[[package]] -name = "prost-derive" -version = "0.13.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" -dependencies = [ - "anyhow", - "itertools 0.14.0", - "proc-macro2", - "quote", - "syn 2.0.119", -] - [[package]] name = "prost-derive" version = "0.14.4" @@ -15583,16 +15508,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "01b80ea363c31af2de2b92e3c07ed1156628f7838c4afb4df75ee78a37fedbd1" dependencies = [ "prost 0.14.4", - "prost-types 0.14.4", -] - -[[package]] -name = "prost-types" -version = "0.13.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52c2c1bf36ddb1a1c396b3601a3cec27c2462e45f07c386894ec3ccf5332bd16" -dependencies = [ - "prost 0.13.5", + "prost-types", ] [[package]] @@ -15624,6 +15540,26 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "pulldown-cmark" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "278e9268c93b9e498a2da46c34d90ba93a3255eb99009646398f34bc73ab36eb" +dependencies = [ + "bitflags 2.13.1", + "memchr", + "unicase", +] + +[[package]] +name = "pulldown-cmark-to-cmark" +version = "22.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "84bbb29c624230c4bd1047bbdb2aa47e41c860e9665ce62ba9504eebe91bf867" +dependencies = [ + "pulldown-cmark", +] + [[package]] name = "pxfm" version = "0.1.30" @@ -15930,7 +15866,7 @@ dependencies = [ "hashbrown 0.17.1", "itertools 0.14.0", "kasuari", - "lru 0.18.2", + "lru 0.18.5", "palette", "serde", "strum 0.28.0", @@ -16326,7 +16262,7 @@ checksum = "1e061d1b48cb8d38042de4ae0a7a6401009d6143dc80d2e2d6f31f0bdd6470c7" [[package]] name = "reth-basic-payload-builder" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -16353,7 +16289,7 @@ dependencies = [ [[package]] name = "reth-chain-state" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -16384,7 +16320,7 @@ dependencies = [ [[package]] name = "reth-chainspec" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-chains", "alloy-consensus 2.4.1", @@ -16404,7 +16340,7 @@ dependencies = [ [[package]] name = "reth-cli" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-genesis", "clap", @@ -16417,7 +16353,7 @@ dependencies = [ [[package]] name = "reth-cli-commands" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-chains", "alloy-consensus 2.4.1", @@ -16506,7 +16442,7 @@ dependencies = [ [[package]] name = "reth-cli-runner" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "reth-tasks", "tokio", @@ -16516,7 +16452,7 @@ dependencies = [ [[package]] name = "reth-cli-util" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-eips 2.4.1", "alloy-primitives", @@ -16567,7 +16503,7 @@ dependencies = [ [[package]] name = "reth-config" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "eyre", "humantime-serde", @@ -16584,7 +16520,7 @@ dependencies = [ [[package]] name = "reth-consensus" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eip7928 0.4.5", @@ -16598,7 +16534,7 @@ dependencies = [ [[package]] name = "reth-consensus-common" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -16611,7 +16547,7 @@ dependencies = [ [[package]] name = "reth-consensus-debug-client" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -16636,7 +16572,7 @@ dependencies = [ [[package]] name = "reth-db" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "derive_more 2.1.1", @@ -16665,7 +16601,7 @@ dependencies = [ [[package]] name = "reth-db-api" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -16691,7 +16627,7 @@ dependencies = [ [[package]] name = "reth-db-common" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-genesis", @@ -16721,7 +16657,7 @@ dependencies = [ [[package]] name = "reth-db-models" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-eips 2.4.1", "alloy-primitives", @@ -16736,7 +16672,7 @@ dependencies = [ [[package]] name = "reth-discv4" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "alloy-rlp", @@ -16761,7 +16697,7 @@ dependencies = [ [[package]] name = "reth-discv5" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "alloy-rlp", @@ -16785,7 +16721,7 @@ dependencies = [ [[package]] name = "reth-dns-discovery" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "dashmap", @@ -16809,7 +16745,7 @@ dependencies = [ [[package]] name = "reth-downloaders" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -16844,7 +16780,7 @@ dependencies = [ [[package]] name = "reth-e2e-test-utils" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -16901,7 +16837,7 @@ dependencies = [ [[package]] name = "reth-ecies" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "aes", "alloy-primitives", @@ -16928,7 +16864,7 @@ dependencies = [ [[package]] name = "reth-engine-local" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -16951,7 +16887,7 @@ dependencies = [ [[package]] name = "reth-engine-primitives" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -16978,7 +16914,7 @@ dependencies = [ [[package]] name = "reth-engine-tree" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eip7928 0.4.5", @@ -17035,7 +16971,7 @@ dependencies = [ [[package]] name = "reth-engine-util" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -17065,7 +17001,7 @@ dependencies = [ [[package]] name = "reth-era" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -17083,7 +17019,7 @@ dependencies = [ [[package]] name = "reth-era-downloader" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "bytes", @@ -17099,7 +17035,7 @@ dependencies = [ [[package]] name = "reth-era-utils" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -17123,7 +17059,7 @@ dependencies = [ [[package]] name = "reth-errors" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "reth-consensus", "reth-execution-errors", @@ -17134,7 +17070,7 @@ dependencies = [ [[package]] name = "reth-eth-wire" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-chains", "alloy-primitives", @@ -17163,7 +17099,7 @@ dependencies = [ [[package]] name = "reth-eth-wire-types" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-chains", "alloy-consensus 2.4.1", @@ -17189,7 +17125,7 @@ dependencies = [ [[package]] name = "reth-ethereum-consensus" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -17205,7 +17141,7 @@ dependencies = [ [[package]] name = "reth-ethereum-engine-primitives" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-eips 2.4.1", "alloy-primitives", @@ -17221,7 +17157,7 @@ dependencies = [ [[package]] name = "reth-ethereum-forks" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-eip2124", "alloy-hardforks 0.4.7", @@ -17235,7 +17171,7 @@ dependencies = [ [[package]] name = "reth-ethereum-payload-builder" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -17265,7 +17201,7 @@ dependencies = [ [[package]] name = "reth-ethereum-primitives" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -17279,7 +17215,7 @@ dependencies = [ [[package]] name = "reth-etl" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "rayon", "reth-db-api", @@ -17289,7 +17225,7 @@ dependencies = [ [[package]] name = "reth-evm" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eip7928 0.4.5", @@ -17315,7 +17251,7 @@ dependencies = [ [[package]] name = "reth-evm-ethereum" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -17335,7 +17271,7 @@ dependencies = [ [[package]] name = "reth-execution-cache" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "fixed-cache", @@ -17353,7 +17289,7 @@ dependencies = [ [[package]] name = "reth-execution-errors" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-evm", "alloy-primitives", @@ -17366,7 +17302,7 @@ dependencies = [ [[package]] name = "reth-execution-types" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -17385,7 +17321,7 @@ dependencies = [ [[package]] name = "reth-exex" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -17423,7 +17359,7 @@ dependencies = [ [[package]] name = "reth-exex-test-utils" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-eips 2.4.1", "eyre", @@ -17455,7 +17391,7 @@ dependencies = [ [[package]] name = "reth-exex-types" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-eips 2.4.1", "alloy-primitives", @@ -17469,7 +17405,7 @@ dependencies = [ [[package]] name = "reth-firehose" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eip7928 0.4.5", @@ -17498,7 +17434,7 @@ dependencies = [ [[package]] name = "reth-firehose-tests" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -17525,7 +17461,7 @@ dependencies = [ [[package]] name = "reth-fs-util" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "serde", "serde_json", @@ -17535,7 +17471,7 @@ dependencies = [ [[package]] name = "reth-invalid-block-hooks" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -17561,7 +17497,7 @@ dependencies = [ [[package]] name = "reth-ipc" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "bytes", "futures", @@ -17581,7 +17517,7 @@ dependencies = [ [[package]] name = "reth-libmdbx" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "bitflags 2.13.1", "byteorder", @@ -17598,7 +17534,7 @@ dependencies = [ [[package]] name = "reth-mdbx-sys" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "bindgen 0.72.1", "cc", @@ -17607,7 +17543,7 @@ dependencies = [ [[package]] name = "reth-metrics" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "futures", "libc", @@ -17621,7 +17557,7 @@ dependencies = [ [[package]] name = "reth-net-banlist" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "ipnet", @@ -17630,7 +17566,7 @@ dependencies = [ [[package]] name = "reth-net-nat" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "futures-util", "if-addrs 0.14.0", @@ -17644,7 +17580,7 @@ dependencies = [ [[package]] name = "reth-network" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -17703,7 +17639,7 @@ dependencies = [ [[package]] name = "reth-network-api" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -17728,7 +17664,7 @@ dependencies = [ [[package]] name = "reth-network-p2p" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eip7928 0.4.5", @@ -17752,7 +17688,7 @@ dependencies = [ [[package]] name = "reth-network-peers" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "alloy-rlp", @@ -17767,7 +17703,7 @@ dependencies = [ [[package]] name = "reth-network-types" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-eip2124", "humantime-serde", @@ -17781,7 +17717,7 @@ dependencies = [ [[package]] name = "reth-nippy-jar" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "anyhow", "bincode 1.3.3", @@ -17798,7 +17734,7 @@ dependencies = [ [[package]] name = "reth-node-api" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-rpc-types-engine", "eyre", @@ -17822,7 +17758,7 @@ dependencies = [ [[package]] name = "reth-node-builder" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -17891,7 +17827,7 @@ dependencies = [ [[package]] name = "reth-node-core" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -17946,7 +17882,7 @@ dependencies = [ [[package]] name = "reth-node-ethereum" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -17995,7 +17931,7 @@ dependencies = [ [[package]] name = "reth-node-ethstats" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -18019,7 +17955,7 @@ dependencies = [ [[package]] name = "reth-node-events" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -18043,7 +17979,7 @@ dependencies = [ [[package]] name = "reth-node-metrics" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "bytes", "eyre", @@ -18072,7 +18008,7 @@ dependencies = [ [[package]] name = "reth-node-types" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "reth-chainspec", "reth-db-api", @@ -18084,7 +18020,7 @@ dependencies = [ [[package]] name = "reth-payload-builder" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -18108,7 +18044,7 @@ dependencies = [ [[package]] name = "reth-payload-builder-primitives" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "pin-project", "reth-payload-primitives", @@ -18120,7 +18056,7 @@ dependencies = [ [[package]] name = "reth-payload-primitives" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -18143,7 +18079,7 @@ dependencies = [ [[package]] name = "reth-payload-util" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -18153,7 +18089,7 @@ dependencies = [ [[package]] name = "reth-payload-validator" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-rpc-types-engine", @@ -18196,7 +18132,7 @@ dependencies = [ [[package]] name = "reth-provider" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eip7928 0.4.5", @@ -18246,7 +18182,7 @@ dependencies = [ [[package]] name = "reth-prune" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -18273,7 +18209,7 @@ dependencies = [ [[package]] name = "reth-prune-types" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "arbitrary", @@ -18289,7 +18225,7 @@ dependencies = [ [[package]] name = "reth-revm" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "alloy-rlp", @@ -18304,7 +18240,7 @@ dependencies = [ [[package]] name = "reth-rpc" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-dyn-abi", @@ -18380,7 +18316,7 @@ dependencies = [ [[package]] name = "reth-rpc-api" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-eips 2.4.1", "alloy-genesis", @@ -18410,7 +18346,7 @@ dependencies = [ [[package]] name = "reth-rpc-builder" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-network 2.4.1", "alloy-provider", @@ -18453,7 +18389,7 @@ dependencies = [ [[package]] name = "reth-rpc-convert" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-evm", @@ -18473,7 +18409,7 @@ dependencies = [ [[package]] name = "reth-rpc-engine-api" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-eips 2.4.1", "alloy-primitives", @@ -18504,7 +18440,7 @@ dependencies = [ [[package]] name = "reth-rpc-eth-api" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-dyn-abi", @@ -18551,7 +18487,7 @@ dependencies = [ [[package]] name = "reth-rpc-eth-types" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-chains", "alloy-consensus 2.4.1", @@ -18602,7 +18538,7 @@ dependencies = [ [[package]] name = "reth-rpc-layer" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-rpc-types-engine", "http 1.4.2", @@ -18616,7 +18552,7 @@ dependencies = [ [[package]] name = "reth-rpc-server-types" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-eips 2.4.1", "alloy-primitives", @@ -18647,7 +18583,7 @@ dependencies = [ [[package]] name = "reth-stages" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -18698,7 +18634,7 @@ dependencies = [ [[package]] name = "reth-stages-api" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-eips 2.4.1", "alloy-primitives", @@ -18726,7 +18662,7 @@ dependencies = [ [[package]] name = "reth-stages-types" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "arbitrary", @@ -18740,7 +18676,7 @@ dependencies = [ [[package]] name = "reth-static-file" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "parking_lot", @@ -18760,7 +18696,7 @@ dependencies = [ [[package]] name = "reth-static-file-types" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "clap", @@ -18775,7 +18711,7 @@ dependencies = [ [[package]] name = "reth-storage-api" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eip7928 0.4.5", @@ -18801,7 +18737,7 @@ dependencies = [ [[package]] name = "reth-storage-errors" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-eips 2.4.1", "alloy-primitives", @@ -18818,7 +18754,7 @@ dependencies = [ [[package]] name = "reth-storage-overlay" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-eips 2.4.1", "alloy-primitives", @@ -18845,7 +18781,7 @@ dependencies = [ [[package]] name = "reth-tasks" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "crossbeam-utils", "dashmap", @@ -18866,7 +18802,7 @@ dependencies = [ [[package]] name = "reth-testing-utils" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -18882,7 +18818,7 @@ dependencies = [ [[package]] name = "reth-tokio-util" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "tokio", "tokio-stream", @@ -18892,7 +18828,7 @@ dependencies = [ [[package]] name = "reth-tracing" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "clap", "eyre", @@ -18912,7 +18848,7 @@ dependencies = [ [[package]] name = "reth-tracing-otlp" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "base64 0.22.1", "clap", @@ -18931,7 +18867,7 @@ dependencies = [ [[package]] name = "reth-transaction-pool" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -18974,7 +18910,7 @@ dependencies = [ [[package]] name = "reth-trie" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-eips 2.4.1", @@ -19000,7 +18936,7 @@ dependencies = [ [[package]] name = "reth-trie-common" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-consensus 2.4.1", "alloy-primitives", @@ -19027,7 +18963,7 @@ dependencies = [ [[package]] name = "reth-trie-db" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "reth-db-api", @@ -19043,7 +18979,7 @@ dependencies = [ [[package]] name = "reth-trie-parallel" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-evm", "alloy-primitives", @@ -19067,7 +19003,7 @@ dependencies = [ [[package]] name = "reth-trie-sparse" version = "2.5.2" -source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.8-fh3.1-2#b30d7e3cfdd3d56f736e1e37493bbcda60bed848" +source = "git+https://github.com/streamingfast/reth.git?tag=base-v2.5.2.11-fh3.1#9f9f5efa18eaa89b8ea23ed8c15c47b1a76bafb2" dependencies = [ "alloy-primitives", "alloy-trie", @@ -19895,15 +19831,6 @@ dependencies = [ "wait-timeout", ] -[[package]] -name = "ruzstd" -version = "0.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7c1c839d570d835527c9a5e4db7cb2198683a988cb9d7293fc8674e6bd58fc8" -dependencies = [ - "twox-hash", -] - [[package]] name = "rw-stream-sink" version = "0.4.0" @@ -20648,18 +20575,18 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" [[package]] name = "slop-air" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c6593a86d09bddebc8a9b894b528a5762fc775d2b83034a207c97630a7306e7f" +checksum = "42dc4b363ccac0f2017ae318e314f235e1141de4e101e80786de4e26c2d5ace9" dependencies = [ "p3-air", ] [[package]] name = "slop-algebra" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "733d477823f02646fa8025c5e2d4227ed806ca33a469def68994f63122f9f106" +checksum = "6b44536b0d3267951634cd7f768c3ba73e0b64e05b2ca4783616d4e54cbf324d" dependencies = [ "itertools 0.14.0", "p3-field", @@ -20668,9 +20595,9 @@ dependencies = [ [[package]] name = "slop-alloc" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "140e458c2ee44d8d061f417e19715af7ba7a1e0432b09566a9ad61b57419a425" +checksum = "59327ed65482fd7232e9e0ed1cc90f79068318f45e8d223208a601a01fb8fa84" dependencies = [ "serde", "slop-algebra", @@ -20679,9 +20606,9 @@ dependencies = [ [[package]] name = "slop-baby-bear" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b4ee5356138711b134dee8a730a7fd7514d55db4311f24c76113fd52eafb962d" +checksum = "b35ddf97b060a47957ab22b50910454a5dbd4f6b49f272f5400b9890142db0c6" dependencies = [ "lazy_static", "p3-baby-bear", @@ -20694,9 +20621,9 @@ dependencies = [ [[package]] name = "slop-basefold" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4da5c8d86bf73032dec5e9725c1e51ad03bdfa1cd3b4c5b5a66f940fe840d12" +checksum = "cb9816a01a646ffd5560b374e705842fca86c5a753827878b66f069cef67447d" dependencies = [ "derive-where", "itertools 0.14.0", @@ -20717,9 +20644,9 @@ dependencies = [ [[package]] name = "slop-basefold-prover" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4228cc21ec3ccb015bcd3d8627455a09ebecda7d934e927496ff3f239113af7" +checksum = "f0c34b125fc7670eb925883924067ae1ef5f123d0c42cd7ebe2106cb31bb1790" dependencies = [ "derive-where", "itertools 0.14.0", @@ -20743,9 +20670,9 @@ dependencies = [ [[package]] name = "slop-bn254" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0207095beb89a56e3875a0ca43e5782a2efc8637124b43ce453df42410e12a4c" +checksum = "ae41b4d69a3cd4848f91f21e768cf4e623529dab91b0eb85d0fe80455c7a0d4e" dependencies = [ "ff 0.13.1", "p3-bn254-fr", @@ -20758,9 +20685,9 @@ dependencies = [ [[package]] name = "slop-challenger" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfc4121b9b84df835c8a6677a90dc9935562172a4a370758987407c793633334" +checksum = "aebbf4fde396e40eeb534981be75547ab89b4a348c4136961627c06d0cbcd51f" dependencies = [ "p3-challenger", "serde", @@ -20770,9 +20697,9 @@ dependencies = [ [[package]] name = "slop-commit" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ebb108e80118a5df3aa7fe8185d32f83bf01f9fa6493f847180b6821c84044cc" +checksum = "0aef0897bdd3aef3b7f139b9cb15dd33beb4e467dcc0463f530400208ad9c89c" dependencies = [ "p3-commit", "serde", @@ -20781,9 +20708,9 @@ dependencies = [ [[package]] name = "slop-dft" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "967949400f2bc80c6d5cafe93601b4fcc018724af345a6926305172c563f5c72" +checksum = "90d48f0b85d59371b587ee614e42740f0157320417738f33a7e626c0d3c342cd" dependencies = [ "p3-dft", "serde", @@ -20795,18 +20722,18 @@ dependencies = [ [[package]] name = "slop-fri" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "068417681d6f0a727248a6a4e433caa4ab37e37e7a5c8b9e492676aca7b7f514" +checksum = "408f8bf591fcdac205c9c86ec5add57395a7d643c637715a9776f6d50a2bc273" dependencies = [ "p3-fri", ] [[package]] name = "slop-futures" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "121bb5d17da3e897dc1566e53da1f7cdbb06edda23ed70cb8a6d75ecb8e24f4a" +checksum = "964c2323823022979fff449d5e6b406a42c5d0e54b2eb0a200c7311426104321" dependencies = [ "crossbeam", "futures", @@ -20819,9 +20746,9 @@ dependencies = [ [[package]] name = "slop-jagged" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c61650d72b5489fd7f8d2a85f992e640d42be93738a1297dd395eaa1943434d4" +checksum = "b1cb6126061ed2509d2466fbdab6b74223de1339efefa1c57856c300eac6051c" dependencies = [ "itertools 0.14.0", "num_cpus", @@ -20850,18 +20777,18 @@ dependencies = [ [[package]] name = "slop-keccak-air" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "65f349d2b3ef84b275abc8b2729a4418a61b376deef4e7f23ec5ef5a55e0bd8f" +checksum = "412d09928af74ea4fef23e30614e75bb945b7ba2d31ea5e7f57c1e8a524f4772" dependencies = [ "p3-keccak-air", ] [[package]] name = "slop-koala-bear" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b27fec4aae6811dbf6a384cc7993e74621aede443356ef67c301c19c1f302865" +checksum = "627eb3ed4c0918f2dd2e70d2a1d312049ec3d2aa57fa0f7dbe2c3d9cd94993e4" dependencies = [ "lazy_static", "p3-koala-bear", @@ -20874,27 +20801,27 @@ dependencies = [ [[package]] name = "slop-matrix" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b668966ea5646cfc06a1eb6f637c8c6f338daca2dbb653c9a189ef5af0a831b8" +checksum = "625c7de1ad72d33d2b9d5941807da232fad6584636a62c50610730c5919cde8a" dependencies = [ "p3-matrix", ] [[package]] name = "slop-maybe-rayon" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b8cf2b701f28e73aa2ebbb7a416d1e28725f58a8ad7d8b8db386dd60bd2f5aa" +checksum = "811861d48788e78d342723679b70a1bb329c7d813692e970ca46522f46887e36" dependencies = [ "p3-maybe-rayon", ] [[package]] name = "slop-merkle-tree" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "58ce5396280532acb12643e11b0f9d662c915dd9d58bf5ed4c80fdda46a2797a" +checksum = "bbd70d6774da92caf2253dda01f10cfb2259335d5c478cac9b2401ce68edb71c" dependencies = [ "itertools 0.14.0", "p3-merkle-tree", @@ -20916,9 +20843,9 @@ dependencies = [ [[package]] name = "slop-multilinear" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c973e169c7c9584ec77c0a281b4a29ec20de9c3968796bbcea6002547cedabe2" +checksum = "7e019371d6e5fe1875045f41d4c1f7faf933c2056720a8564300975dcf7569f8" dependencies = [ "derive-where", "num_cpus", @@ -20935,24 +20862,24 @@ dependencies = [ [[package]] name = "slop-poseidon2" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13388d3c92bac6214ef7585869b968cf895d2715ebbb8b1a0e1464dd2ada30b2" +checksum = "51676e05a2891b15cd624b74c8aaa03cc10d5eb51542eeab5019294ad47c21fb" dependencies = [ "p3-poseidon2", ] [[package]] name = "slop-primitives" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93f569cac42b24e672f6b06ffde272d4086c2c2620f16d31c1d1ce88cddff681" +checksum = "91992a8acf6a9a4c52dee00142840d30c8cc9bbb3ff30ca3d56dcde428a591dd" [[package]] name = "slop-stacked" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc9b17529d7484d1c8d7a869db83c5aa741d8bfdfa9ed7903fc9c028410f4503" +checksum = "b671daca6c4c90d45cd67ecf1dc31fbada8330c7e71c452a3e4c06f9abee962e" dependencies = [ "derive-where", "itertools 0.14.0", @@ -20971,9 +20898,9 @@ dependencies = [ [[package]] name = "slop-sumcheck" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d87410940ec3d14912b44a86a420ea2973a94e805b9e318fb127cd5ee4d6de3" +checksum = "a73ecbade933db008a6833b9ca1e6481b3a8a983aa3b7052699727056519132a" dependencies = [ "itertools 0.14.0", "rayon", @@ -20988,18 +20915,18 @@ dependencies = [ [[package]] name = "slop-symmetric" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "693b50b55c3f5a2bb20dd3b975da79b80e4bf15b839b90f82e5dc7ba7262d3dc" +checksum = "9cb116ad23028ab41fe5228905a17bbeaf7b5fb7ec6790163bb8e8fef7092ffc" dependencies = [ "p3-symmetric", ] [[package]] name = "slop-tensor" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3582c1eee176cbcc4d42dbfff1a8aea254c48ca907d0d1687729822491756a48" +checksum = "6ef56dac9fcd661cfe9dd85557ac6ec0b2b04148ae510df734843ca3d133f2a0" dependencies = [ "arrayvec", "derive-where", @@ -21016,18 +20943,18 @@ dependencies = [ [[package]] name = "slop-uni-stark" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4afe09c7813c51c06d5a6df9a6e03590e3149db853526d3e110925a950df6232" +checksum = "aca7893516f6f62dcd8560e341f7d921ca7ed0b499ea8589d90331069a542a6e" dependencies = [ "p3-uni-stark", ] [[package]] name = "slop-utils" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "085f402d3f3ce1c843fdabf9d750e266e982186f3701fc56ce204fbf06bf18f0" +checksum = "dd4f9dab0f96b5352dee9ef86b6c39a423ecc69dbe22f0d5baf52e4cf97d43c5" dependencies = [ "p3-util", "tracing-forest", @@ -21036,9 +20963,9 @@ dependencies = [ [[package]] name = "slop-whir" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b7f6246ea3143f45dd48c224452943a0ca992a4c4766b537522b5f1eeef07ea" +checksum = "bdc0b0189eb12c209869d51a92781dfa2dea247484f60c1bc3d1ec1e90996b6e" dependencies = [ "derive-where", "itertools 0.14.0", @@ -21160,9 +21087,9 @@ dependencies = [ [[package]] name = "sp1-build" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40672b22ed12ff30619b043c1cbb2290f51261a1862e21a5a96270efda3583bd" +checksum = "43f02e6fc9d7ee6413f924de5a18ad32710f9fd88927a1162261ef32f70fcbb1" dependencies = [ "anyhow", "cargo_metadata", @@ -21174,8 +21101,8 @@ dependencies = [ [[package]] name = "sp1-cluster-artifact" -version = "2.8.4" -source = "git+https://github.com/succinctlabs/sp1-cluster?tag=v2.8.4#e878e6d5d03613de39ab574d6dbeef0b76f0dfab" +version = "2.9.0" +source = "git+https://github.com/succinctlabs/sp1-cluster?tag=v2.9.0#3249344d523f148438c50bef9b42a6cbd9ffb032" dependencies = [ "anyhow", "async-scoped", @@ -21200,13 +21127,14 @@ dependencies = [ [[package]] name = "sp1-cluster-common" -version = "2.8.4" -source = "git+https://github.com/succinctlabs/sp1-cluster?tag=v2.8.4#e878e6d5d03613de39ab574d6dbeef0b76f0dfab" +version = "2.9.0" +source = "git+https://github.com/succinctlabs/sp1-cluster?tag=v2.9.0#3249344d523f148438c50bef9b42a6cbd9ffb032" dependencies = [ "backoff", "chrono", "eyre", "futures", + "http 1.4.2", "if-addrs 0.13.4", "lazy_static", "log", @@ -21214,16 +21142,18 @@ dependencies = [ "opentelemetry-appender-tracing 0.4.0", "opentelemetry-otlp 0.16.0", "opentelemetry_sdk 0.23.0", - "prost 0.13.5", "rustls 0.23.43", "serde", "serde_json", "sp1-prover-types", + "subtle", "tokio", "tokio-blocked", "tokio-util", - "tonic 0.12.3", - "tonic-build", + "tonic 0.14.6", + "tonic-web", + "tower 0.5.3", + "tower-http", "tracing", "tracing-opentelemetry 0.24.0", "tracing-subscriber", @@ -21231,8 +21161,8 @@ dependencies = [ [[package]] name = "sp1-cluster-utils" -version = "2.8.4" -source = "git+https://github.com/succinctlabs/sp1-cluster?tag=v2.8.4#e878e6d5d03613de39ab574d6dbeef0b76f0dfab" +version = "2.9.0" +source = "git+https://github.com/succinctlabs/sp1-cluster?tag=v2.9.0#3249344d523f148438c50bef9b42a6cbd9ffb032" dependencies = [ "eyre", "sp1-cluster-artifact", @@ -21245,9 +21175,9 @@ dependencies = [ [[package]] name = "sp1-core-executor" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f45ca924caf27ceee7b00adfd4b3858c2cbaa1f394fb00cb4e584a62223d8d02" +checksum = "7e7704b5634746b510c614e377fffcb94c077a83e5b612b9a23ed9224f8aefee" dependencies = [ "bincode 1.3.3", "cfg-if", @@ -21286,9 +21216,9 @@ dependencies = [ [[package]] name = "sp1-core-executor-runner" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "20d74373eee66ec16ceb1e9aaa28d9d703210f13a3f1fe6b514c194df5b246f9" +checksum = "32945178fe9179e404de98f9408ba1153ecc4a6ccf21242a3767b6641e711408" dependencies = [ "base64 0.22.1", "bincode 1.3.3", @@ -21308,9 +21238,9 @@ dependencies = [ [[package]] name = "sp1-core-executor-runner-binary" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "211f1b73314b13fe4da22fe91d9a1a77d64df6be3172132f96c94aa0019babdf" +checksum = "10dfc42bdc4990b5069bf3911489a3f8d75c6a53b4d5befb012134c30138c484" dependencies = [ "bincode 1.3.3", "crash-handler", @@ -21323,9 +21253,9 @@ dependencies = [ [[package]] name = "sp1-core-machine" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13e440279541a50a1e166ea008b70c0d30a64830418b829d9e36b920eea1eb7c" +checksum = "dac96c5bb29221231f639d431a549a6ee587a585da565fcfd2399718fb22dc21" dependencies = [ "bincode 1.3.3", "cfg-if", @@ -21367,9 +21297,9 @@ dependencies = [ [[package]] name = "sp1-curves" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac61970510b3b965a58935e986e0dd04815f82247966c5823dbe3cc451db6e8f" +checksum = "37ea1b7850db9c947af190057e76f42bf0a2786380bbb702092b38ea924ed4e6" dependencies = [ "cfg-if", "curve25519-dalek", @@ -21390,9 +21320,9 @@ dependencies = [ [[package]] name = "sp1-derive" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7cbe5112430ab736c540513cf18d7e4e69d715ac347b90cefb45c4280f5da880" +checksum = "8fce1a95bd3a435c374e634a1a8ee5f2c54e649f0f7393ff3d959a7a75fc390c" dependencies = [ "proc-macro2", "quote", @@ -21401,9 +21331,9 @@ dependencies = [ [[package]] name = "sp1-hypercube" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "422e4ae52c18e39f07a7f38b4593c4b002339193dc6d9f51e993b5cc2a5718df" +checksum = "5184ecc088169a9014236d1ccb4233c5cf69c80eef00e85c719bd727dd091821" dependencies = [ "arrayref", "deepsize2", @@ -21448,9 +21378,9 @@ dependencies = [ [[package]] name = "sp1-jit" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d0fc35717922ed7133f18a48c2cf257c31a3c09190e7951239b0fad2a1f8632" +checksum = "64fdd42d05e3a31056df3665a32caee76553b58df924ae2e4bb4c26c2581e44f" dependencies = [ "dynasmrt", "hashbrown 0.14.5", @@ -21465,20 +21395,21 @@ dependencies = [ [[package]] name = "sp1-lib" -version = "6.3.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "629b673d6aa3c666b41e14d699323413fa90195c906365ba8c49d1b8e4531151" +checksum = "e8a36aae1b35d6eae35b479d23c818bb32c22e7c0e34cc1e1bc7c68319c94368" dependencies = [ "bincode 1.3.3", + "elliptic-curve 0.13.8", "serde", "sp1-primitives", ] [[package]] name = "sp1-primitives" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d387e4babbee68d30774ef52aad88aad3607eda8145a0badfa823e1759187c0c" +checksum = "f2ad891be0e48ab9ca110486f122cf799a707e5ca327a75669cfc2792b086882" dependencies = [ "bincode 1.3.3", "blake3", @@ -21500,9 +21431,9 @@ dependencies = [ [[package]] name = "sp1-prover" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2f0f1d58a3144090a2250f48d4743e3319e861a6537d3262d848380e2d284e59" +checksum = "4f04756def83f48b4c769eaab2910a706ec96d880ef07c3071c5ea69ec2f8238" dependencies = [ "anyhow", "bincode 1.3.3", @@ -21516,7 +21447,7 @@ dependencies = [ "hex", "indicatif 0.17.11", "itertools 0.14.0", - "lru 0.12.5", + "lru 0.18.5", "mti", "num-bigint 0.4.8", "opentelemetry 0.23.0", @@ -21550,16 +21481,16 @@ dependencies = [ "tempfile", "thiserror 1.0.69", "tokio", - "tonic 0.12.3", + "tonic 0.14.6", "tracing", "tracing-subscriber", ] [[package]] name = "sp1-prover-types" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9506d53b429a903c028517d325b12153a06015c7a0d698f9bde41b9de2f33072" +checksum = "fb4f10717325bf407bc960ef33b7379dd12635533341d06445ca8f9c077b5277" dependencies = [ "anyhow", "async-scoped", @@ -21568,22 +21499,23 @@ dependencies = [ "futures-util", "hashbrown 0.14.5", "mti", - "prost 0.13.5", + "prost 0.14.4", "serde", "sp1-core-machine", "sp1-hypercube", "sp1-primitives", "tokio", - "tonic 0.12.3", - "tonic-build", + "tonic 0.14.6", + "tonic-prost", + "tonic-prost-build", "tracing", ] [[package]] name = "sp1-recursion-circuit" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "691d4940e635589ef3bc77e82d883c02c93cd6361adde60f5e75715cb3e97e99" +checksum = "7baade5b429f1c6024986ad0dfd21e19de07c1fd11ae3a1c81cee6534bfd1092" dependencies = [ "bincode 1.3.3", "itertools 0.14.0", @@ -21621,9 +21553,9 @@ dependencies = [ [[package]] name = "sp1-recursion-compiler" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b4e7e8bd9f63df9cbf683fe148ccc1fbe645b211470e470ac7f3b93794064f4d" +checksum = "31ed9d346c90880398f8d6cab68bb6d0c252bb8d210fb70a34f704452a58427a" dependencies = [ "backtrace", "cfg-if", @@ -21642,9 +21574,9 @@ dependencies = [ [[package]] name = "sp1-recursion-executor" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "115fa5ab8ff56deb955244f7525628a99b8d951883fb390678ea5bbd0c74f893" +checksum = "a75e397541774b5c35ec963aec113797346765614dc131d5aec3eaec38780833" dependencies = [ "backtrace", "cfg-if", @@ -21666,9 +21598,9 @@ dependencies = [ [[package]] name = "sp1-recursion-gnark-ffi" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "365c2d0b7da67108703dfd9c8f897e28ed24c6460f5af1ce5c7077222303421d" +checksum = "e56340adcfa47da2f24dde5da6b6bfd25bb3a80b3c9deb3b8ad6a518effa6b7f" dependencies = [ "anyhow", "bincode 1.3.3", @@ -21690,9 +21622,9 @@ dependencies = [ [[package]] name = "sp1-recursion-machine" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2978f9522abc4e53e55e58a2bd3b5ce812e0e275dce705b55c4c451648c083ea" +checksum = "b0bb544ab428659976308a875b5863d2417b0a77a86dd6f963d2571372ba4b43" dependencies = [ "itertools 0.14.0", "rand 0.8.7", @@ -21712,9 +21644,9 @@ dependencies = [ [[package]] name = "sp1-sdk" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b2e5996edef461e0e7351ed0312423e12b78a2f68d62eeb3aae1a535dca30644" +checksum = "699ee2e78537efab20f76aa2049058aa5fa803c804cc6cd4bce25a50ee59c341" dependencies = [ "alloy-primitives", "alloy-signer 1.8.3", @@ -21733,7 +21665,7 @@ dependencies = [ "itertools 0.14.0", "k256 0.13.4", "num-bigint 0.4.8", - "prost 0.13.5", + "prost 0.14.4", "reqwest 0.12.28", "reqwest-middleware", "rustls 0.23.43", @@ -21752,17 +21684,17 @@ dependencies = [ "tempfile", "thiserror 1.0.69", "tokio", - "tonic 0.12.3", + "tonic 0.14.6", + "tonic-prost", "tracing", - "twirp-rs", "zstd", ] [[package]] name = "sp1-verifier" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aca7223cc7a77e42536c7229c8016672737aab07f6149097d9c7a26324b29814" +checksum = "c4afe3f5047f5cff291b41ef80f9815c14b308df399c1c61809b0e7ce6ac7cd6" dependencies = [ "bincode 1.3.3", "blake3", @@ -21786,9 +21718,9 @@ dependencies = [ [[package]] name = "sp1_bls12_381" -version = "0.8.0-sp1-6.0.0" +version = "0.8.0-sp1-6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f23e41cd36168cc2e51e5d3e35ff0c34b204d945769a65591a76286d04b51e43" +checksum = "9b2ad1e32db3c05d37e4385e3e48795f14d687a5797bf2eee0a7753d7eaa9661" dependencies = [ "cfg-if", "ff 0.13.1", @@ -22225,9 +22157,9 @@ dependencies = [ [[package]] name = "substrate-bn-succinct-rs" -version = "0.6.0" +version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a241fd7c1016fb8ad30fcf5a20986c0c4538e8f15a1b41a1761516299e377ec1" +checksum = "bc5ad6ccb8ae10730213d4d5bdb0a9c4ddf351210e7c6ad9c66c02f2ad0b4256" dependencies = [ "bytemuck", "byteorder", @@ -22237,6 +22169,7 @@ dependencies = [ "num-bigint 0.4.8", "rand 0.8.7", "rustc-hex", + "sp1-lib", ] [[package]] @@ -22785,9 +22718,9 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" [[package]] name = "tokio" -version = "1.53.1" +version = "1.54.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed" +checksum = "b52fd3255258d26fac7264ab5a76b53c13367136dc011d3d6ad4f8ed070396a3" dependencies = [ "bytes", "libc", @@ -23093,40 +23026,6 @@ dependencies = [ "tracing", ] -[[package]] -name = "tonic" -version = "0.12.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877c5b330756d856ffcc4553ab34a5684481ade925ecc54bcd1bf02b1d0d4d52" -dependencies = [ - "async-stream", - "async-trait", - "axum 0.7.9", - "base64 0.22.1", - "bytes", - "h2 0.4.15", - "http 1.4.2", - "http-body 1.1.0", - "http-body-util", - "hyper 1.11.0", - "hyper-timeout 0.5.2", - "hyper-util", - "percent-encoding", - "pin-project", - "prost 0.13.5", - "rustls-native-certs", - "rustls-pemfile", - "socket2 0.5.10", - "tokio", - "tokio-rustls 0.26.4", - "tokio-stream", - "tower 0.4.13", - "tower-layer", - "tower-service", - "tracing", - "webpki-roots 0.26.11", -] - [[package]] name = "tonic" version = "0.14.6" @@ -23156,18 +23055,17 @@ dependencies = [ "tower-layer", "tower-service", "tracing", + "webpki-roots 1.0.9", ] [[package]] name = "tonic-build" -version = "0.12.3" +version = "0.14.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9557ce109ea773b399c9b9e5dca39294110b74f1f342cb347a80d1fce8c26a11" +checksum = "c68f61875ac5293cf72e6c8cf0158086428c82c37229e98c840878f1706b0322" dependencies = [ "prettyplease", "proc-macro2", - "prost-build", - "prost-types 0.13.5", "quote", "syn 2.0.119", ] @@ -23183,6 +23081,22 @@ dependencies = [ "tonic 0.14.6", ] +[[package]] +name = "tonic-prost-build" +version = "0.14.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "654e5643eff75d7f8c99197ce1440ed19a3474eada74c12bbac488b2cafdae27" +dependencies = [ + "prettyplease", + "proc-macro2", + "prost-build", + "prost-types", + "quote", + "syn 2.0.119", + "tempfile", + "tonic-build", +] + [[package]] name = "tonic-types" version = "0.14.6" @@ -23190,10 +23104,28 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "73ab1b02061f83d519bba3caa167f88f261ef05720ab8ebc954ade70de3348e8" dependencies = [ "prost 0.14.4", - "prost-types 0.14.4", + "prost-types", "tonic 0.14.6", ] +[[package]] +name = "tonic-web" +version = "0.14.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b5e6a1b6319ca4b61a4c0f0c94d439c8f3ed344cca56fe0df40e1fe4be11380b" +dependencies = [ + "base64 0.22.1", + "bytes", + "http 1.4.2", + "http-body 1.1.0", + "pin-project", + "tokio-stream", + "tonic 0.14.6", + "tower-layer", + "tower-service", + "tracing", +] + [[package]] name = "tower" version = "0.4.13" @@ -23622,34 +23554,6 @@ dependencies = [ "thiserror 2.0.19", ] -[[package]] -name = "twirp-rs" -version = "0.13.0-succinct" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "27dfcc06b8d9262bc2d4b8d1847c56af9971a52dd8a0076876de9db763227d0d" -dependencies = [ - "async-trait", - "axum 0.7.9", - "futures", - "http 1.4.2", - "http-body-util", - "hyper 1.11.0", - "prost 0.13.5", - "reqwest 0.12.28", - "serde", - "serde_json", - "thiserror 1.0.69", - "tokio", - "tower 0.5.3", - "url", -] - -[[package]] -name = "twox-hash" -version = "2.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8464ec13c3691491391d9fce00f6416c9a48e46972f72d7865688be2080192c9" - [[package]] name = "typeid" version = "1.0.3" @@ -24265,7 +24169,7 @@ dependencies = [ [[package]] name = "websocket-proxy" -version = "1.5.1" +version = "1.5.4" dependencies = [ "axum 0.8.9", "backoff", @@ -24287,7 +24191,7 @@ dependencies = [ [[package]] name = "websocket-proxy-bin" -version = "1.5.1" +version = "1.5.4" dependencies = [ "axum 0.8.9", "base-cli-utils", diff --git a/Cargo.toml b/Cargo.toml index dc6abd8f8b..992108faa1 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,5 +1,5 @@ [workspace.package] -version = "1.5.1" +version = "1.5.4" edition = "2024" rust-version = "1.96" license = "MIT" @@ -349,77 +349,77 @@ alloy-network-primitives = { version = "2.3.0", default-features = false } reth-zstd-compressors = { version = "0.6.0", default-features = false } reth-primitives-traits = { version = "0.6.0", default-features = false } reth-codecs = { version = "0.6.0", default-features = false, features = ["alloy"] } -reth-db = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-cli = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-ipc = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-evm = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-rpc = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-revm = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-errors = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-trie = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-exex = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-tasks = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-ecies = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-db-api = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-discv4 = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-discv5 = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-trie-db = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-rpc-api = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-network = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-net-nat = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-tracing = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-eth-wire = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-provider = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-cli-util = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-node-api = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-engine-primitives = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-db-models = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-db-common = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-rpc-layer = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-node-core = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-consensus = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-chainspec = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-cli-runner = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-tokio-util = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-rpc-convert = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-network-api = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-network-p2p = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-storage-api = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-rpc-eth-api = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-chain-state = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-trie-common = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-engine-tree = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-payload-util = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-node-builder = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-node-metrics = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-cli-commands = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-evm-ethereum = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-tracing-otlp = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-testing-utils = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-rpc-eth-types = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-network-peers = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-node-ethereum = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-rpc-engine-api = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-e2e-test-utils = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-eth-wire-types = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-payload-builder = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-execution-types = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-execution-cache = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-exex-test-utils = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-rpc-server-types = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-transaction-pool = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-execution-errors = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-payload-validator = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-payload-primitives = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-ethereum-primitives = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-basic-payload-builder = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-payload-builder-primitives = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } -reth-prune-types = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8", default-features = false } -reth-stages-types = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8", default-features = false } -reth-storage-errors = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8", default-features = false } -reth-ethereum-forks = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8", default-features = false } -reth-consensus-common = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8", default-features = false } -reth-trie-parallel = { git = "https://github.com/base/reth", tag = "base-v2.5.2.8" } +reth-db = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-cli = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-ipc = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-evm = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-rpc = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-revm = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-errors = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-trie = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-exex = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-tasks = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-ecies = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-db-api = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-discv4 = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-discv5 = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-trie-db = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-rpc-api = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-network = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-net-nat = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-tracing = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-eth-wire = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-provider = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-cli-util = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-node-api = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-engine-primitives = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-db-models = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-db-common = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-rpc-layer = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-node-core = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-consensus = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-chainspec = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-cli-runner = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-tokio-util = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-rpc-convert = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-network-api = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-network-p2p = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-storage-api = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-rpc-eth-api = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-chain-state = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-trie-common = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-engine-tree = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-payload-util = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-node-builder = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-node-metrics = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-cli-commands = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-evm-ethereum = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-tracing-otlp = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-testing-utils = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-rpc-eth-types = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-network-peers = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-node-ethereum = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-rpc-engine-api = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-e2e-test-utils = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-eth-wire-types = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-payload-builder = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-execution-types = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-execution-cache = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-exex-test-utils = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-rpc-server-types = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-transaction-pool = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-execution-errors = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-payload-validator = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-payload-primitives = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-ethereum-primitives = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-basic-payload-builder = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-payload-builder-primitives = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } +reth-prune-types = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11", default-features = false } +reth-stages-types = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11", default-features = false } +reth-storage-errors = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11", default-features = false } +reth-ethereum-forks = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11", default-features = false } +reth-consensus-common = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11", default-features = false } +reth-trie-parallel = { git = "https://github.com/base/reth", tag = "base-v2.5.2.11" } # tokio tokio = "1.53.1" @@ -476,12 +476,12 @@ aws-sdk-ec2 = { version = "1.217.0", default-features = false } aws-credential-types = { version = "1.1.7", default-features = false } aws-sdk-elasticloadbalancingv2 = { version = "1.109.0", default-features = false } -# SP1 v6.8.1 (Hypercube) — used by crates/proof/zk -sp1-prover-types = "=6.8.1" -sp1-sdk = { version = "=6.8.1" } -sp1-cluster-utils = { git = "https://github.com/succinctlabs/sp1-cluster", tag = "v2.8.4" } -sp1-cluster-common = { git = "https://github.com/succinctlabs/sp1-cluster", tag = "v2.8.4" } -sp1-cluster-artifact = { git = "https://github.com/succinctlabs/sp1-cluster", tag = "v2.8.4" } +# SP1 v6.10.0 (Hypercube) — used by crates/proof/zk +sp1-prover-types = "=6.10.0" +sp1-sdk = { version = "=6.10.0" } +sp1-cluster-utils = { git = "https://github.com/succinctlabs/sp1-cluster", tag = "v2.9.0" } +sp1-cluster-common = { git = "https://github.com/succinctlabs/sp1-cluster", tag = "v2.9.0" } +sp1-cluster-artifact = { git = "https://github.com/succinctlabs/sp1-cluster", tag = "v2.9.0" } # crypto sha3 = "0.10" @@ -496,7 +496,7 @@ hex-literal = "1.1" x509-parser = "0.17" sha2 = { version = "0.10", default-features = false } c-kzg = { version = "2.1.5", default-features = false } -kzg-rs = { version = "0.2.8", default-features = false } +kzg-rs = { version = "0.2.9", default-features = false } ark-ff = { version = "0.6.0", default-features = false } ark-bls12-381 = { version = "0.6.0", default-features = false } @@ -640,8 +640,8 @@ notify = "8.2" governor = "0.7" # Firehose tracer (defined by the streamingfast reth fork, release/base-2.x branch). -reth-firehose = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-firehose-tests = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } +reth-firehose = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-firehose-tests = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } firehose-tracer = "5.7.0" # Shared tracing-regression framework (capture / invariants / projection / golden). @@ -656,74 +656,74 @@ firehose-tracer-prestate = "5.7.0" alloy-evm = { git = "https://github.com/streamingfast/evm.git", tag = "v0.38.0-sf" } [patch."https://github.com/base/reth"] -reth-basic-payload-builder = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-chain-state = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-chainspec = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-cli = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-cli-commands = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-cli-runner = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-cli-util = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-consensus = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-consensus-common = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-db = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-db-api = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-db-common = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-db-models = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-discv4 = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-discv5 = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-e2e-test-utils = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-ecies = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-engine-primitives = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-engine-tree = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-errors = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-eth-wire = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-eth-wire-types = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-ethereum-forks = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-ethereum-primitives = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-evm = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-evm-ethereum = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-execution-cache = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-execution-errors = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-execution-types = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-exex = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-exex-test-utils = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-ipc = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-net-nat = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-network = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-network-api = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-network-p2p = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-network-peers = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-node-api = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-node-builder = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-node-core = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-node-ethereum = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-node-metrics = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-payload-builder = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-payload-builder-primitives = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-payload-primitives = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-payload-util = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-payload-validator = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-provider = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-prune-types = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-revm = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-rpc = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-rpc-api = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-rpc-convert = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-rpc-engine-api = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-rpc-eth-api = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-rpc-eth-types = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-rpc-layer = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-rpc-server-types = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-stages-types = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-storage-api = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-storage-errors = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-tasks = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-testing-utils = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-tokio-util = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-tracing = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-tracing-otlp = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-transaction-pool = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-trie = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-trie-common = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-trie-db = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } -reth-trie-parallel = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.8-fh3.1-2" } +reth-basic-payload-builder = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-chain-state = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-chainspec = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-cli = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-cli-commands = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-cli-runner = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-cli-util = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-consensus = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-consensus-common = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-db = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-db-api = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-db-common = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-db-models = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-discv4 = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-discv5 = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-e2e-test-utils = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-ecies = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-engine-primitives = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-engine-tree = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-errors = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-eth-wire = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-eth-wire-types = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-ethereum-forks = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-ethereum-primitives = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-evm = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-evm-ethereum = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-execution-cache = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-execution-errors = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-execution-types = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-exex = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-exex-test-utils = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-ipc = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-net-nat = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-network = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-network-api = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-network-p2p = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-network-peers = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-node-api = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-node-builder = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-node-core = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-node-ethereum = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-node-metrics = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-payload-builder = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-payload-builder-primitives = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-payload-primitives = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-payload-util = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-payload-validator = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-provider = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-prune-types = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-revm = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-rpc = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-rpc-api = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-rpc-convert = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-rpc-engine-api = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-rpc-eth-api = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-rpc-eth-types = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-rpc-layer = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-rpc-server-types = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-stages-types = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-storage-api = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-storage-errors = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-tasks = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-testing-utils = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-tokio-util = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-tracing = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-tracing-otlp = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-transaction-pool = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-trie = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-trie-common = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-trie-db = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } +reth-trie-parallel = { git = "https://github.com/streamingfast/reth.git", tag = "base-v2.5.2.11-fh3.1" } diff --git a/actions/harness/tests/shadow_sequencer.rs b/actions/harness/tests/shadow_sequencer.rs index 03dc968a6e..40713fbc72 100644 --- a/actions/harness/tests/shadow_sequencer.rs +++ b/actions/harness/tests/shadow_sequencer.rs @@ -57,7 +57,8 @@ async fn late_shadow_catches_up_safe_then_unsafe() { assert_eq!(shadow.run_until_idle().await, SAFE_BLOCKS); assert_eq!(shadow.l2_safe_number(), SAFE_BLOCKS as u64); - let payloads = catchup.contiguous_payloads(shadow.l2_unsafe()); + let payloads = + catchup.contiguous_payloads(shadow.l2_unsafe(), CanonicalUnsafeCatchup::MAX_APPLY_BATCH); assert_eq!(payloads.len(), UNSAFE_BLOCKS); for payload in payloads { let block = ExecutionPayloadConverter::block_from_envelope(&payload) diff --git a/bin/base/src/commands/sequencer.rs b/bin/base/src/commands/sequencer.rs index c9c5b61b36..de80342c23 100644 --- a/bin/base/src/commands/sequencer.rs +++ b/bin/base/src/commands/sequencer.rs @@ -183,8 +183,11 @@ impl SequencerCommand { #[cfg(test)] mod tests { + use std::sync::Arc; + + use base_builder_core::MeteringProvider; use base_consensus_cli::ConsensusNodeConfigArgs; - use clap::Parser; + use clap::{CommandFactory, Parser}; use crate::{cli::BaseCli, commands::BaseCommand}; @@ -272,6 +275,242 @@ mod tests { assert!(config.enabled); } + #[test] + fn parses_resource_metering_schedule_in_sequencer_command() { + let dir = tempfile::tempdir().expect("tempdir"); + let schedule_path = dir.path().join("schedule.json"); + std::fs::write( + &schedule_path, + r#"{ + "version": 1, + "dimensions": [{ + "name": "compute", + "blockLimit": 1000, + "transactionLimit": 500, + "baseGasWeight": 1, + "operations": [{"name": "SSTORE", "gasUsedWeight": 2, "countCost": 10}], + "dryRun": true + }] + }"#, + ) + .expect("write temp schedule"); + + let schedule_path_str = schedule_path.to_str().expect("utf-8 schedule path"); + let args = sequencer_args(&[ + "base", + "sequencer", + "--p2p.sequencer.key", + SEQUENCER_KEY, + "--builder.enable-resource-metering", + ]); + let cli = BaseCli::try_parse_from( + args.into_iter().chain(["--payload.resource-metering-schedule", schedule_path_str]), + ) + .expect("sequencer command should accept --payload.resource-metering-schedule"); + + let BaseCommand::Sequencer(sequencer) = cli.command else { + panic!("expected sequencer command"); + }; + + assert!(sequencer.builder.enable_resource_metering); + assert_eq!( + sequencer.builder.resource_metering_schedule.as_deref(), + Some(schedule_path.as_path()) + ); + + let metering_provider: base_builder_core::SharedMeteringProvider = + Arc::new(sequencer.builder.build_metering_store()); + let builder_config = sequencer + .builder + .into_builder_config(Arc::clone(&metering_provider)) + .expect("valid schedule should build active BuilderConfig"); + + assert!(builder_config.resource_metering.enabled); + assert!(builder_config.resource_metering.is_active()); + assert_eq!(builder_config.resource_metering.schedule.dimensions.len(), 1); + } + + #[test] + fn builder_and_sequencer_parse_identical_metering_flags_envs_and_validation() { + #[derive(clap::Parser)] + struct StandaloneBuilderCli { + #[command(flatten)] + args: base_builder_cli::Args, + } + let builder_cmd = StandaloneBuilderCli::command(); + let seq_cmd = BaseCli::command(); + let seq_sub = seq_cmd.find_subcommand("sequencer").expect("sequencer subcommand"); + for cmd in [&builder_cmd, seq_sub] { + let enable = cmd + .get_arguments() + .find(|a| a.get_long() == Some("builder.enable-resource-metering")) + .unwrap(); + assert_eq!( + enable.get_env(), + Some(std::ffi::OsStr::new("BUILDER_ENABLE_RESOURCE_METERING")) + ); + let sched = cmd + .get_arguments() + .find(|a| a.get_long() == Some("payload.resource-metering-schedule")) + .unwrap(); + assert_eq!( + sched.get_env(), + Some(std::ffi::OsStr::new("PAYLOAD_RESOURCE_METERING_SCHEDULE")) + ); + } + let dir = tempfile::tempdir().expect("tempdir"); + let valid_path = dir.path().join("valid.json"); + std::fs::write(&valid_path, r#"{"version":1,"dimensions":[{"name":"compute","blockLimit":1000,"transactionLimit":500,"baseGasWeight":1,"dryRun":true}]}"#).unwrap(); + let valid_str = valid_path.to_str().unwrap(); + let b_on = StandaloneBuilderCli::try_parse_from([ + "base-builder", + "--builder.enable-resource-metering", + "--payload.resource-metering-schedule", + valid_str, + ]) + .unwrap() + .args; + let BaseCommand::Sequencer(s_on) = BaseCli::try_parse_from( + sequencer_args(&[ + "base", + "sequencer", + "--p2p.sequencer.key", + SEQUENCER_KEY, + "--builder.enable-resource-metering", + ]) + .into_iter() + .chain(["--payload.resource-metering-schedule", valid_str]), + ) + .unwrap() + .command + else { + panic!() + }; + let b_store = Arc::new(b_on.build_metering_store()); + let s_store = Arc::new(s_on.builder.build_metering_store()); + assert!(b_store.is_enabled() && s_store.is_enabled()); + let b_cfg = b_on.into_builder_config(b_store).unwrap(); + let s_cfg = s_on.builder.into_builder_config(s_store).unwrap(); + assert!(b_cfg.resource_metering.is_active() && s_cfg.resource_metering.is_active()); + assert_eq!(b_cfg.resource_metering.schedule, s_cfg.resource_metering.schedule); + let no_dry_run = dir.path().join("missing-dry-run.json"); + std::fs::write(&no_dry_run, r#"{"version":1,"dimensions":[{"name":"compute","blockLimit":1000,"baseGasWeight":1}]}"#).unwrap(); + let no_dry_str = no_dry_run.to_str().unwrap(); + let b_bad = StandaloneBuilderCli::try_parse_from([ + "base-builder", + "--builder.enable-resource-metering", + "--payload.resource-metering-schedule", + no_dry_str, + ]) + .unwrap() + .args; + let BaseCommand::Sequencer(s_bad) = BaseCli::try_parse_from( + sequencer_args(&[ + "base", + "sequencer", + "--p2p.sequencer.key", + SEQUENCER_KEY, + "--builder.enable-resource-metering", + ]) + .into_iter() + .chain(["--payload.resource-metering-schedule", no_dry_str]), + ) + .unwrap() + .command + else { + panic!() + }; + let b_err = b_bad + .into_builder_config(Arc::new(base_builder_metering::MeteringStore::default())) + .unwrap_err() + .to_string(); + let s_err = s_bad + .builder + .into_builder_config(Arc::new(base_builder_metering::MeteringStore::default())) + .unwrap_err() + .to_string(); + assert_eq!(b_err, s_err); + assert!(b_err.contains(no_dry_str)); + } + + /// The unified binary's `sequencer` command flattens `base_builder_cli::Args` and hands the + /// resulting [`BuilderConfig`](base_builder_core::BuilderConfig) to + /// `MultiplexingServiceBuilder`, which owns the native payload builder's config. This drives + /// that whole chain from CLI arguments, including the inert cases. + #[test] + fn sequencer_schedule_wiring_feeds_the_multiplexed_native_payload_builder() { + use base_builder_multiplex::MultiplexingServiceBuilder; + + fn sequencer_config( + extra: &[&str], + schedule: Option<&str>, + ) -> (base_builder_core::BuilderConfig, base_builder_core::SharedMeteringProvider) { + let mut args = ["base", "sequencer", "--p2p.sequencer.key", SEQUENCER_KEY] + .map(str::to_string) + .to_vec(); + args.extend(extra.iter().map(|arg| (*arg).to_string())); + args.extend(REQUIRED_CONSENSUS_ARGS.iter().map(|arg| (*arg).to_string())); + if let Some(schedule) = schedule { + args.push("--payload.resource-metering-schedule".to_string()); + args.push(schedule.to_string()); + } + let cli = BaseCli::try_parse_from(args).expect("sequencer args should parse"); + let BaseCommand::Sequencer(sequencer) = cli.command else { + panic!("expected sequencer command"); + }; + let provider: base_builder_core::SharedMeteringProvider = + Arc::new(sequencer.builder.build_metering_store()); + let config = sequencer + .builder + .into_builder_config(Arc::clone(&provider)) + .expect("builder config should build"); + (config, provider) + } + + let dir = tempfile::tempdir().expect("tempdir"); + let schedule_path = dir.path().join("schedule.json"); + std::fs::write( + &schedule_path, + r#"{ + "version": 1, + "dimensions": [{ + "name": "compute", + "blockLimit": 1000, + "transactionLimit": 500, + "baseGasWeight": 1, + "operations": [{"name": "SSTORE", "gasUsedWeight": 2, "countCost": 10}], + "dryRun": true + }] + }"#, + ) + .expect("write schedule"); + let schedule_str = schedule_path.to_str().expect("utf-8 schedule path"); + + // Switch on with a schedule: the native payload config the multiplexer derives is active. + let (config, provider) = + sequencer_config(&["--builder.enable-resource-metering"], Some(schedule_str)); + assert!(config.resource_metering.enabled); + assert!(config.resource_metering.is_active()); + assert_eq!(config.resource_metering.schedule.dimensions[0].name, "compute"); + assert!(Arc::ptr_eq(&config.resource_metering.provider, &provider)); + let multiplexer = MultiplexingServiceBuilder::new(config); + assert!(multiplexer.builder_config.resource_metering.is_active()); + assert!(Arc::ptr_eq(&multiplexer.builder_config.metering_provider, &provider)); + + // Switch on without a schedule: Flashblocks metering runs, native budgets stay inert. + let (config, _) = sequencer_config(&["--builder.enable-resource-metering"], None); + assert!(config.resource_metering.enabled); + assert!(!config.resource_metering.is_active()); + assert!( + !MultiplexingServiceBuilder::new(config).builder_config.resource_metering.is_active() + ); + + // Switch off with a schedule: nothing is metered, on either builder. + let (config, _) = sequencer_config(&[], Some(schedule_str)); + assert!(!config.resource_metering.enabled); + assert!(!config.resource_metering.is_active()); + } + #[test] fn shadow_indexer_defaults_to_disabled() { let cli = BaseCli::parse_from(sequencer_args(&[ diff --git a/bin/base/src/main.rs b/bin/base/src/main.rs index ace5778268..2e0e41bfbd 100644 --- a/bin/base/src/main.rs +++ b/bin/base/src/main.rs @@ -18,6 +18,7 @@ fn main() { // Keep reth_info and network identity aligned with the Base binary version. base_reth_cli::init_reth!(); base_reth_cli::init_snapshots!(); + base_reth_cli::init_engine!(); base_cli_utils::run_cli_main!(cli::BaseCli); } diff --git a/bin/builder/src/main.rs b/bin/builder/src/main.rs index 4f7c3464f1..4cf0c34e80 100644 --- a/bin/builder/src/main.rs +++ b/bin/builder/src/main.rs @@ -31,6 +31,7 @@ fn main() { base_cli_utils::init_common!(); base_reth_cli::init_reth!(); base_reth_cli::init_snapshots!(); + base_reth_cli::init_engine!(); let cli = base_cli_utils::parse_cli!(BuilderCli); @@ -50,9 +51,7 @@ fn main() { let shadow_indexer_config = ShadowIndexerConfig::try_from(&builder_args.shadow_indexer)?; let payload_builder_cutover = builder_args.payload_builder_cutover; let basic_payload_builder = builder_args.basic_payload_builder; - let builder_config = builder_args - .into_builder_config(Arc::clone(&metering_provider)) - .expect("Failed to convert rollup args to builder config"); + let builder_config = builder_args.into_builder_config(Arc::clone(&metering_provider))?; let da_config = builder_config.da_config.clone(); let gas_limit_config = builder_config.gas_limit_config.clone(); let manifest_precheck_enabled = builder_config.manifest_precheck_enabled; diff --git a/bin/node/src/main.rs b/bin/node/src/main.rs index f00f29a5b2..35246df4d4 100644 --- a/bin/node/src/main.rs +++ b/bin/node/src/main.rs @@ -41,6 +41,7 @@ fn main() { base_cli_utils::init_common!(); base_reth_cli::init_reth!(); base_reth_cli::init_snapshots!(); + base_reth_cli::init_engine!(); firehose::init(); diff --git a/crates/batcher/comp/README.md b/crates/batcher/comp/README.md index cfa3ae9921..e510ff1e8b 100644 --- a/crates/batcher/comp/README.md +++ b/crates/batcher/comp/README.md @@ -5,9 +5,11 @@ Channel compression for Base. ## Overview `CompressionStream::append` incrementally compresses input and returns the new -stable output without flushing or recompressing earlier input. `finish` returns -the remaining suffix. Concatenating those bytes produces one protocol-compatible -channel, including Brotli's channel-version byte. +stable output without recompressing earlier input. With a flush interval, it +flushes the encoder each time that many input bytes have been appended, so the +output returned so far covers all but less than one interval of the input. +`finish` returns the remaining suffix. Concatenating those bytes produces one +protocol-compatible channel, including Brotli's channel-version byte. `BrotliLevel::compress_channel` is the one-shot API used by fixtures. `BrotliCompressor` exposes raw stateless Brotli output. diff --git a/crates/batcher/comp/src/stream.rs b/crates/batcher/comp/src/stream.rs index 84533f4bb2..68543dee00 100644 --- a/crates/batcher/comp/src/stream.rs +++ b/crates/batcher/comp/src/stream.rs @@ -1,6 +1,7 @@ //! Incremental derivation-channel compression. use alloc::{boxed::Box, vec::Vec}; +use core::num::NonZeroUsize; use std::io::Write; use brotli::{CompressorWriter, enc::BrotliEncoderMaxCompressedSize}; @@ -15,6 +16,10 @@ pub struct CompressionStream { compressor: Box>>, /// Total bytes returned by prior appends. output_size: usize, + /// Input bytes after which an append flushes the encoder, if any. + flush_interval: Option, + /// Input bytes appended since the last flush. + unflushed_input: usize, } impl CompressionStream { @@ -24,8 +29,18 @@ impl CompressionStream { /// Brotli sliding-window exponent (`2^22` bytes). const BROTLI_LGWIN: u32 = 22; - /// Creates an empty compressor at `level`. - pub fn new(level: BrotliLevel) -> Self { + /// Worst-case bytes one flush adds: it ends a meta-block early, which costs one + /// more meta-block header (4 bytes), and pads it to a byte boundary (2 bytes). + const FLUSH_OVERHEAD: usize = 6; + + /// Creates an empty compressor at `level` that flushes every `flush_interval` + /// input bytes, or never when `None`. + /// + /// Unflushed, Brotli holds up to about one window of input before emitting + /// anything, so [`output_size`](Self::output_size) can trail the real + /// compressed size by that much input. A flush keeps the window, so it costs + /// only a meta-block boundary. + pub fn new(level: BrotliLevel, flush_interval: Option) -> Self { let mut output = Vec::with_capacity(Self::BROTLI_BUFFER_SIZE); output.push(BrotliLevel::CHANNEL_VERSION); let compressor = CompressorWriter::new( @@ -34,12 +49,26 @@ impl CompressionStream { level.as_u32(), Self::BROTLI_LGWIN, ); - Self { compressor: Box::new(compressor), output_size: 0 } + Self { + compressor: Box::new(compressor), + output_size: 0, + flush_interval, + unflushed_input: 0, + } } /// Append input and return newly emitted compressed bytes. + /// + /// Flushes once the flush interval's worth of input has been appended since the + /// last flush, so the bytes returned so far always cover all but less than one + /// flush interval of the input. pub fn append(&mut self, input: &[u8]) -> Result, CompressionError> { self.compressor.write_all(input)?; + self.unflushed_input += input.len(); + if self.flush_interval.is_some_and(|interval| self.unflushed_input >= interval.get()) { + self.compressor.flush()?; + self.unflushed_input = 0; + } let output = core::mem::take(self.compressor.get_mut()); self.output_size += output.len(); Ok(output) @@ -52,8 +81,12 @@ impl CompressionStream { /// Returns a conservative upper bound for a finished stream of `input_size`. pub fn max_output_size(&self, input_size: usize) -> usize { - // Channel-version prefix byte. - BrotliEncoderMaxCompressedSize(input_size).saturating_add(1) + // Each flush follows at least one flush interval of new input. + let max_flushes = self.flush_interval.map_or(0, |interval| input_size / interval); + BrotliEncoderMaxCompressedSize(input_size) + .saturating_add(max_flushes.saturating_mul(Self::FLUSH_OVERHEAD)) + // Channel-version prefix byte. + .saturating_add(1) } /// Finishes the stream and returns compressed bytes not previously transferred. @@ -75,7 +108,7 @@ mod tests { #[test] fn brotli_roundtrip_across_appends() { let expected = CHUNKS.concat(); - let mut compressor = CompressionStream::new(BrotliLevel::Brotli10); + let mut compressor = CompressionStream::new(BrotliLevel::Brotli10, None); let mut compressed = Vec::new(); for chunk in CHUNKS { @@ -94,14 +127,14 @@ mod tests { fn chunking_does_not_change_the_compressed_stream() { let input = CHUNKS.concat(); let level = BrotliLevel::Brotli10; - let mut chunked = CompressionStream::new(level); + let mut chunked = CompressionStream::new(level, None); let mut chunked_output = Vec::new(); for chunk in CHUNKS { chunked_output.extend(chunked.append(chunk).unwrap()); } chunked_output.extend(chunked.finish().unwrap()); - let mut single = CompressionStream::new(level); + let mut single = CompressionStream::new(level, None); let mut single_output = single.append(&input).unwrap(); single_output.extend(single.finish().unwrap()); @@ -111,7 +144,7 @@ mod tests { #[test] fn reported_bounds_cover_incompressible_streams() { let mut state = 1u64; - let input: Vec = (0..100_000) + let input: Vec = (0..300_000) .map(|_| { state ^= state << 13; state ^= state >> 7; @@ -120,13 +153,15 @@ mod tests { }) .collect(); - let mut compressor = CompressionStream::new(BrotliLevel::Brotli10); - let bound = compressor.max_output_size(input.len()); - let mut output_size = 0usize; - for chunk in input.chunks(7919) { - output_size += compressor.append(chunk).unwrap().len(); + for flush_interval in [None, NonZeroUsize::new(1), NonZeroUsize::new(100_000)] { + let mut compressor = CompressionStream::new(BrotliLevel::Brotli10, flush_interval); + let bound = compressor.max_output_size(input.len()); + let mut output_size = 0usize; + for chunk in input.chunks(7919) { + output_size += compressor.append(chunk).unwrap().len(); + } + output_size += compressor.finish().unwrap().len(); + assert!(output_size <= bound, "flush interval {flush_interval:?}"); } - output_size += compressor.finish().unwrap().len(); - assert!(output_size <= bound); } } diff --git a/crates/batcher/encoder/README.md b/crates/batcher/encoder/README.md index faa63da103..8212147c31 100644 --- a/crates/batcher/encoder/README.md +++ b/crates/batcher/encoder/README.md @@ -7,8 +7,9 @@ historical derivation. The writable channel is the FIFO tail. Each accepted batch is compressed once; hard limits are checked before the stream is mutated. `compressed_size_target` -tracks stable output emitted by the compression stream and closes after the -batch that reaches it. +tracks stable output emitted by the compression stream, which a channel with a +target flushes every quarter target of input, and closes after the batch that +reaches it. `DaEgress` frames compressor output against remaining blob capacity, so a full blob can emit while the channel is still open. Artifacts are immutable after diff --git a/crates/batcher/encoder/src/channel.rs b/crates/batcher/encoder/src/channel.rs index 77047f98fd..bb86592c1f 100644 --- a/crates/batcher/encoder/src/channel.rs +++ b/crates/batcher/encoder/src/channel.rs @@ -1,6 +1,6 @@ //! One derivation channel and its admission / close types. -use std::{collections::VecDeque, sync::Arc}; +use std::{collections::VecDeque, num::NonZeroUsize, sync::Arc}; use alloy_primitives::Bytes; use alloy_rlp::Encodable; @@ -176,7 +176,10 @@ impl Channel { Ok(Self { id, rollup_config, - compressor: Some(CompressionStream::new(config.brotli_level)), + compressor: Some(CompressionStream::new( + config.brotli_level, + config.compressed_size_target.and_then(Self::flush_interval), + )), output: VecDeque::new(), available_output: 0, compressed_bytes: 0, @@ -196,6 +199,17 @@ impl Channel { }) } + /// Input bytes between compression-stream flushes for a soft `target`. + /// + /// The target is checked against the bytes the stream has emitted, which cover + /// all but less than one flush interval of the input. A quarter of the target + /// keeps that lag, and so the overshoot, small relative to the target, at a + /// constant few flushes per channel whatever the target. Without a target, or + /// with a zero one, the stream never flushes. + pub const fn flush_interval(target: usize) -> Option { + NonZeroUsize::new(target.div_ceil(4)) + } + /// Returns the channel identifier. pub const fn id(&self) -> ChannelId { self.id @@ -464,6 +478,9 @@ impl Channel { mod tests { use alloy_primitives::B256; use base_common_genesis::UpgradeConfig; + use base_comp::BrotliLevel; + use base_protocol::Brotli; + use rstest::rstest; use super::*; @@ -486,6 +503,38 @@ mod tests { } } + /// A block of token-transfer-like transactions totalling about `block_len` bytes: + /// a few selectors and recipients, mostly-zero amounts, random signatures. + fn transfer_batch(state: &mut u64, block_len: usize) -> SingleBatch { + let mut next = || { + *state ^= *state << 13; + *state ^= *state >> 7; + *state ^= *state << 17; + *state + }; + let mut transactions = Vec::new(); + let mut len = 0; + while len < block_len { + let mut transaction = Vec::with_capacity(160); + transaction.extend_from_slice(&(next() % 4).to_be_bytes()[4..]); + transaction.extend_from_slice(&[(next() % 32) as u8; 20]); + transaction.extend_from_slice(&[0; 24]); + transaction.extend_from_slice(&next().to_be_bytes()); + transaction.extend_from_slice(&(next() % 1_000).to_be_bytes()); + transaction.extend_from_slice(&[(next() % 8) as u8; 96]); + transaction.extend((0..9).flat_map(|_| next().to_be_bytes())); + len += transaction.len(); + transactions.push(Bytes::from(transaction)); + } + SingleBatch { + parent_hash: B256::ZERO, + epoch_num: 0, + epoch_hash: B256::ZERO, + timestamp: 0, + transactions, + } + } + /// A channel on a rollup config with Holocene, hence Fjord's channel size limit, active. fn channel(config: EncoderConfig) -> Channel { let rollup_config = RollupConfig { @@ -513,4 +562,58 @@ mod tests { assert_eq!(channel.blocks_added(), 0); assert_eq!(channel.compressed_bytes(), 0); } + + /// Unflushed, Brotli emits nothing until it holds about a window of input, so these + /// channels would first see output, and close, at about 1.17 MB. The stream's flush + /// makes them close right after the batch that reaches the target, for small and + /// large targets and blocks alike, and the closed channel still decodes to the + /// accepted batches. + #[rstest] + fn soft_target_closes_near_the_target( + #[values(150_000, 700_000)] target: usize, + #[values(4_600, 60_000)] block_len: usize, + ) { + let mut channel = channel(EncoderConfig { + compressed_size_target: Some(target), + ..EncoderConfig::default() + }); + + let mut state = 1u64; + let mut input = Vec::new(); + let mut last_batch_start; + loop { + let batch = transfer_batch(&mut state, block_len); + let outcome = channel.add_batch(&batch, 1).unwrap(); + last_batch_start = input.len(); + batch.rlp_header().encode(&mut input); + input.push(BatchType::Single as u8); + batch.encode(&mut input); + if outcome == ChannelAddOutcome::TargetReached { + break; + } + assert_eq!(outcome, ChannelAddOutcome::Accepted); + } + + // Before the closing batch, the emitted bytes trail the input by less than one + // flush interval; the closing batch adds at most its own compressed bytes. + let lagging_input = + &input[last_batch_start - Channel::flush_interval(target).unwrap().get()..]; + let max_overshoot = BrotliLevel::DEFAULT.compress_channel(lagging_input).unwrap().len(); + let closed_at = channel.compressed_bytes() as usize; + assert!( + (target..=target + max_overshoot).contains(&closed_at), + "closed at {closed_at} bytes, expected {target}..={}", + target + max_overshoot + ); + + channel.close().unwrap(); + let frame = channel.take_frame(channel.available_output(), true).unwrap(); + let decoded = Brotli + .decompress( + &frame.data[1..], + channel.rollup_config.max_rlp_bytes_per_channel(0) as usize, + ) + .unwrap(); + assert_eq!(decoded, input); + } } diff --git a/crates/builder/cli/Cargo.toml b/crates/builder/cli/Cargo.toml index 84c41b68fe..c02b2d0bb5 100644 --- a/crates/builder/cli/Cargo.toml +++ b/crates/builder/cli/Cargo.toml @@ -31,5 +31,8 @@ tracing.workspace = true base-bundles.workspace = true # misc -alloy-primitives.workspace = true rstest.workspace = true +metrics.workspace = true +tempfile.workspace = true +alloy-primitives.workspace = true +metrics-exporter-prometheus.workspace = true diff --git a/crates/builder/cli/src/args.rs b/crates/builder/cli/src/args.rs index 942707c6f6..6152909efb 100644 --- a/crates/builder/cli/src/args.rs +++ b/crates/builder/cli/src/args.rs @@ -1,12 +1,12 @@ //! Builder CLI arguments and config conversion helpers. use core::{net::SocketAddr, time::Duration}; -use std::path::PathBuf; +use std::{path::PathBuf, sync::Arc}; use base_builder_core::{ BuilderApiExtensionConfig, BuilderConfig, DEFAULT_MAX_VALIDITY_PREDICATES, - ExecutionMeteringMode, RejectionCache, RestingPredicateMode, ShadowValidityConfig, - SharedMeteringProvider, ValiditySignatureMode, + ExecutionMeteringMode, RejectionCache, ResourceMeteringConfig, RestingPredicateMode, + ShadowValidityConfig, SharedMeteringProvider, ValiditySignatureMode, }; use base_builder_metering::MeteringStore; use base_execution_cli::ShadowIndexerArgs; @@ -15,7 +15,8 @@ use base_observability_events::{ DEFAULT_MAX_FILE_BYTES, DEFAULT_MAX_FILES, DEFAULT_QUEUE_CAPACITY, TransactionEventProducer, TransactionEventWriterConfig, }; -use tracing::warn; +use eyre::WrapErr; +use tracing::{info, warn}; /// Parameters for Flashblocks configuration. /// @@ -150,7 +151,8 @@ pub struct Args { #[arg(long = "builder.max_gas_per_txn")] pub max_gas_per_txn: Option, - /// Maximum execution time per transaction in microseconds (requires resource metering) + /// Legacy Flashblocks-only wall-clock execution-time ceiling per transaction in microseconds. + /// Retired after Mainnet Denim soaks; native payload resource metering uses `--payload.resource-metering-schedule`. #[arg(long = "builder.max-execution-time-per-tx-us")] pub max_execution_time_per_tx_us: Option, @@ -174,7 +176,8 @@ pub struct Args { #[arg(long = "builder.state-root-gas-anchor-us", hide = true)] pub state_root_gas_anchor_us: Option, - /// Execution metering mode: off, dry-run, or enforce + /// Legacy Flashblocks-only wall-clock execution-metering mode (`off`, `dry-run`, or `enforce`), retired after Mainnet Denim soaks. + /// Native payload resource metering uses per-dimension `dryRun` in `--payload.resource-metering-schedule`. #[arg(long = "builder.execution-metering-mode", value_enum, default_value = "off")] pub execution_metering_mode: ExecutionMeteringMode, @@ -182,10 +185,33 @@ pub struct Args { #[arg(long = "builder.extra-block-deadline-secs", default_value = "20")] pub extra_block_deadline_secs: u64, - /// Whether to enable TIPS Resource Metering - #[arg(long = "builder.enable-resource-metering", default_value = "false")] + /// Enables builder resource metering across both the Flashblocks metering + /// store (`MeteringStore`) and native payload resource-metering schedule + /// evaluation (`--payload.resource-metering-schedule`). + /// + /// This is the single on/off switch for builder metering: when enabled, + /// native schedule evaluation and Flashblocks metering run on whichever + /// builder is live; when disabled, neither runs. + #[arg( + long = "builder.enable-resource-metering", + env = "BUILDER_ENABLE_RESOURCE_METERING", + default_value = "false" + )] pub enable_resource_metering: bool, + /// JSON file containing the startup resource-metering schedule. + /// + /// Evaluated by the native payload builder when `--builder.enable-resource-metering` + /// (`BUILDER_ENABLE_RESOURCE_METERING`) is enabled. Each dimension in the + /// schedule must explicitly set `dryRun` (`true` to observe budgets without + /// excluding transactions, `false` to enforce). + /// + /// Uses the same `--payload.resource-metering-schedule` flag and + /// `PAYLOAD_RESOURCE_METERING_SCHEDULE` environment variable as the + /// standalone execution node so operators can share one schedule path. + #[arg(long = "payload.resource-metering-schedule", env = "PAYLOAD_RESOURCE_METERING_SCHEDULE")] + pub resource_metering_schedule: Option, + /// Maximum validity predicates accepted per validity transaction. /// /// Capped at [`DEFAULT_MAX_VALIDITY_PREDICATES`], the fixed wire ceiling the @@ -244,9 +270,10 @@ pub struct Args { )] pub predicate_bucket_ordered_threshold: usize, - /// Hold back validity transactions whose predicate was unsatisfied earlier in the block until - /// a commit changes the state it reads: off, shadow (track and count mismatches only), or - /// enforce. + /// Hold back validity transactions whose predicate was found unsatisfied until a state change + /// wakes them: off, shadow (track and count mismatches only), or enforce. Resting transactions + /// carry across flashblocks and across blocks this builder finishes in sequence; the native + /// builder carries them across blocks from Denim onward. #[arg(long = "builder.resting-predicates", value_enum, default_value = "off")] pub resting_predicate_mode: RestingPredicateMode, @@ -320,7 +347,7 @@ impl Args { /// Creates a [`MeteringStore`] from the CLI arguments. pub fn build_metering_store(&self) -> MeteringStore { MeteringStore::new( - self.enable_resource_metering || self.execution_metering_mode.is_enabled(), + self.enable_resource_metering, self.tx_data_store_buffer_size, Duration::from_secs(self.metering_store_ttl_secs), ) @@ -341,6 +368,7 @@ impl Default for Args { execution_metering_mode: ExecutionMeteringMode::Off, extra_block_deadline_secs: 20, enable_resource_metering: false, + resource_metering_schedule: None, validity_max_predicates: DEFAULT_MAX_VALIDITY_PREDICATES, validity_signature_mode: ValiditySignatureMode::Off, shadow_validity_injection_enabled: false, @@ -396,6 +424,39 @@ impl Args { { warn!("deprecated builder resource limit flags are ignored"); } + if !self.enable_resource_metering && self.execution_metering_mode.is_enabled() { + warn!( + execution_metering_mode = ?self.execution_metering_mode, + enable_resource_metering = self.enable_resource_metering, + "Flashblocks wall-clock execution limit is inert because builder resource metering is disabled" + ); + } + if self.enable_resource_metering && self.resource_metering_schedule.is_none() { + warn!( + "--builder.enable-resource-metering (BUILDER_ENABLE_RESOURCE_METERING) is enabled without --payload.resource-metering-schedule (PAYLOAD_RESOURCE_METERING_SCHEDULE); native payload resource-metering schedule evaluation is inactive" + ); + } + + let resource_metering = ResourceMeteringConfig::from_parts( + self.enable_resource_metering, + self.resource_metering_schedule.as_deref(), + Arc::clone(&metering_provider), + ) + .wrap_err_with(|| { + self.resource_metering_schedule.as_ref().map_or_else( + || "failed to configure resource metering".to_string(), + |path| format!("failed to load resource metering schedule from {}", path.display()), + ) + })?; + if let Some(path) = self.resource_metering_schedule.as_deref() + && resource_metering.is_active() + { + info!( + path = %path.display(), + dimensions = resource_metering.schedule.dimensions.len(), + "native resource metering schedule loaded" + ); + } let flashblocks_ws_addr = SocketAddr::new( self.flashblocks.flashblocks_addr.parse()?, @@ -423,6 +484,7 @@ impl Args { predicate_bucket_ordered_threshold: self.predicate_bucket_ordered_threshold, resting_predicate_mode: self.resting_predicate_mode, metering_provider, + resource_metering, rejection_cache: RejectionCache::new( self.rejection_cache_max_capacity, Duration::from_secs(self.rejection_cache_ttl_secs), @@ -434,12 +496,10 @@ impl Args { #[cfg(test)] mod tests { - use std::sync::Arc; - use alloy_primitives::{B256, TxHash, U256}; use base_builder_core::{MeteringProvider, NoopMeteringProvider}; use base_bundles::MeterBundleResponse; - use clap::Parser; + use clap::{CommandFactory, Parser}; use rstest::rstest; use super::*; @@ -764,4 +824,231 @@ mod tests { assert_eq!(config.flashblocks_interval, Duration::from_millis(200)); assert_eq!(config.flashblocks_leeway_time, Duration::from_millis(50)); } + + const VALID_RESOURCE_METERING_SCHEDULE_JSON: &str = r#"{ + "version": 1, + "dimensions": [{ + "name": "compute", + "blockLimit": 1000, + "transactionLimit": 500, + "baseGasWeight": 1, + "operations": [ + {"name": "SSTORE", "gasUsedWeight": 2, "countCost": 10}, + {"name": "STATE_CHANGED_STORAGE_SLOT", "countCost": 25} + ], + "dryRun": true + }] + }"#; + + #[test] + fn parses_resource_metering_schedule_flag_and_env() { + let parsed = CommandParser::try_parse_from([ + "builder", + "--builder.enable-resource-metering", + "--payload.resource-metering-schedule", + "/tmp/resource-metering-schedule.json", + ]) + .expect("schedule CLI flag should be accepted"); + + assert!(parsed.args.enable_resource_metering); + assert_eq!( + parsed.args.resource_metering_schedule.as_deref(), + Some(std::path::Path::new("/tmp/resource-metering-schedule.json")) + ); + + let cmd = CommandParser::command(); + let schedule_arg = cmd + .get_arguments() + .find(|arg| arg.get_long() == Some("payload.resource-metering-schedule")) + .expect("schedule argument should be registered"); + assert_eq!( + schedule_arg.get_env(), + Some(std::ffi::OsStr::new("PAYLOAD_RESOURCE_METERING_SCHEDULE")) + ); + } + + #[test] + fn default_args_leave_resource_metering_disabled_with_empty_schedule() { + let config = convert(Args::default()); + assert!(!config.resource_metering.enabled); + assert!(config.resource_metering.schedule.is_empty()); + assert!(!config.resource_metering.is_active()); + + let enabled_without_schedule = + convert(Args { enable_resource_metering: true, ..Default::default() }); + assert!(enabled_without_schedule.resource_metering.enabled); + assert!(enabled_without_schedule.resource_metering.schedule.is_empty()); + assert!(!enabled_without_schedule.resource_metering.is_active()); + + let dir = tempfile::tempdir().expect("tempdir"); + let schedule_path = dir.path().join("schedule.json"); + std::fs::write(&schedule_path, VALID_RESOURCE_METERING_SCHEDULE_JSON) + .expect("write schedule"); + let disabled_with_schedule = convert(Args { + enable_resource_metering: false, + resource_metering_schedule: Some(schedule_path), + ..Default::default() + }); + assert!(!disabled_with_schedule.resource_metering.enabled); + assert!(disabled_with_schedule.resource_metering.schedule.is_empty()); + assert!(!disabled_with_schedule.resource_metering.is_active()); + } + + #[test] + fn into_builder_config_loads_schedule_when_resource_metering_is_enabled() { + let dir = tempfile::tempdir().expect("tempdir"); + let schedule_path = dir.path().join("schedule.json"); + std::fs::write(&schedule_path, VALID_RESOURCE_METERING_SCHEDULE_JSON) + .expect("write schedule"); + + let provider: SharedMeteringProvider = + Arc::new(MeteringStore::new(true, 100, Duration::from_secs(30))); + let args = Args { + enable_resource_metering: true, + resource_metering_schedule: Some(schedule_path), + ..Default::default() + }; + + let config = + args.into_builder_config(Arc::clone(&provider)).expect("valid schedule should convert"); + assert!(config.resource_metering.enabled); + assert!(config.resource_metering.is_active()); + assert_eq!(config.resource_metering.schedule.dimensions.len(), 1); + assert_eq!(config.resource_metering.schedule.dimensions[0].name, "compute"); + assert!(Arc::ptr_eq(&config.resource_metering.provider, &provider)); + } + + #[test] + fn into_builder_config_rejects_invalid_or_missing_schedule_when_enabled() { + let dir = tempfile::tempdir().expect("tempdir"); + let missing_path = dir.path().join("missing.json"); + let missing_display = missing_path.display().to_string(); + let missing_args = Args { + enable_resource_metering: true, + resource_metering_schedule: Some(missing_path), + ..Default::default() + }; + let missing_err = missing_args + .into_builder_config(Arc::new(NoopMeteringProvider)) + .expect_err("missing schedule file should fail startup conversion"); + assert!( + missing_err.to_string().contains(&missing_display), + "error should name the schedule path: {missing_err:#}" + ); + + let invalid_path = dir.path().join("invalid.json"); + let invalid_display = invalid_path.display().to_string(); + std::fs::write(&invalid_path, "{ not valid json").expect("write invalid json"); + let invalid_args = Args { + enable_resource_metering: true, + resource_metering_schedule: Some(invalid_path), + ..Default::default() + }; + let invalid_err = invalid_args + .into_builder_config(Arc::new(NoopMeteringProvider)) + .expect_err("invalid schedule JSON should fail startup conversion"); + assert!( + invalid_err.to_string().contains(&invalid_display), + "error should name the schedule path: {invalid_err:#}" + ); + + let empty_path = dir.path().join("empty.json"); + let empty_display = empty_path.display().to_string(); + std::fs::write(&empty_path, r#"{"version": 1, "dimensions": []}"#) + .expect("write empty schedule"); + let empty_args = Args { + enable_resource_metering: true, + resource_metering_schedule: Some(empty_path), + ..Default::default() + }; + let empty_err = empty_args + .into_builder_config(Arc::new(NoopMeteringProvider)) + .expect_err("empty schedule dimensions should fail closed when metering is enabled"); + assert!( + empty_err.to_string().contains(&empty_display), + "error should name the schedule path: {empty_err:#}" + ); + } + + #[test] + fn enable_resource_metering_is_only_switch_and_supports_env() { + let cmd = CommandParser::command(); + let enable_arg = cmd + .get_arguments() + .find(|arg| arg.get_long() == Some("builder.enable-resource-metering")) + .expect("enable-resource-metering argument should be registered"); + assert_eq!( + enable_arg.get_env(), + Some(std::ffi::OsStr::new("BUILDER_ENABLE_RESOURCE_METERING")) + ); + + let dir = tempfile::tempdir().expect("tempdir"); + let schedule_path = dir.path().join("schedule.json"); + std::fs::write(&schedule_path, VALID_RESOURCE_METERING_SCHEDULE_JSON) + .expect("write schedule"); + + for mode in [ + ExecutionMeteringMode::Off, + ExecutionMeteringMode::DryRun, + ExecutionMeteringMode::Enforce, + ] { + let off_args = Args { + enable_resource_metering: false, + execution_metering_mode: mode, + resource_metering_schedule: Some(schedule_path.clone()), + ..Default::default() + }; + let store = off_args.build_metering_store(); + assert!( + !store.is_enabled(), + "MeteringStore must stay disabled when --builder.enable-resource-metering is false (mode={mode:?})" + ); + let config = + off_args.into_builder_config(Arc::new(store)).expect("conversion should succeed"); + assert!(!config.resource_metering.enabled); + assert!(!config.resource_metering.is_active()); + } + let on_args = Args { + enable_resource_metering: true, + resource_metering_schedule: Some(schedule_path), + ..Default::default() + }; + let on_store = on_args.build_metering_store(); + assert!(on_store.is_enabled()); + let on_config = + on_args.into_builder_config(Arc::new(on_store)).expect("conversion should succeed"); + assert!(on_config.resource_metering.enabled); + assert!(on_config.resource_metering.is_active()); + } + + #[test] + fn live_enabled_without_schedule_configs_boot_with_warning_metric() { + let recorder = metrics_exporter_prometheus::PrometheusBuilder::new().build_recorder(); + let handle = recorder.handle(); + + for mode in [ExecutionMeteringMode::DryRun, ExecutionMeteringMode::Enforce] { + let args = Args { + enable_resource_metering: true, + execution_metering_mode: mode, + resource_metering_schedule: None, + ..Default::default() + }; + let store = args.build_metering_store(); + assert!(store.is_enabled()); + let provider: SharedMeteringProvider = Arc::new(store); + // `base sequencer` performs `into_builder_config` before Reth installs its + // recorder during launch; record the counter from the post-install startup + // hook so it is not lost there. + let config = args + .into_builder_config(provider) + .expect("live enabled-without-schedule config must boot without error"); + metrics::with_local_recorder(&recorder, || { + config.record_startup_metrics(); + }); + assert!(config.resource_metering.enabled); + assert!(!config.resource_metering.is_active()); + } + let rendered = handle.render(); + assert!(rendered.contains("base_builder_resource_metering_schedule_missing_total 2")); + } } diff --git a/crates/builder/core/Cargo.toml b/crates/builder/core/Cargo.toml index 165888abc2..6afec219ae 100644 --- a/crates/builder/core/Cargo.toml +++ b/crates/builder/core/Cargo.toml @@ -56,7 +56,7 @@ reth-payload-primitives.workspace = true base-execution-chainspec.workspace = true base-execution-consensus.workspace = true reth-basic-payload-builder.workspace = true -base-execution-payload-builder.workspace = true +base-execution-payload-builder = { workspace = true, features = ["clap"] } reth-ipc = { workspace = true, optional = true } reth-payload-builder-primitives.workspace = true reth-node-core = { workspace = true, features = ["otlp"] } @@ -244,6 +244,10 @@ test-utils = [ name = "state_root" harness = false +[[bench]] +name = "flashblocks_resting_sweep" +harness = false + [[bench]] name = "tx_selection" harness = false diff --git a/crates/builder/core/benches/flashblocks_resting_sweep.rs b/crates/builder/core/benches/flashblocks_resting_sweep.rs new file mode 100644 index 0000000000..10e83846a6 --- /dev/null +++ b/crates/builder/core/benches/flashblocks_resting_sweep.rs @@ -0,0 +1,342 @@ +//! Per-block latency of the flashblocks payload builder carrying a large resting set, including +//! the blocks that sweep members no build yielded. +//! +//! The pool holds [`LIVE_RESTING`] validity transactions whose predicates never become satisfied, +//! so every flashblock yields them and holds them back, plus enough plain transfers to fill every +//! flashblock. Each block also rests [`CHURN_PER_BLOCK`] members whose transactions then leave the +//! pool, so the sweep always has members to drop and the set stays at a steady size. +//! +//! Blocks run back to back through the same store calls as the flashblocks builder: take the set +//! and carry it into the block, run [`FLASHBLOCKS`] flashblocks of the selection loop, then finish +//! the block and store the set. By default the next block starts immediately, without the state +//! root and leeway the builder overlaps an end-of-block sweep with, so a sweep still running is +//! counted against the next block's carry. `RESTING_SWEEP_BENCH_GAP_MS` adds an untimed pause +//! between blocks to model them. +//! +//! The carry column is the work between the forkchoice update and the first flashblock. Before +//! the sweep moved off the block-start path, the builder ran it before deriving the flashblock +//! schedule, so every millisecond of carry was time drift: at a 170ms flashblock interval, a +//! 170ms carry costs a flashblock. +//! +//! ```sh +//! cargo bench -p base-builder-core --bench flashblocks_resting_sweep +//! ``` + +use std::{ + sync::Arc, + time::{Duration, Instant}, +}; + +use alloy_consensus::{Header, SignableTransaction, TxEip1559}; +use alloy_eips::eip2718::Encodable2718; +use alloy_primitives::{Address, B256, Bytes, Signature, TxKind, U256, keccak256}; +use base_builder_core::{ + BasePayloadBuilderCtx, BestFlashblocksTxs, BlockDeferrals, ExecutionInfo, + ParkableBestPayloadTransactions, RejectionCache, ResourceLimits, RestingPredicateMode, +}; +use base_common_consensus::{BaseTransactionSigned, BaseTxEnvelope}; +use base_execution_chainspec::BaseChainSpec; +use base_execution_payload_builder::{ + DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, RESTING_UNSEEN_RETAINED_BLOCKS, + SharedRestingPredicates, +}; +use base_execution_txpool::{ + BaseOrdering, BasePooledTransaction, BestTransactionsHold, ParkedBestTransactions, + ValidityOperator, ValidityPredicate, +}; +use reth_chainspec::ChainSpec; +use reth_primitives_traits::{Recovered, SealedHeader}; +use reth_revm::State; +use reth_transaction_pool::{ + BestTransactions, TransactionOrigin, ValidPoolTransaction, identifier::TransactionId, + pool::PendingPool, +}; +use revm::{ + database::{InMemoryDB, TransitionState}, + state::AccountInfo, +}; +use tokio_util::sync::CancellationToken; + +type Ordering = BaseOrdering; +type Pool = PendingPool; +type BestTxs = BestFlashblocksTxs< + BasePooledTransaction, + ParkableBestPayloadTransactions, +>; + +const CHAIN_ID: u64 = 901; +/// Resting validity transactions the pool yields in every flashblock. +const LIVE_RESTING: usize = 100_000; +/// Members rested in each block whose transactions leave the pool afterwards. +const CHURN_PER_BLOCK: u64 = 4_800; +/// Flashblocks per block. +const FLASHBLOCKS: usize = 10; +/// Plain transfers that fill each flashblock. +const FILLER_PER_FLASHBLOCK: usize = 100; +/// Blocks built before measuring, so the set reaches its steady-state size. +const WARMUP_BLOCKS: u64 = 3 * RESTING_UNSEEN_RETAINED_BLOCKS; +/// Blocks measured. +const MEASURED_BLOCKS: u64 = 4 * RESTING_UNSEEN_RETAINED_BLOCKS; +/// Long enough that a block always waits for the previous block's sweep, so carry includes it. +const SWEEP_WAIT_LIMIT: Duration = Duration::from_secs(1); +const TRANSFER_GAS: u64 = 21_000; +const SENDER_OFFSET: usize = 1_000_000; + +fn address(index: usize) -> Address { + Address::from_word(U256::from(index + 1).into()) +} + +fn transaction( + sender_index: usize, + priority_fee: u128, + predicates: Vec, +) -> Arc> { + let tx = TxEip1559 { + chain_id: CHAIN_ID, + nonce: 0, + gas_limit: TRANSFER_GAS, + max_fee_per_gas: 1_000_000_000_000, + max_priority_fee_per_gas: priority_fee, + to: TxKind::Call(Address::repeat_byte(0xee)), + value: U256::from(sender_index), + access_list: Default::default(), + input: Bytes::new(), + }; + let envelope = BaseTxEnvelope::Eip1559(tx.into_signed(Signature::test_signature())); + let encoded_length = envelope.encode_2718_len(); + let sender = address(SENDER_OFFSET + sender_index); + let transaction = BasePooledTransaction::new( + Recovered::new_unchecked(BaseTransactionSigned::from(envelope), sender), + encoded_length, + ) + .with_validity_predicates(predicates); + Arc::new(ValidPoolTransaction { + transaction_id: TransactionId::new(((SENDER_OFFSET + sender_index) as u64).into(), 0), + transaction, + propagate: true, + timestamp: Instant::now(), + origin: TransactionOrigin::External, + authority_ids: None, + }) +} + +/// [`LIVE_RESTING`] never-satisfied validity transactions, each watching its own balance, plus +/// filler for every flashblock, and the parent state that funds every sender. +fn resting_pool() -> (Pool, InMemoryDB) { + let mut pool = PendingPool::new(Ordering::coinbase_tip()); + let mut db = InMemoryDB::default(); + for sender_index in 0..LIVE_RESTING + FLASHBLOCKS * FILLER_PER_FLASHBLOCK { + let (priority_fee, predicates) = if sender_index < LIVE_RESTING { + ( + 10, + vec![ValidityPredicate::Balance { + address: address(sender_index), + op: ValidityOperator::Equal, + value: U256::ONE, + }], + ) + } else { + (1, Vec::new()) + }; + pool.add_transaction(transaction(sender_index, priority_fee, predicates), 0); + db.insert_account_info( + address(SENDER_OFFSET + sender_index), + AccountInfo { balance: U256::from(u128::MAX), ..Default::default() }, + ); + } + (pool, db) +} + +fn parkable( + pool: &Pool, + hold: Option>>, +) -> ParkableBestPayloadTransactions { + let mut best = pool.best(); + best.no_updates(); + ParkableBestPayloadTransactions::new(Box::new( + ParkedBestTransactions::new(best, Ordering::coinbase_tip(), 0).with_hold(hold), + )) +} + +fn payload_context() -> BasePayloadBuilderCtx { + let genesis = serde_json::from_value(serde_json::json!({ + "config": { "chainId": CHAIN_ID }, + "gasLimit": "0x1C9C380", + "timestamp": "0x0" + })) + .expect("valid genesis"); + let chain_spec = + ChainSpec::builder().chain(CHAIN_ID.into()).genesis(genesis).cancun_activated().build(); + let parent = Header { gas_limit: 30_000_000, timestamp: 0, ..Default::default() }; + BasePayloadBuilderCtx::for_test( + Arc::new(BaseChainSpec::from(chain_spec)), + Arc::new(SealedHeader::seal_slow(parent)), + ) +} + +/// Runs every flashblock of one block through the selection loop. +fn run_flashblocks( + ctx: &BasePayloadBuilderCtx, + pool: &Pool, + state: &mut State, + best: &mut BestTxs, +) -> usize { + let mut info = ExecutionInfo::default(); + let mut deferrals = BlockDeferrals::default(); + let mut committed_until = 0; + for flashblock in 0..FLASHBLOCKS { + if flashblock > 0 { + best.refresh_iterator(parkable(pool, best.iterator_hold(&CancellationToken::new()))); + } + let limits = ResourceLimits { + block_gas_limit: ((flashblock + 1) * FILLER_PER_FLASHBLOCK) as u64 * TRANSFER_GAS, + ..Default::default() + }; + ctx.execute_best_transactions(&mut info, &mut deferrals, state, best, &limits) + .expect("in-memory execution cannot fail"); + let committed = info.executed_transactions[committed_until..] + .iter() + .map(|transaction| transaction.tx_hash()) + .collect::>(); + best.mark_committed(&committed); + committed_until = info.executed_transactions.len(); + } + info.executed_transactions.len() +} + +/// A storage predicate on its own slot, the shape of a smart-account nonce check. +fn churn_predicate(index: u64) -> ValidityPredicate { + ValidityPredicate::Storage { + address: Address::repeat_byte(0xcc), + slot: U256::from(index), + mask: U256::MAX, + op: ValidityOperator::Equal, + value: U256::from(index + 1), + } +} + +fn block_hash(block: u64) -> B256 { + keccak256(block.to_be_bytes()) +} + +/// Store calls that differ between builder versions. +mod store { + use alloy_primitives::B256; + use base_execution_payload_builder::{RestingPredicateStore, RestingPredicates}; + + pub(super) fn new() -> RestingPredicateStore { + RestingPredicateStore::new(reth_tasks::Runtime::test()) + } + + pub(super) fn finish_and_put( + store: &RestingPredicateStore, + resting: RestingPredicates, + block_hash: B256, + ) { + store.put(store.finish(resting), block_hash); + } +} + +/// Untimed pause between finishing a block and starting the next, from +/// `RESTING_SWEEP_BENCH_GAP_MS`, standing in for the state root and leeway an end-of-block +/// sweep overlaps in the builder. Unset, the next block starts immediately. +fn block_gap() -> Option { + std::env::var("RESTING_SWEEP_BENCH_GAP_MS").ok().map(|ms| { + Duration::from_millis(ms.parse().expect("RESTING_SWEEP_BENCH_GAP_MS is a number")) + }) +} + +struct BlockTiming { + block: u64, + carry: Duration, + selection: Duration, + carried: usize, +} + +fn main() { + let (pool, db) = resting_pool(); + let ctx = payload_context(); + let rejection_cache = RejectionCache::new(100_000, Duration::from_secs(1_800)); + let store = store::new(); + let gap = block_gap(); + let mut churn = 0u64; + let mut timings = Vec::new(); + + for block in 1..=WARMUP_BLOCKS + MEASURED_BLOCKS { + let start = Instant::now(); + let mut resting = store.take(DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, SWEEP_WAIT_LIMIT); + resting.begin_block(block_hash(block - 1), block, Some(&TransitionState::default())); + let carry = start.elapsed(); + let carried = resting.len(); + + let resting = SharedRestingPredicates::new(resting); + let mut best = BestFlashblocksTxs::new( + parkable(&pool, Some(resting.iterator_hold())), + rejection_cache.clone(), + ) + .with_resting_predicate_mode(RestingPredicateMode::Enforce) + .with_resting_predicates(resting); + let mut state = State::builder().with_database(db.clone()).with_bundle_update().build(); + let executed = run_flashblocks(&ctx, &pool, &mut state, &mut best); + assert_eq!(executed, FLASHBLOCKS * FILLER_PER_FLASHBLOCK); + let selection = start.elapsed() - carry; + + let mut resting = best.take_resting_predicates(); + for _ in 0..CHURN_PER_BLOCK { + resting.rest(keccak256(churn.to_be_bytes()), churn_predicate(churn)); + churn += 1; + } + store::finish_and_put(&store, resting, block_hash(block)); + if let Some(gap) = gap { + std::thread::sleep(gap); + } + + if block > WARMUP_BLOCKS { + timings.push(BlockTiming { block, carry, selection, carried }); + } + } + + println!("gap_ms={}", gap.map_or(0, |gap| gap.as_millis())); + println!( + "{:>6} {:>9} {:>12} {:>14} {:>12}", + "block", "carried", "carry (ms)", "selection (ms)", "total (ms)" + ); + for t in &timings { + println!( + "{:>6} {:>9} {:>12.3} {:>14.3} {:>12.3}", + t.block, + t.carried, + t.carry.as_secs_f64() * 1e3, + t.selection.as_secs_f64() * 1e3, + (t.carry + t.selection).as_secs_f64() * 1e3 + ); + } + let summarize = |label: &str, filter: &dyn Fn(&BlockTiming) -> bool| { + let mut carry: Vec = + timings.iter().filter(|t| filter(t)).map(|t| t.carry.as_secs_f64() * 1e3).collect(); + let mut total: Vec = timings + .iter() + .filter(|t| filter(t)) + .map(|t| (t.carry + t.selection).as_secs_f64() * 1e3) + .collect(); + carry.sort_by(f64::total_cmp); + total.sort_by(f64::total_cmp); + let mean = |v: &[f64]| v.iter().sum::() / v.len() as f64; + let quantile = |v: &[f64], q: f64| v[((v.len() - 1) as f64 * q).round() as usize]; + println!( + "SUMMARY {label} blocks={} carry_mean_ms={:.3} carry_p50_ms={:.3} carry_p90_ms={:.3} carry_max_ms={:.3} total_mean_ms={:.3} total_p50_ms={:.3} total_p90_ms={:.3} total_max_ms={:.3}", + carry.len(), + mean(&carry), + quantile(&carry, 0.5), + quantile(&carry, 0.9), + quantile(&carry, 1.0), + mean(&total), + quantile(&total, 0.5), + quantile(&total, 0.9), + quantile(&total, 1.0), + ); + }; + summarize("all", &|_| true); + summarize("sweep_blocks", &|t| t.block.is_multiple_of(RESTING_UNSEEN_RETAINED_BLOCKS)); + summarize("other_blocks", &|t| !t.block.is_multiple_of(RESTING_UNSEEN_RETAINED_BLOCKS)); +} diff --git a/crates/builder/core/benches/tx_selection.rs b/crates/builder/core/benches/tx_selection.rs index 3612775b84..fa9a0c6c67 100644 --- a/crates/builder/core/benches/tx_selection.rs +++ b/crates/builder/core/benches/tx_selection.rs @@ -21,9 +21,12 @@ use base_builder_core::{ }; use base_common_consensus::{BaseTransactionSigned, BaseTxEnvelope}; use base_execution_chainspec::BaseChainSpec; +use base_execution_payload_builder::{ + DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, RestingPredicates, SharedRestingPredicates, +}; use base_execution_txpool::{ - BaseOrdering, BasePooledTransaction, ParkedBestTransactions, PredicateContext, - ValidityOperator, ValidityPredicate, + BaseOrdering, BasePooledTransaction, BestTransactionsHold, ParkedBestTransactions, + PredicateContext, ValidityOperator, ValidityPredicate, }; use base_observability_events::{ GlobalTransactionEventWriter, TransactionEventProducer, TransactionEventWriterConfig, @@ -41,6 +44,7 @@ use revm::{ database::InMemoryDB, state::{Account, AccountInfo, EvmState}, }; +use tokio_util::sync::CancellationToken; type Ordering = BaseOrdering; type Pool = PendingPool; @@ -144,6 +148,19 @@ fn parkable(pool: &Pool) -> ParkableBestPayloadTransactions>>, +) -> ParkableBestPayloadTransactions { + let mut inner = pool.best(); + inner.no_updates(); + ParkableBestPayloadTransactions::new(Box::new( + ParkedBestTransactions::new(inner, Ordering::coinbase_tip(), 0).with_hold(hold), + )) +} + fn selection_benches(c: &mut Criterion) { let mut plain = c.benchmark_group("tx_selection/best_transactions"); plain.sample_size(10); @@ -461,16 +478,24 @@ fn run_resting_block( ) -> usize { let mut info = ExecutionInfo::default(); let mut deferrals = BlockDeferrals::default(); + let resting = SharedRestingPredicates::new(RestingPredicates::new( + DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, + )); + let hold = mode.is_enforced().then(|| resting.iterator_hold()); let mut best = BestFlashblocksTxs::new( - parkable(pool), + resting_parkable(pool, hold), RejectionCache::new(1_000, Duration::from_secs(60)), ) - .with_resting_predicate_mode(mode); + .with_resting_predicate_mode(mode) + .with_resting_predicates(resting); let mut committed_until = 0; for flashblock in 0..RESTING_FLASHBLOCKS { if flashblock > 0 { - best.refresh_iterator(parkable(pool)); + best.refresh_iterator(resting_parkable( + pool, + best.iterator_hold(&CancellationToken::new()), + )); } let limits = ResourceLimits { block_gas_limit: ((flashblock + 1) * RESTING_FILLER_PER_FLASHBLOCK) as u64 diff --git a/crates/builder/core/src/config.rs b/crates/builder/core/src/config.rs index 712659504a..415232c1d2 100644 --- a/crates/builder/core/src/config.rs +++ b/crates/builder/core/src/config.rs @@ -12,8 +12,8 @@ use base_execution_payload_builder::{ }; use crate::{ - ExecutionMeteringMode, NoopMeteringProvider, RejectionCache, RestingPredicateMode, - SharedMeteringProvider, + BuilderMetrics, ExecutionMeteringMode, NoopMeteringProvider, RejectionCache, + ResourceMeteringConfig, RestingPredicateMode, SharedMeteringProvider, }; /// Configuration values for the flashblocks builder. @@ -73,11 +73,15 @@ pub struct BuilderConfig { /// Number of parked predicates that converts one state bucket to ordered wakeups. pub predicate_bucket_ordered_threshold: usize, - /// Whether validity transactions resting under an unchanged predicate are held back. + /// Whether validity transactions resting under an unchanged predicate are held back. The + /// resting set carries across flashblocks and across blocks this builder finishes in sequence. pub resting_predicate_mode: RestingPredicateMode, /// Resource metering provider pub metering_provider: SharedMeteringProvider, + /// Resource-unit metering used to throttle transactions in the native + /// payload builder. + pub resource_metering: ResourceMeteringConfig, /// Cache of permanently rejected transaction hashes, shared across blocks. /// Transactions in this cache are skipped by the iterator without re-evaluation. @@ -126,6 +130,7 @@ impl core::fmt::Debug for BuilderConfig { .field("predicate_bucket_ordered_threshold", &self.predicate_bucket_ordered_threshold) .field("resting_predicate_mode", &self.resting_predicate_mode) .field("metering_provider", &self.metering_provider) + .field("resource_metering", &self.resource_metering) .field("rejection_cache_size", &self.rejection_cache.entry_count()) .field("manifest_precheck_enabled", &self.manifest_precheck_enabled) .field("state_provider_metrics", &self.state_provider_metrics) @@ -135,6 +140,7 @@ impl core::fmt::Debug for BuilderConfig { impl Default for BuilderConfig { fn default() -> Self { + let metering_provider: SharedMeteringProvider = Arc::new(NoopMeteringProvider); Self { block_time: Duration::from_secs(2), block_time_leeway: Duration::from_millis(500), @@ -152,7 +158,11 @@ impl Default for BuilderConfig { predicate_eval_hard_cutoff: Duration::from_millis(10), predicate_bucket_ordered_threshold: DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, resting_predicate_mode: RestingPredicateMode::Off, - metering_provider: Arc::new(NoopMeteringProvider), + metering_provider: Arc::clone(&metering_provider), + resource_metering: ResourceMeteringConfig { + provider: metering_provider, + ..ResourceMeteringConfig::default() + }, rejection_cache: RejectionCache::default(), manifest_precheck_enabled: true, state_provider_metrics: false, @@ -160,6 +170,16 @@ impl Default for BuilderConfig { } } +impl BuilderConfig { + /// Records startup resource-metering configuration metrics once the node's + /// Prometheus recorder is installed. + pub fn record_startup_metrics(&self) { + if self.resource_metering.enabled && self.resource_metering.schedule.is_empty() { + BuilderMetrics::resource_metering_schedule_missing_total().increment(1); + } + } +} + #[cfg(any(test, feature = "test-utils"))] impl BuilderConfig { /// Creates a new [`BuilderConfig`] suitable for testing with a randomized flashblocks port. diff --git a/crates/builder/core/src/flashblocks/best_txs.rs b/crates/builder/core/src/flashblocks/best_txs.rs index b2cffd5c46..9895dd23fe 100644 --- a/crates/builder/core/src/flashblocks/best_txs.rs +++ b/crates/builder/core/src/flashblocks/best_txs.rs @@ -1,12 +1,18 @@ //! Flashblocks adapters for parkable best-transaction iterators. -use std::{collections::HashSet, marker::PhantomData, time::Instant}; +use std::{collections::HashSet, marker::PhantomData, sync::Arc}; -use alloy_primitives::{Address, TxHash, map::B256Set}; -use base_execution_payload_builder::{ParkablePayloadTransactions, ParkedPredicateIndex}; -use base_execution_txpool::{BasePooledTx, ValidityConditions, ValidityPredicate}; +use alloy_primitives::{Address, TxHash}; +use base_execution_payload_builder::{ + DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, ParkablePayloadTransactions, RestingPredicates, + SharedRestingPredicates, +}; +use base_execution_txpool::{ + BasePooledTx, BestTransactionsHold, ValidityConditions, ValidityPredicate, +}; use reth_payload_util::PayloadTransactions; use revm::state::EvmState; +use tokio_util::sync::CancellationToken; use crate::{ BuilderMetrics, RejectionCache, RestingPayloadTransactions, RestingPredicateMode, RestingStats, @@ -14,10 +20,11 @@ use crate::{ /// An adapter that skips transactions already committed or permanently rejected by flashblocks. /// -/// It also holds back validity transactions resting under an unchanged predicate, see -/// [`RestingPayloadTransactions`]. A resting transaction is parked in the inner iterator, so its -/// nonce lane stays blocked, and is promoted back at its priority position once a commit changes -/// the state its predicate reads. +/// It also owns the validity transactions resting under an unchanged predicate, see +/// [`RestingPayloadTransactions`]. Under [`RestingPredicateMode::Enforce`], every pool iterator +/// the flashblocks builder creates takes an [`Self::iterator_hold`] that parks resting members at +/// the source, so their nonce lane stays blocked without the build loop seeing them. A commit that +/// changes the state a member's predicate reads promotes it back at its priority position. pub struct BestFlashblocksTxs where T: BasePooledTx, @@ -32,15 +39,24 @@ where // Identity of the transaction most recently returned to the build loop. current_transaction: Option<(TxHash, Address, u64)>, resting_predicate_mode: RestingPredicateMode, - // Transactions resting in this block, indexed by the predicate last found unsatisfied. - resting: ParkedPredicateIndex<()>, - // Resting transactions parked by `next` in the current inner iterator. Transactions the build - // loop parked are woken by its own predicate index instead. - parked_resting: B256Set, - resting_stats: RestingStats, + // Transactions resting under the predicate last found unsatisfied, carried across flashblocks + // and blocks, and shared with the holds of the pool iterators. + resting: SharedRestingPredicates, transaction: PhantomData, } +/// Returns a hold that parks members of `resting` in a flashblock's pool iterator and stops the +/// iterator's walk past them once `cancel`, the flashblock's cancellation, fires. The build loop +/// checks the same token before each candidate, so a deadline that fires mid-walk ends the +/// flashblock as promptly as one that fires between candidates. +pub fn flashblock_hold( + resting: &SharedRestingPredicates, + cancel: &CancellationToken, +) -> Box> { + let cancel = cancel.clone(); + resting.clone().with_interrupt(Arc::new(move || cancel.is_cancelled())).iterator_hold() +} + impl std::fmt::Debug for BestFlashblocksTxs where T: BasePooledTx, @@ -51,7 +67,7 @@ where .field("committed_transactions", &self.committed_transactions) .field("rejection_cache_size", &self.rejection_cache.entry_count()) .field("resting_predicate_mode", &self.resting_predicate_mode) - .field("parked_resting", &self.parked_resting.len()) + .field("resting", &self.resting.len()) .finish_non_exhaustive() } } @@ -69,26 +85,49 @@ where rejection_cache, current_transaction: None, resting_predicate_mode: RestingPredicateMode::Off, - resting: ParkedPredicateIndex::default(), - parked_resting: B256Set::default(), - resting_stats: RestingStats::default(), + resting: SharedRestingPredicates::new(RestingPredicates::new( + DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, + )), transaction: PhantomData, } } - /// Sets whether resting validity transactions are tracked and held back. + /// Sets whether resting validity transactions are tracked and held back, within a block and + /// across blocks. #[must_use] pub const fn with_resting_predicate_mode(mut self, mode: RestingPredicateMode) -> Self { self.resting_predicate_mode = mode; self } + /// Starts from `resting`, typically carried over from the previous block. The pool iterators + /// this adapter wraps must take their hold from [`Self::iterator_hold`] of the same adapter. + #[must_use] + pub fn with_resting_predicates(mut self, resting: SharedRestingPredicates) -> Self { + self.resting = resting; + self + } + + /// Returns the hold a new inner iterator parks resting members with, or `None` when resting + /// transactions are not enforced and the build loop evaluates them. `cancel` is the + /// cancellation of the flashblock the iterator serves. + pub fn iterator_hold( + &self, + cancel: &CancellationToken, + ) -> Option>> { + self.resting_predicate_mode.is_enforced().then(|| flashblock_hold(&self.resting, cancel)) + } + + /// Takes the resting set, leaving an empty one, so it can be carried into the next block. + pub fn take_resting_predicates(&mut self) -> RestingPredicates { + self.resting.take_predicates() + } + /// Replaces current iterator with new one. We use it on new flashblock building, to refresh /// priority boundaries pub fn refresh_iterator(&mut self, inner: I) { self.inner = inner; self.current_transaction = None; - self.parked_resting.clear(); } /// Remove transaction from next iteration since it is already in the state @@ -135,24 +174,6 @@ where continue; } - if self.resting_predicate_mode.is_enforced() - && !self.resting.is_empty() - && !tx.validity_conditions().is_empty() - { - let started = Instant::now(); - let resting = self.is_resting(hash, tx.validity_conditions()); - if resting { - self.inner.park_current(); - self.parked_resting.insert(hash); - self.current_transaction = None; - self.resting_stats.parked += 1; - } - self.resting_stats.duration += started.elapsed(); - if resting { - continue; - } - } - return Some(tx); } } @@ -203,37 +224,36 @@ where T: BasePooledTx, I: ParkablePayloadTransactions, { - /// Flashblock-index predicates are not recorded because the index changes between - /// flashblocks without any commit. fn rest(&mut self, transaction_hash: TxHash, predicate: &ValidityPredicate) { - if !self.resting_predicate_mode.is_enabled() - || matches!(predicate, ValidityPredicate::FlashblockIndex { .. }) - { - return; + if self.resting_predicate_mode.is_enabled() { + self.resting.rest(transaction_hash, predicate.clone()); + } + } + + // With resting off the set stays empty, so these skip its lock rather than take it once per + // evaluated validity candidate and once per commit. + fn release(&mut self, transaction_hash: TxHash) { + if self.resting_predicate_mode.is_enabled() { + self.resting.remove(transaction_hash); } - self.resting.park(transaction_hash, (), predicate.clone()); } fn record_committed_state(&mut self, state: &EvmState) { - if self.resting.is_empty() { + if !self.resting_predicate_mode.is_enabled() { return; } - for transaction_hash in self.resting.affected_by_state(state).affected_transactions { - self.resting.remove(transaction_hash); - if self.parked_resting.remove(&transaction_hash) { - self.inner.promote(transaction_hash); - } + for transaction_hash in self.resting.wake_by_state(state) { + self.inner.promote(transaction_hash); } } - /// A hash re-added to the pool with a batch that no longer contains the recorded predicate - /// does not rest. fn is_resting(&self, transaction_hash: TxHash, predicates: &ValidityConditions) -> bool { - self.resting.predicate(transaction_hash).is_some_and(|blocker| predicates.contains(blocker)) + self.resting_predicate_mode.is_enabled() + && self.resting.member(transaction_hash, predicates).is_some() } fn take_resting_stats(&mut self) -> RestingStats { - std::mem::take(&mut self.resting_stats) + self.resting.take_stats() } } @@ -243,11 +263,11 @@ mod tests { use alloy_consensus::{SignableTransaction, Transaction, TxEip1559}; use alloy_eips::eip2718::Encodable2718; - use alloy_primitives::{Address, Signature, TxHash, TxKind, U256}; + use alloy_primitives::{Address, B256, Signature, TxHash, TxKind, U256}; use base_common_consensus::{BaseTransactionSigned, BaseTxEnvelope}; use base_execution_txpool::{ - BaseOrdering, BasePooledTransaction, ParkedBestTransactions, ValidityOperator, - ValidityPredicate, + BaseOrdering, BasePooledTransaction, HOLD_INTERRUPT_INTERVAL, ParkedBestTransactions, + ValidityOperator, ValidityPredicate, }; use reth_payload_util::PayloadTransactions; use reth_primitives_traits::Recovered; @@ -255,8 +275,12 @@ mod tests { PoolTransaction, TransactionOrigin, ValidPoolTransaction, identifier::TransactionId, pool::PendingPool, }; - use revm::state::{Account, EvmState}; + use revm::{ + database::TransitionState, + state::{Account, EvmState}, + }; + use super::*; use crate::{ BestFlashblocksTxs, ParkableBestPayloadTransactions, ParkablePayloadTransactions, RejectionCache, RestingPayloadTransactions, RestingPredicateMode, @@ -644,11 +668,48 @@ mod tests { EvmState::from_iter([(address, account)]) } + /// Builds the production lane-parking iterator over a snapshot of `pool` whose hold parks the + /// resting members of `resting`, as the pool does for an enforcing build. + fn parkable_with_hold( + pool: &PendingPool, + hold: Option>>, + ) -> Parkable { + ParkableBestPayloadTransactions::new(Box::new( + ParkedBestTransactions::new(pool.best(), Ordering::coinbase_tip(), 0).with_hold(hold), + )) + } + + fn resting_iterator_in_mode( + pool: &PendingPool, + mode: RestingPredicateMode, + resting: RestingPredicates, + ) -> BestFlashblocksTxs { + let resting = SharedRestingPredicates::new(resting); + let hold = mode.is_enforced().then(|| resting.iterator_hold()); + BestFlashblocksTxs::new(parkable_with_hold(pool, hold), test_rejection_cache()) + .with_resting_predicate_mode(mode) + .with_resting_predicates(resting) + } + fn resting_iterator( pool: &PendingPool, ) -> BestFlashblocksTxs { - BestFlashblocksTxs::new(parkable(pool), test_rejection_cache()) - .with_resting_predicate_mode(RestingPredicateMode::Enforce) + resting_iterator_in_mode( + pool, + RestingPredicateMode::Enforce, + RestingPredicates::new(DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD), + ) + } + + /// Starts the next flashblock over a fresh snapshot of `pool`. + fn refresh( + iterator: &mut BestFlashblocksTxs, + pool: &PendingPool, + ) { + iterator.refresh_iterator(parkable_with_hold( + pool, + iterator.iterator_hold(&CancellationToken::new()), + )); } /// Plays the build loop's part for a candidate whose predicate is unsatisfied. @@ -657,7 +718,8 @@ mod tests { transaction: &BasePooledTransaction, ) { iterator.park_current(); - iterator.rest(*transaction.hash(), &transaction.validity_predicates()[0]); + let predicate = transaction.validity_predicates().next().unwrap(); + iterator.rest(*transaction.hash(), predicate); } #[test] @@ -681,7 +743,7 @@ mod tests { // Flashblock 2: the resting transaction is not yielded, and a commit to unrelated state // does not release it. - iterator.refresh_iterator(parkable(&pool)); + refresh(&mut iterator, &pool); assert_eq!(*iterator.next(()).unwrap().hash(), *unrelated.hash()); iterator.record_committed_state(&balance_change(UNRELATED, 0, 1)); iterator.mark_current_committed(); @@ -696,6 +758,37 @@ mod tests { assert_eq!(*iterator.next(()).unwrap().hash(), *low.hash()); } + #[test] + fn cancelled_flashblock_stops_walking_past_resting_transactions() { + let members = u64::from(HOLD_INTERRUPT_INTERVAL) + 1; + let mut transactions = (0..members) + .map(|sender| validity_transaction(sender, 0, 10, vec![balance_at_least(WATCHED, 1)])) + .collect::>(); + let plain = transaction(members, 0, 1); + transactions.push(Arc::clone(&plain)); + let pool = pending_pool(&transactions); + let mut iterator = resting_iterator(&pool); + + // Flashblock 1: the build loop finds every validity candidate unsatisfied. + for _ in 0..members { + let candidate = iterator.next(()).unwrap(); + park_unsatisfied(&mut iterator, &candidate); + } + + // Flashblock 2 is cancelled before its walk past the resting members reaches `plain`, so + // the iterator returns control after one check interval instead of finishing the walk. + let cancelled = CancellationToken::new(); + cancelled.cancel(); + iterator.refresh_iterator(parkable_with_hold(&pool, iterator.iterator_hold(&cancelled))); + assert!(iterator.next(()).is_none()); + assert_eq!(iterator.take_resting_stats().parked, u64::from(HOLD_INTERRUPT_INTERVAL)); + + // Flashblock 3 runs to its deadline and walks past every member. + refresh(&mut iterator, &pool); + assert_eq!(*iterator.next(()).unwrap().hash(), *plain.hash()); + assert_eq!(iterator.take_resting_stats().parked, members); + } + #[test] fn nonce_descendant_waits_for_resting_parent() { let parent = validity_transaction(0, 0, 10, vec![balance_at_least(WATCHED, 1)]); @@ -707,7 +800,7 @@ mod tests { let first = iterator.next(()).unwrap(); park_unsatisfied(&mut iterator, &first); - iterator.refresh_iterator(parkable(&pool)); + refresh(&mut iterator, &pool); assert_eq!(*iterator.next(()).unwrap().hash(), *other.hash()); iterator.mark_current_committed(); assert!(iterator.next(()).is_none()); @@ -732,7 +825,7 @@ mod tests { iterator.record_committed_state(&balance_change(WATCHED, 0, 1)); assert!(iterator.next(()).is_none()); - iterator.refresh_iterator(parkable(&pool)); + refresh(&mut iterator, &pool); assert_eq!(*iterator.next(()).unwrap().hash(), *resting.hash()); } @@ -749,9 +842,10 @@ mod tests { let first = iterator.next(()).unwrap(); park_unsatisfied(&mut iterator, &first); - iterator.rest(*resting.hash(), &first.validity_predicates()[1]); + let unrelated = first.validity_predicates().nth(1).unwrap(); + iterator.rest(*resting.hash(), unrelated); - iterator.refresh_iterator(parkable(&pool)); + refresh(&mut iterator, &pool); assert!(iterator.next(()).is_none()); iterator.record_committed_state(&balance_change(WATCHED, 0, 1)); assert!(iterator.next(()).is_none()); @@ -769,7 +863,7 @@ mod tests { let first = iterator.next(()).unwrap(); park_unsatisfied(&mut iterator, &first); - iterator.refresh_iterator(parkable(&pending_pool(&[Arc::clone(&readded)]))); + refresh(&mut iterator, &pending_pool(&[Arc::clone(&readded)])); assert_eq!(*iterator.next(()).unwrap().hash(), *readded.hash()); } @@ -790,7 +884,7 @@ mod tests { let first = iterator.next(()).unwrap(); park_unsatisfied(&mut iterator, &first); - iterator.refresh_iterator(parkable(&pool)); + refresh(&mut iterator, &pool); assert_eq!(*iterator.next(()).unwrap().hash(), *resting.hash()); } @@ -798,13 +892,16 @@ mod tests { fn shadow_mode_tracks_resting_transactions_without_holding_them_back() { let resting = validity_transaction(0, 0, 10, vec![balance_at_least(WATCHED, 1)]); let pool = pending_pool(&[Arc::clone(&resting)]); - let mut iterator = BestFlashblocksTxs::new(parkable(&pool), test_rejection_cache()) - .with_resting_predicate_mode(RestingPredicateMode::Shadow); + let mut iterator = resting_iterator_in_mode( + &pool, + RestingPredicateMode::Shadow, + RestingPredicates::new(DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD), + ); let first = iterator.next(()).unwrap(); park_unsatisfied(&mut iterator, &first); - iterator.refresh_iterator(parkable(&pool)); + refresh(&mut iterator, &pool); let yielded = iterator.next(()).unwrap(); assert_eq!(*yielded.hash(), *resting.hash()); assert!(iterator.is_resting(*resting.hash(), yielded.validity_conditions())); @@ -813,14 +910,81 @@ mod tests { #[test] fn off_mode_does_not_track_resting_transactions() { let resting = validity_transaction(0, 0, 10, vec![balance_at_least(WATCHED, 1)]); - let predicates = resting.transaction.validity_predicates(); + let predicate = resting.transaction.validity_predicates().next().unwrap(); let mut iterator = BestFlashblocksTxs::new( parkable(&pending_pool(&[Arc::clone(&resting)])), test_rejection_cache(), ); - iterator.rest(*resting.hash(), &predicates[0]); + iterator.rest(*resting.hash(), predicate); assert!(!iterator.is_resting(*resting.hash(), resting.transaction.validity_conditions())); } + + /// Finishes the block `iterator` built and returns the next block's iterator over `pool`, + /// carrying the resting set. + fn next_block_iterator( + mut iterator: BestFlashblocksTxs, + pool: &PendingPool, + mode: RestingPredicateMode, + ) -> BestFlashblocksTxs { + const FINISHED: B256 = B256::repeat_byte(0x01); + let mut resting = iterator.take_resting_predicates(); + resting.end_block(FINISHED); + resting.begin_block(FINISHED, 1, Some(&TransitionState::default())); + resting_iterator_in_mode(pool, mode, resting) + } + + #[test] + fn carried_resting_transaction_is_held_back_until_its_state_changes() { + let resting = validity_transaction(0, 0, 10, vec![balance_at_least(WATCHED, 1)]); + let trigger = transaction(1, 0, 5); + let pool = pending_pool(&[Arc::clone(&resting), Arc::clone(&trigger)]); + let mut iterator = resting_iterator(&pool); + let first = iterator.next(()).unwrap(); + park_unsatisfied(&mut iterator, &first); + + let mut iterator = next_block_iterator(iterator, &pool, RestingPredicateMode::Enforce); + + assert_eq!(*iterator.next(()).unwrap().hash(), *trigger.hash()); + iterator.record_committed_state(&balance_change(WATCHED, 0, 1)); + iterator.mark_current_committed(); + assert_eq!(*iterator.next(()).unwrap().hash(), *resting.hash()); + let stats = iterator.take_resting_stats(); + assert_eq!((stats.parked, stats.carried_parked), (1, 1)); + } + + #[test] + fn shadow_mode_carries_resting_transactions_without_holding_them_back() { + let resting = validity_transaction(0, 0, 10, vec![balance_at_least(WATCHED, 1)]); + let pool = pending_pool(&[Arc::clone(&resting)]); + let mut iterator = resting_iterator_in_mode( + &pool, + RestingPredicateMode::Shadow, + RestingPredicates::new(DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD), + ); + let first = iterator.next(()).unwrap(); + park_unsatisfied(&mut iterator, &first); + + let mut iterator = next_block_iterator(iterator, &pool, RestingPredicateMode::Shadow); + + let yielded = iterator.next(()).unwrap(); + assert_eq!(*yielded.hash(), *resting.hash()); + assert!(iterator.is_resting(*resting.hash(), yielded.validity_conditions())); + } + + #[test] + fn released_transaction_is_not_carried_into_the_next_block() { + let resting = validity_transaction(0, 0, 10, vec![balance_at_least(WATCHED, 1)]); + let pool = pending_pool(&[Arc::clone(&resting)]); + let mut iterator = resting_iterator(&pool); + let first = iterator.next(()).unwrap(); + park_unsatisfied(&mut iterator, &first); + // A later evaluation in the same block finds the predicate satisfied. + iterator.release(*resting.hash()); + + let mut iterator = next_block_iterator(iterator, &pool, RestingPredicateMode::Enforce); + + assert_eq!(*iterator.next(()).unwrap().hash(), *resting.hash()); + } } diff --git a/crates/builder/core/src/flashblocks/context.rs b/crates/builder/core/src/flashblocks/context.rs index 438b686cd1..fc763de4df 100644 --- a/crates/builder/core/src/flashblocks/context.rs +++ b/crates/builder/core/src/flashblocks/context.rs @@ -775,7 +775,7 @@ impl BasePayloadBuilderCtx { // Total validity-predicate evaluation time (inclusive of the state loads each // evaluation performs) across this flashblock build. `None` until the first // evaluation, so it both accumulates and records whether any validity - // transaction was seen; emitted once when the loop finishes. + // transaction was seen; emitted once when the loop ends, including on cancellation. let mut predicate_eval_total: Option = None; let mut validity_candidates_evaluated = 0_u64; let mut validity_candidates_deferred = 0_u64; @@ -784,12 +784,12 @@ impl BasePayloadBuilderCtx { let mut validity_handling = Duration::ZERO; while let Some(tx) = best_txs.next(()) { + // Break rather than return so the per-build metrics below are still recorded. A + // flashblock whose selection loop runs until the deadline is the case where + // validity handling costs the most, so dropping its observations would hide it. if self.cancel.is_cancelled() { diag.cancelled = true; - diag.txs_considered = num_txs_considered; - diag.txs_included = - (info.executed_transactions.len() as u64).saturating_sub(min_tx_index); - return Ok(diag); + break; } num_txs_considered += 1; @@ -876,14 +876,18 @@ impl BasePayloadBuilderCtx { "matched" }; ValidityMetrics::validity_predicate_evaluations_total(outcome).increment(1); + // Only matched candidates pay for the resting lookup. if outcome == "matched" && best_txs.is_resting(tx_hash, tx.validity_conditions()) { - BuilderMetrics::resting_predicate_shadow_mismatches_total().increment(1); + ValidityMetrics::resting_predicate_shadow_mismatches_total().increment(1); debug!( target: "payload_builder", tx_hash = ?tx_hash, "resting validity transaction matched when evaluated" ); } + if predicate_read_failed || predicate_expired || blocking_predicate.is_none() { + best_txs.release(tx_hash); + } } if predicate_read_failed || blocking_predicate.is_some() { let (reason, detail) = if predicate_read_failed { @@ -1474,12 +1478,14 @@ impl BasePayloadBuilderCtx { ValidityMetrics::validity_predicate_evaluations_total(outcome).increment(1); if predicate_read_failed { predicate_index.remove(*parked_hash); + best_txs.release(*parked_hash); best_txs.discard_parked(*parked_hash); } else if let Some((_, predicate)) = blocking_predicate { best_txs.rest(*parked_hash, &predicate); predicate_index.reindex(*parked_hash, predicate); } else { predicate_index.remove(*parked_hash); + best_txs.release(*parked_hash); best_txs.promote(*parked_hash); } } @@ -1538,7 +1544,11 @@ impl BasePayloadBuilderCtx { } let resting_stats = best_txs.take_resting_stats(); if resting_stats.parked > 0 { - BuilderMetrics::resting_predicate_parked_total().increment(resting_stats.parked); + ValidityMetrics::resting_predicate_parked_total().increment(resting_stats.parked); + } + if resting_stats.carried_parked > 0 { + ValidityMetrics::resting_predicate_carried_parked_total() + .increment(resting_stats.carried_parked); } let validity_handling = validity_handling + resting_stats.duration; if !validity_handling.is_zero() { @@ -1564,6 +1574,9 @@ impl BasePayloadBuilderCtx { &predicate_index, ); + // A pool iterator whose hold the cancellation interrupted returns `None` before the loop + // reaches its per-candidate check, so record the cancellation here as well. + diag.cancelled |= self.cancel.is_cancelled(); diag.txs_considered = num_txs_considered; diag.txs_included = (info.executed_transactions.len() as u64).saturating_sub(min_tx_index); @@ -1901,6 +1914,64 @@ mod tests { assert!(best_txs.over_limit_remaining > OVER_LIMIT - 10); } + #[test] + fn cancelled_selection_still_records_per_build_metrics() { + let ctx = test_builder_context(); + let mut best_txs = LimitRejectionTransactions::new(0, 2, ctx.cancel.clone()); + let db = StateProviderDatabase::new(NoopProvider::default()); + let mut state = State::builder().with_database(db).with_bundle_update().build(); + let mut info = ExecutionInfo::default(); + let limits = ResourceLimits { block_gas_limit: 0, ..Default::default() }; + let recorder = metrics_exporter_prometheus::PrometheusBuilder::new().build_recorder(); + let handle = recorder.handle(); + + let diagnostics = metrics::with_local_recorder(&recorder, || { + ctx.execute_best_transactions( + &mut info, + &mut BlockDeferrals::default(), + &mut state, + &mut best_txs, + &limits, + ) + .expect("cancelled selection should succeed") + }); + + assert!(diagnostics.cancelled); + let rendered = handle.render(); + assert!( + rendered + .contains("base_builder_validity_predicate_candidates_evaluated_per_build_count 1"), + "cancelled build must record per-build validity metrics:\n{rendered}" + ); + assert!( + rendered.contains("base_builder_payload_num_tx_considered_count 1"), + "cancelled build must record payload builder metrics:\n{rendered}" + ); + } + + #[test] + fn cancellation_that_ends_the_iterator_is_reported_as_cancelled() { + // An interrupted resting hold ends the pool iterator without yielding a candidate, so the + // loop never reaches its per-candidate cancellation check. + let ctx = test_builder_context(); + ctx.cancel.cancel(); + let mut best_txs = LimitRejectionTransactions::new(0, 0, ctx.cancel.clone()); + let db = StateProviderDatabase::new(NoopProvider::default()); + let mut state = State::builder().with_database(db).with_bundle_update().build(); + + let diagnostics = ctx + .execute_best_transactions( + &mut ExecutionInfo::default(), + &mut BlockDeferrals::default(), + &mut state, + &mut best_txs, + &ResourceLimits::default(), + ) + .expect("cancelled selection should succeed"); + + assert_eq!(diagnostics.selection_outcome(), FlashblockSelectionOutcome::Cancelled); + } + #[test] fn diagnostics_report_selection_outcome() { let diag = FlashblockDiagnostics::default(); diff --git a/crates/builder/core/src/flashblocks/mod.rs b/crates/builder/core/src/flashblocks/mod.rs index cc58bc4cd7..40ba645a68 100644 --- a/crates/builder/core/src/flashblocks/mod.rs +++ b/crates/builder/core/src/flashblocks/mod.rs @@ -4,10 +4,10 @@ mod best_txs; pub use base_execution_payload_builder::{ BlockDeferrals, FLOW_STANDARD, FLOW_VALIDITY, InclusionFlow, InclusionTracker, ParkableBestPayloadTransactions, ParkablePayloadTransactions, ParkedPredicateIndex, - PayloadTransactionInvalidated, PredicateLoadTracker, PredicateReadRecorder, StateChangeEffects, - ValidityPredicateEvaluation, ValidityPredicateKey, + PayloadTransactionInvalidated, PredicateLoadTracker, PredicateReadRecorder, RestingStats, + StateChangeEffects, ValidityPredicateEvaluation, ValidityPredicateKey, }; -pub use best_txs::BestFlashblocksTxs; +pub use best_txs::{BestFlashblocksTxs, flashblock_hold}; mod deadline; pub use deadline::PayloadJobDeadline; @@ -29,7 +29,7 @@ pub use context::{ mod payload; mod resting; -pub use resting::{RestingPayloadTransactions, RestingStats}; +pub use resting::RestingPayloadTransactions; mod service; pub use service::FlashblocksServiceBuilder; diff --git a/crates/builder/core/src/flashblocks/payload.rs b/crates/builder/core/src/flashblocks/payload.rs index 854887a6fc..a919c9300f 100644 --- a/crates/builder/core/src/flashblocks/payload.rs +++ b/crates/builder/core/src/flashblocks/payload.rs @@ -26,7 +26,7 @@ use base_execution_consensus::{calculate_receipt_root_no_memo, isthmus}; use base_execution_evm::{BaseEvmConfig, BaseNextBlockEnvAttributes}; use base_execution_payload_builder::{ BaseBuiltPayload, BasePayloadBuilderAttributes, BuilderMetrics as SharedBuilderMetrics, - ValidityMetrics, + RestingPredicateStore, RestingPredicates, SharedRestingPredicates, ValidityMetrics, }; use base_execution_txpool::AccountStateDiff; use base_observability_events::{GlobalTransactionEventWriter, TransactionEventType}; @@ -44,6 +44,7 @@ use reth_provider::{ use reth_revm::{ State, database::StateProviderDatabase, db::states::bundle_state::BundleRetention, }; +use reth_tasks::Runtime; use reth_transaction_pool::TransactionPool; use reth_trie::{HashedPostState, updates::TrieUpdates}; use revm::Database as _; @@ -51,7 +52,7 @@ use serde::{Deserialize, Serialize}; use serde_with::skip_serializing_none; use tokio::sync::{mpsc, watch}; use tokio_util::sync::CancellationToken; -use tracing::{debug, error, info, metadata::Level, span, warn}; +use tracing::{debug, debug_span, error, info, metadata::Level, span, warn}; use crate::{ BlockDeferrals, BuilderConfig, BuilderMetrics, ExecutionInfo, PayloadBuilder, ResourceLimits, @@ -122,6 +123,8 @@ pub(super) struct BasePayloadBuilder { pub outputs: BuilderOutputs, /// Last flashblock emitted by this builder instance. last_emitted_flashblock_id: Arc, + /// Validity transactions resting from the last block this builder finished. + resting_predicates: RestingPredicateStore, } impl BasePayloadBuilder { @@ -132,6 +135,7 @@ impl BasePayloadBuilder { client: Client, config: BuilderConfig, outputs: BuilderOutputs, + executor: Runtime, ) -> Self { Self { evm_config, @@ -140,6 +144,7 @@ impl BasePayloadBuilder { config, outputs, last_emitted_flashblock_id: Arc::default(), + resting_predicates: RestingPredicateStore::new(executor), } } @@ -300,9 +305,37 @@ where BuilderMetrics::sequencer_tx_duration().record(sequencer_tx_time); BuilderMetrics::sequencer_tx_gauge().set(sequencer_tx_time); - // We adjust our flashblocks timings based on time_drift if dynamic adjustment enable + // Derive the flashblock schedule before carrying the resting set into the block, so the + // carry cannot delay the schedule; the timer below takes it out of the first flashblock. let (flashblocks_per_block, first_flashblock_offset) = self.calculate_flashblocks(timestamp); + BuilderMetrics::flashblocks_schedule_delay().record(block_build_start_time.elapsed()); + let resting_carry_start = Instant::now(); + + // Resting transactions carry across flashblocks and across the blocks this builder finishes + // in sequence. Checked against the parent before the fallback build merges the transitions, + // which hold the system calls and deposits applied on top of the parent state. + let resting = (!ctx.attributes().no_tx_pool + && self.config.resting_predicate_mode.is_enabled()) + .then(|| { + let _span = debug_span!(target: "payload_builder", "resting_begin_block").entered(); + let start = Instant::now(); + // The sweep has already overlapped the previous block's state root and the leeway, so + // this rarely waits. Waiting longer than one flashblock would cost more than starting + // with an empty set, which only means evaluating every validity transaction again. + let mut resting = self.resting_predicates.take( + self.config.predicate_bucket_ordered_threshold, + self.config.flashblocks_interval, + ); + let carry_over = resting.begin_block( + ctx.parent().hash(), + ctx.block_number(), + state.transition_state.as_ref(), + ); + ValidityMetrics::record_resting_carry_over(&carry_over, start.elapsed()); + resting + }); + let resting_carry = resting_carry_start.elapsed(); let skip_flashblocks_building = ctx.attributes().no_tx_pool || flashblocks_per_block == 0; @@ -416,15 +449,25 @@ where let mut fb_cancel = block_cancel.child_token(); ctx = ctx.with_cancel(fb_cancel.clone()).with_extra_ctx(extra); - // Create best_transaction iterator + // Create best_transaction iterator. The resting set is shared with every pool iterator of + // this block, whose hold parks resting members before the build loop sees them. + let resting = SharedRestingPredicates::new(resting.unwrap_or_else(|| { + RestingPredicates::new(self.config.predicate_bucket_ordered_threshold) + })); + let hold = self + .config + .resting_predicate_mode + .is_enforced() + .then(|| crate::flashblock_hold(&resting, &fb_cancel)); let best_txs_attributes = ctx.best_transaction_attributes(); let mut best_txs = BestFlashblocksTxs::new( ParkableBestPayloadTransactions::new( - self.pool.best_transactions_with_attributes_and_parking(best_txs_attributes), + self.pool.best_transactions_with_attributes_and_parking(best_txs_attributes, hold), ), self.config.rejection_cache.clone(), ) - .with_resting_predicate_mode(self.config.resting_predicate_mode); + .with_resting_predicate_mode(self.config.resting_predicate_mode) + .with_resting_predicates(resting); let interval = self.config.flashblocks_interval; let (tx, mut rx) = mpsc::channel((self.config.flashblocks_per_block() + 1) as usize); @@ -434,7 +477,7 @@ where async move { let mut timer = tokio::time::interval_at( tokio::time::Instant::now() - .checked_add(first_flashblock_offset) + .checked_add(first_flashblock_delay(first_flashblock_offset, resting_carry)) .expect("can add flashblock offset to current time"), interval, ); @@ -490,7 +533,7 @@ where &span, "Payload building complete, target flashblock count reached", ); - return self.finalize_payload(&mut state, &ctx, &mut info); + return self.finalize_block(&mut state, &ctx, &mut info, &mut best_txs); } // build first flashblock immediately @@ -517,7 +560,7 @@ where &span, "Payload building complete, job cancelled or target flashblock count reached", ); - return self.finalize_payload(&mut state, &ctx, &mut info); + return self.finalize_block(&mut state, &ctx, &mut info, &mut best_txs); } Err(err) => { error!( @@ -543,7 +586,7 @@ where &span, "Payload building complete, channel closed or job cancelled", ); - return self.finalize_payload(&mut state, &ctx, &mut info); + return self.finalize_block(&mut state, &ctx, &mut info, &mut best_txs); } } } @@ -625,7 +668,10 @@ where let best_txs_start_time = Instant::now(); let best_txs_attributes = ctx.best_transaction_attributes(); best_txs.refresh_iterator(ParkableBestPayloadTransactions::new( - self.pool.best_transactions_with_attributes_and_parking(best_txs_attributes), + self.pool.best_transactions_with_attributes_and_parking( + best_txs_attributes, + best_txs.iterator_hold(&ctx.cancel), + ), )); let transaction_pool_fetch_time = best_txs_start_time.elapsed(); BuilderMetrics::transaction_pool_fetch_duration().record(transaction_pool_fetch_time); @@ -952,6 +998,35 @@ where span.record("flashblock_count", ctx.flashblock_index()); } + /// Finalizes the payload and carries the resting set into the next block. + /// + /// The finalized payload is the one returned to the payload resolver, so its hash is the parent + /// the next block is checked against. + fn finalize_block( + &self, + state: &mut State, + ctx: &BasePayloadBuilderCtx, + info: &mut ExecutionInfo, + best_txs: &mut NextBestFlashblocksTxs, + ) -> Result + where + DB: Database + AsRef

, + P: StateRootProvider + HashedPostStateProvider + StorageRootProvider, + { + // Start the end-of-block sweep before the state root so the two overlap. The set is only + // stored once the block is sealed, so a block that fails to seal does not hand it on. + let resting = self + .config + .resting_predicate_mode + .is_enabled() + .then(|| self.resting_predicates.finish(best_txs.take_resting_predicates())); + let payload = self.finalize_payload(state, ctx, info)?; + if let Some(resting) = resting { + self.resting_predicates.put(resting, payload.block().hash()); + } + Ok(payload) + } + /// Finalize the payload by computing the state root. fn finalize_payload( &self, @@ -1378,9 +1453,22 @@ where )) } +/// Time from starting the flashblock timer to its first tick. +/// +/// The resting carry runs after the schedule is derived, so it comes out of the first flashblock +/// instead of moving the schedule. It never takes more than the whole first flashblock: a first +/// tick in the past would fire missed ticks back to back and cancel flashblocks before they build +/// anything. A longer carry delays the schedule by the excess. +const fn first_flashblock_delay( + first_flashblock_offset: Duration, + resting_carry: Duration, +) -> Duration { + first_flashblock_offset.saturating_sub(resting_carry) +} + #[cfg(test)] mod tests { - use std::sync::Arc; + use std::{sync::Arc, time::Duration}; use alloy_consensus::{Header, Receipt}; use alloy_primitives::{Address, B256, Log, U256, map::foldhash::HashMap}; @@ -1393,7 +1481,7 @@ mod tests { use reth_provider::{StateProviderBox, noop::NoopProvider}; use reth_revm::{State, database::StateProviderDatabase}; - use super::{FlashblocksMetadata, build_block}; + use super::{FlashblocksMetadata, build_block, first_flashblock_delay}; use crate::{ExecutionInfo, flashblocks::context::BasePayloadBuilderCtx}; /// Creates a minimal [`BaseChainSpec`] with all L1 upgrades through Cancun @@ -1422,6 +1510,23 @@ mod tests { Arc::new(SealedHeader::seal_slow(header)) } + #[test] + fn resting_carry_comes_out_of_the_first_flashblock() { + let offset = Duration::from_millis(120); + assert_eq!(first_flashblock_delay(offset, Duration::ZERO), offset); + assert_eq!( + first_flashblock_delay(offset, Duration::from_millis(20)), + Duration::from_millis(100) + ); + } + + #[test] + fn resting_carry_longer_than_the_first_flashblock_never_starts_the_timer_in_the_past() { + let offset = Duration::from_millis(5); + assert_eq!(first_flashblock_delay(offset, offset), Duration::ZERO); + assert_eq!(first_flashblock_delay(offset, Duration::from_millis(300)), Duration::ZERO); + } + #[test] fn canonical_state_provider_uses_shared_cache_without_filling_misses() { let address = Address::random(); diff --git a/crates/builder/core/src/flashblocks/resting.rs b/crates/builder/core/src/flashblocks/resting.rs index 30d59b90dd..d02eda0ae1 100644 --- a/crates/builder/core/src/flashblocks/resting.rs +++ b/crates/builder/core/src/flashblocks/resting.rs @@ -1,31 +1,25 @@ //! Payload iterators that hold back validity transactions resting under an unchanged predicate. -use std::time::Duration; - use alloy_primitives::TxHash; +use base_execution_payload_builder::RestingStats; use base_execution_txpool::{ValidityConditions, ValidityPredicate}; use revm::state::EvmState; -/// Work an iterator spent holding back resting transactions since it was last asked. -#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] -pub struct RestingStats { - /// Resting transactions parked without being yielded. - pub parked: u64, - /// Time spent checking and parking resting transactions. - pub duration: Duration, -} - /// Lifecycle callbacks for validity transactions resting in a payload job. /// /// A transaction rests once the build loop finds one of its predicates unsatisfied. It stays -/// unsatisfied until a later commit in the same block changes the state that predicate reads, so -/// an iterator implementing this trait may hold it back in later flashblocks instead of yielding -/// it to be evaluated again. The default methods hold nothing back. +/// unsatisfied until a later commit changes the state that predicate reads, so the pool iterators +/// behind an implementation of this trait may hold it back in later flashblocks, or later blocks, +/// instead of yielding it to be evaluated again. The default methods hold nothing back. pub trait RestingPayloadTransactions { /// Records that `predicate` was unsatisfied for `transaction_hash` at the current build /// position. fn rest(&mut self, _transaction_hash: TxHash, _predicate: &ValidityPredicate) {} + /// Records that the build loop evaluated a transaction and is not leaving it parked under an + /// unsatisfied predicate. + fn release(&mut self, _transaction_hash: TxHash) {} + /// Wakes transactions resting on state changed by one committed transaction. fn record_committed_state(&mut self, _state: &EvmState) {} diff --git a/crates/builder/core/src/flashblocks/service.rs b/crates/builder/core/src/flashblocks/service.rs index 362148430c..48241361fe 100644 --- a/crates/builder/core/src/flashblocks/service.rs +++ b/crates/builder/core/src/flashblocks/service.rs @@ -62,6 +62,7 @@ impl FlashblocksServiceBuilder { ctx.provider().clone(), self.config.clone(), BuilderOutputs { payload_tx: built_payload_tx, ws_pub }, + ctx.task_executor().clone(), ); let payload_generator = BlockPayloadJobGenerator::with_builder( ctx.provider().clone(), diff --git a/crates/builder/core/src/lib.rs b/crates/builder/core/src/lib.rs index be9b2cef2a..846b22c878 100644 --- a/crates/builder/core/src/lib.rs +++ b/crates/builder/core/src/lib.rs @@ -22,13 +22,10 @@ pub use execution::{ mod execution_metering_mode; pub use execution_metering_mode::ExecutionMeteringMode; -mod resting_predicate_mode; -pub use resting_predicate_mode::RestingPredicateMode; - mod traits; pub use base_execution_payload_builder::{ MeteringProvider, NoopMeteringProvider, RejectionCache, ResourceMeteringConfig, - SharedMeteringProvider, + RestingPredicateMode, SharedMeteringProvider, }; pub use traits::{ClientBounds, NodeBounds, PayloadTxsBounds, PoolBounds}; @@ -46,7 +43,7 @@ pub use flashblocks::{ ParkedPredicateIndex, PayloadBuilder, PayloadHandler, PayloadJobDeadline, PayloadTransactionInvalidated, PredicateLoadTracker, PredicateReadRecorder, ResolvePayload, RestingPayloadTransactions, RestingStats, StateChangeEffects, ValidityPredicateEvaluation, - ValidityPredicateKey, + ValidityPredicateKey, flashblock_hold, }; mod extension; diff --git a/crates/builder/core/src/metrics.rs b/crates/builder/core/src/metrics.rs index 3dfa9ce8dc..99ca7a4fe4 100644 --- a/crates/builder/core/src/metrics.rs +++ b/crates/builder/core/src/metrics.rs @@ -94,6 +94,8 @@ base_metrics::define_metrics! { flashblocks_time_drift: histogram, #[describe("Time offset we used for first flashblock")] first_flashblock_time_offset: histogram, + #[describe("Time from the start of the payload job to flashblock scheduling, in seconds")] + flashblocks_schedule_delay: histogram, #[describe("Count of the number of times transactions had metering information")] metering_known_transaction: counter, #[describe("Count of the number of times transactions did not have any metering information")] @@ -104,6 +106,10 @@ base_metrics::define_metrics! { metering_store_ttl_expirations: counter, #[describe("Size of MeteringStore")] metering_store_size: gauge, + #[describe( + "Builder started with resource metering enabled but no resource-metering schedule configured" + )] + resource_metering_schedule_missing_total: counter, #[describe("Transactions inserted into the rejection cache")] rejection_cache_insertions: counter, #[describe("Transactions skipped by the rejection cache (P2P re-insertion prevented)")] @@ -189,10 +195,6 @@ base_metrics::define_metrics! { tx_accounts_modified: histogram, #[describe("Number of storage slots modified by a transaction (from EVM post-state)")] tx_storage_slots_modified: histogram, - #[describe("Resting validity transactions parked in the transaction iterator without reaching the build loop")] - resting_predicate_parked_total: counter, - #[describe("Resting validity transactions whose predicates matched when evaluated in shadow mode")] - resting_predicate_shadow_mismatches_total: counter, #[describe("Time per flashblock spent handling validity transactions, from yield to the predicate gate decision, including resting transactions parked in the iterator, in seconds")] validity_candidate_handling_duration: histogram, #[describe("Builder transaction events successfully enqueued")] diff --git a/crates/builder/core/src/resting_predicate_mode.rs b/crates/builder/core/src/resting_predicate_mode.rs deleted file mode 100644 index c36edea2dd..0000000000 --- a/crates/builder/core/src/resting_predicate_mode.rs +++ /dev/null @@ -1,31 +0,0 @@ -use clap::ValueEnum; - -/// Mode for holding back validity transactions that rest under an unchanged predicate. -/// -/// See [`RestingPayloadTransactions`](crate::RestingPayloadTransactions) for when a transaction -/// rests. -#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, ValueEnum)] -pub enum RestingPredicateMode { - /// Resting transactions are not tracked. - #[default] - Off, - /// Resting transactions are tracked and still evaluated. A resting transaction whose - /// predicates match is counted as a mismatch, which would be a missed inclusion under - /// enforcement. - Shadow, - /// Resting transactions are parked in the transaction iterator without reaching the build - /// loop until a commit changes the state their predicate reads. - Enforce, -} - -impl RestingPredicateMode { - /// Returns whether resting transactions are tracked. - pub const fn is_enabled(&self) -> bool { - matches!(self, Self::Shadow | Self::Enforce) - } - - /// Returns whether resting transactions are held back. - pub const fn is_enforced(&self) -> bool { - matches!(self, Self::Enforce) - } -} diff --git a/crates/builder/metering/src/ext.rs b/crates/builder/metering/src/ext.rs index 7e4e6e31cd..0177db9968 100644 --- a/crates/builder/metering/src/ext.rs +++ b/crates/builder/metering/src/ext.rs @@ -21,7 +21,14 @@ pub trait BaseApiExt { meter: MeterBundleResponse, ) -> RpcResult<()>; - /// Enables or disables resource metering. + /// Operator runtime override that enables or disables reads from the shared + /// [`MeteringStore`](crate::MeteringStore) (simulated bundle lookups and + /// the Flashblocks builder's use of stored metering data) in memory; + /// `base_setMeteringInformation` writes are still accepted while disabled, + /// and only reads return nothing. Because the native payload builder shares + /// this store, `setMeteringEnabled(false)` also makes native + /// `check_simulated_usage` a no-op while native `check_executed_usage` + /// continues running against `--payload.resource-metering-schedule`. #[method(name = "setMeteringEnabled")] async fn set_metering_enabled(&self, enabled: bool) -> RpcResult<()>; diff --git a/crates/builder/multiplex/src/service_builder.rs b/crates/builder/multiplex/src/service_builder.rs index dd0489ee96..a5dedde9f6 100644 --- a/crates/builder/multiplex/src/service_builder.rs +++ b/crates/builder/multiplex/src/service_builder.rs @@ -1,9 +1,7 @@ -use std::sync::Arc; - use base_builder_core::{BuilderConfig, FlashblocksServiceBuilder, NodeBounds, PoolBounds}; use base_execution_evm::BaseEvmConfig; use base_execution_firehose::OpFirehoseEvmConfig; -use base_execution_payload_builder::{ResourceMeteringConfig, config::BaseBuilderConfig}; +use base_execution_payload_builder::config::BaseBuilderConfig; use base_node_core::{ BaseConsensusBuilder, BaseEngineTypes, BaseExecutorBuilder, BaseNetworkBuilder, node::BasePoolBuilder, @@ -74,12 +72,10 @@ impl MultiplexingServiceBuilder { predicate_bucket_ordered_threshold: self .builder_config .predicate_bucket_ordered_threshold, - resource_metering: ResourceMeteringConfig { - provider: Arc::clone(&self.builder_config.metering_provider), - ..ResourceMeteringConfig::default() - }, + resource_metering: self.builder_config.resource_metering.clone(), rejection_cache: self.builder_config.rejection_cache.clone(), state_provider_metrics, + resting_predicate_mode: self.builder_config.resting_predicate_mode, } } } @@ -100,6 +96,7 @@ where !(self.routing_config.cutover_enabled && self.basic_only), "payload builder cutover and basic-only modes are mutually exclusive" ); + self.builder_config.record_startup_metrics(); // Reth's own flag drives build-loop state-read timing, so the builder and the engine's // validation path are instrumented by the same switch. @@ -118,7 +115,8 @@ where ctx.provider().clone(), evm_config.inner.clone(), self.native_payload_config(builder_config.state_provider_metrics), - ); + ) + .with_task_executor(ctx.task_executor().clone()); let payload_config = ctx.config().builder.clone(); let payload_job_config = BasicPayloadJobGeneratorConfig::default() @@ -209,7 +207,10 @@ impl BasePayloadServiceBuilder for MultiplexingServiceBuilder { #[cfg(test)] mod tests { + use std::sync::Arc; + use alloy_primitives::TxHash; + use base_execution_payload_builder::{ResourceMeteringConfig, ResourceMeteringSchedule}; use super::*; @@ -222,6 +223,42 @@ mod tests { let native = MultiplexingServiceBuilder::new(builder_config).native_payload_config(false); assert!(native.rejection_cache.contains_key(&hash)); + assert!(!native.resource_metering.enabled); + assert!(native.resource_metering.schedule.is_empty()); + assert!(!native.resource_metering.is_active()); + assert!(Arc::ptr_eq(&native.resource_metering.provider, &provider)); + } + + #[test] + fn native_payload_config_propagates_configured_resource_metering_schedule() { + let schedule = ResourceMeteringSchedule::from_json( + r#"{ + "version": 1, + "dimensions": [{ + "name": "compute", + "blockLimit": 1000, + "transactionLimit": 500, + "baseGasWeight": 1, + "operations": [{"name": "SSTORE", "gasUsedWeight": 2, "countCost": 10}], + "dryRun": true + }] + }"#, + ) + .expect("valid schedule"); + + let mut builder_config = BuilderConfig::default(); + let provider = Arc::clone(&builder_config.metering_provider); + builder_config.resource_metering = ResourceMeteringConfig { + enabled: true, + schedule: Arc::new(schedule), + provider: Arc::clone(&provider), + }; + + let native = MultiplexingServiceBuilder::new(builder_config).native_payload_config(false); + assert!(native.resource_metering.enabled); + assert!(native.resource_metering.is_active()); + assert_eq!(native.resource_metering.schedule.dimensions.len(), 1); + assert_eq!(native.resource_metering.schedule.dimensions[0].name, "compute"); assert!(Arc::ptr_eq(&native.resource_metering.provider, &provider)); } } diff --git a/crates/consensus/gossip/src/config.rs b/crates/consensus/gossip/src/config.rs index 4868268d86..30315a0ab5 100644 --- a/crates/consensus/gossip/src/config.rs +++ b/crates/consensus/gossip/src/config.rs @@ -22,7 +22,7 @@ pub const MAX_GOSSIP_SIZE: usize = 10 * (1 << 20); pub const MIN_GOSSIP_SIZE: usize = 66; /// The maximum outbound queue. -pub const MAX_OUTBOUND_QUEUE: usize = 256; +pub const MAX_OUTBOUND_QUEUE: usize = 2560; /// The maximum validate queue. pub const MAX_VALIDATE_QUEUE: usize = 256; diff --git a/crates/consensus/service/src/actors/sequencer/actor.rs b/crates/consensus/service/src/actors/sequencer/actor.rs index c0b1dbd7ff..b4f2f0d529 100644 --- a/crates/consensus/service/src/actors/sequencer/actor.rs +++ b/crates/consensus/service/src/actors/sequencer/actor.rs @@ -233,36 +233,41 @@ where loop { let engine_client = Arc::clone(&self.engine_client); let shadow_cycle_coordinated = self.is_shadow_sequencer(); - select! { - biased; - _ = self.cancellation_token.cancelled() => return Ok(()), - Some(query) = self.admin_api_rx.recv() => { - let was_active = self.is_active; - self.handle_admin_query(next_payload, query).await; - if !was_active && self.is_active && !shadow_cycle_coordinated { - return Ok(()); - } + let reset = async { + if shadow_cycle_coordinated { + engine_client + .reset_engine_forkchoice_coordinated(ResetReason::ShadowCycle) + .await + } else { + engine_client.reset_engine_forkchoice(ResetReason::SequencerStartup).await } - result = async { - if shadow_cycle_coordinated { - engine_client - .reset_engine_forkchoice_coordinated(ResetReason::ShadowCycle) - .await - } else { - engine_client - .reset_engine_forkchoice(ResetReason::SequencerStartup) - .await - } - } => match result { - Ok(()) => return Ok(()), - Err(EngineClientError::ELSyncing) => { - info!(target: "sequencer", "EL sync or canonical catch-up in progress; deferring initial engine reset"); - } - Err(err) => { - error!(target: "sequencer", error = ?err, "Failed to send reset request to engine"); - return Err(err.into()); + }; + // Once queued, the engine reset outlives this future. Keep its response receiver + // across admin queries so they cannot discard completion and trigger another reset. + tokio::pin!(reset); + let result = loop { + select! { + biased; + _ = self.cancellation_token.cancelled() => return Ok(()), + Some(query) = self.admin_api_rx.recv() => { + let was_active = self.is_active; + self.handle_admin_query(next_payload, query).await; + if !was_active && self.is_active && !shadow_cycle_coordinated { + return Ok(()); + } } - }, + result = &mut reset => break result, + } + }; + match result { + Ok(()) => return Ok(()), + Err(EngineClientError::ELSyncing) => { + info!(target: "sequencer", "EL sync or canonical catch-up in progress; deferring initial engine reset"); + } + Err(err) => { + error!(target: "sequencer", error = ?err, "Failed to send reset request to engine"); + return Err(err.into()); + } } // Wait one block time before retrying the reset, but service admin queries // and honour cancellation throughout the backoff window. @@ -810,13 +815,79 @@ mod tests { use super::*; use crate::{ - EngineProcessor, EngineRequestReceiver, MockConductor, MockEngineDerivationClient, - MockOriginSelector, MockUnsafePayloadGossipClient, QueuedSequencerEngineClient, - SequencerEngineRequestCoordinator, + EngineActorRequest, EngineProcessor, EngineRequestReceiver, MockConductor, + MockEngineDerivationClient, MockOriginSelector, MockUnsafePayloadGossipClient, + QueuedSequencerEngineClient, SequencerEngineRequestCoordinator, actors::sequencer::tests::test_actor, test_utils::{EngineClientCall, FakeEngineClient, ScriptedForkchoiceResponse}, }; + #[tokio::test(start_paused = true)] + async fn startup_reset_survives_admin_queries() { + let config = Arc::new(RollupConfig { block_time: 2, ..Default::default() }); + let (request_tx, mut request_rx) = mpsc::channel(8); + let (_head_tx, head_rx) = watch::channel(L2BlockInfo::default()); + let (_state_tx, state_rx) = watch::channel(TestEngineStateBuilder::new().build()); + let engine_client = + Arc::new(QueuedSequencerEngineClient::new(request_tx, head_rx, state_rx)); + let (admin_tx, admin_rx) = mpsc::channel(8); + let cancellation = CancellationToken::new(); + let recovery_mode = RecoveryModeGuard::new(false); + let actor = SequencerActor { + admin_api_rx: admin_rx, + builder: PayloadBuilder { + attributes_builder: TestAttributesBuilder::default(), + engine_client: Arc::clone(&engine_client), + origin_selector: MockOriginSelector::new(), + recovery_mode: recovery_mode.clone(), + rollup_config: Arc::clone(&config), + }, + cancellation_token: cancellation.clone(), + conductor: None::, + engine_client, + is_active: false, + mode: NodeOperatingMode::Sequencer, + shadow_funding: None, + recovery_mode, + rollup_config: config, + seal_offset: base_protocol::DEFAULT_SEAL_OFFSET, + unsafe_payload_gossip_client: MockUnsafePayloadGossipClient::new(), + sealer: None, + pending_stop: None, + }; + let actor = tokio::spawn(actor.start(())); + let EngineActorRequest::ResetRequest(reset) = + tokio::time::timeout(Duration::from_secs(1), request_rx.recv()).await.unwrap().unwrap() + else { + panic!("expected the startup reset"); + }; + assert_eq!(reset.reason, ResetReason::SequencerStartup); + + // Receiving the request first ensures that the reset is in flight, rather than + // the admin query winning before it is ever submitted. Hold its reply beyond + // the retry backoff while checking that read-only admin queries stay responsive. + for _ in 0..3 { + tokio::time::advance(Duration::from_secs(1)).await; + let (result_tx, result_rx) = oneshot::channel(); + admin_tx.send(SequencerAdminQuery::SequencerActive(result_tx)).await.unwrap(); + assert!( + !tokio::time::timeout(Duration::from_secs(1), result_rx) + .await + .unwrap() + .unwrap() + .unwrap() + ); + } + + let delivered = reset.result_tx.send(Ok(())).await; + let next_request = tokio::time::timeout(Duration::from_secs(4), request_rx.recv()).await; + cancellation.cancel(); + actor.await.unwrap().unwrap(); + + assert!(delivered.is_ok(), "admin queries must not abandon the in-flight startup reset"); + assert!(next_request.is_err(), "a successful startup reset must not be resubmitted"); + } + #[rstest] #[tokio::test(start_paused = true)] async fn conductor_takeover_without_gossip_starts_building( diff --git a/crates/consensus/service/src/actors/sequencer/engine_request_coordinator.rs b/crates/consensus/service/src/actors/sequencer/engine_request_coordinator.rs index 0bc7efaca9..b5eeb53531 100644 --- a/crates/consensus/service/src/actors/sequencer/engine_request_coordinator.rs +++ b/crates/consensus/service/src/actors/sequencer/engine_request_coordinator.rs @@ -30,7 +30,6 @@ use crate::{ ResetRequestOutcome, actors::engine::ResetOutcome, }; -const MAX_SEQUENCER_EXTERNAL_UNSAFE_GAP: u64 = 300; const SEQUENCER_EL_SYNC_PROBE_INTERVAL: Duration = Duration::from_secs(5); #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -103,6 +102,9 @@ where sequencer_stopped: bool, sequencer_kind: SequencerKind, unsafe_head_tx: watch::Sender, + /// Recent canonical payloads retained during catch-up, and the largest gap between the unsafe + /// head and external gossip still admitted while regular. Both span the same wall-clock window. + catchup_capacity: usize, } impl @@ -119,6 +121,7 @@ where sequencer_stopped: bool, unsafe_head_tx: watch::Sender, ) -> Self { + let catchup_capacity = CanonicalUnsafeCatchup::max_payloads_for(processor.rollup()); Self { processor, sequencer_state: SequencerEngineState::Regular, @@ -126,6 +129,7 @@ where sequencer_stopped, sequencer_kind: mode.into(), unsafe_head_tx, + catchup_capacity, } } @@ -337,7 +341,7 @@ where match &mut self.sequencer_state { SequencerEngineState::ShadowActive(gate) => gate.reanchor(anchor), SequencerEngineState::CatchingUp { catchup } => { - *catchup = CanonicalUnsafeCatchup::default(); + *catchup = CanonicalUnsafeCatchup::new(self.catchup_capacity); } SequencerEngineState::Regular | SequencerEngineState::IsolatedActive => {} } @@ -380,15 +384,22 @@ where Ok(()) } - async fn advance_canonical_catchup(&mut self) -> Result<(), EngineError> { + /// Applies one bounded batch of the buffered canonical suffix. + /// + /// Returns whether more buffered payloads may remain, so the caller can serve queued requests + /// between batches instead of blocking on one long run. + async fn advance_canonical_catchup(&mut self) -> Result { let anchor = self.processor.engine_state().sync_state.unsafe_head(); let payloads = match &mut self.sequencer_state { - SequencerEngineState::CatchingUp { catchup } => catchup.contiguous_payloads(anchor), - _ => return Ok(()), + SequencerEngineState::CatchingUp { catchup } => { + catchup.contiguous_payloads(anchor, CanonicalUnsafeCatchup::MAX_APPLY_BATCH) + } + _ => return Ok(false), }; if payloads.is_empty() { - return Ok(()); + return Ok(false); } + let more_pending = payloads.len() == CanonicalUnsafeCatchup::MAX_APPLY_BATCH; debug!( target: "engine", @@ -410,6 +421,7 @@ where if let SequencerEngineState::CatchingUp { catchup } = &mut self.sequencer_state { catchup.commit(head); } + Ok(more_pending) } Err(error) => { debug!( @@ -418,15 +430,14 @@ where severity = ?error.severity(), "Ordered canonical unsafe payload application deferred" ); - return match error.severity() { + match error.severity() { EngineTaskErrorSeverity::Temporary | EngineTaskErrorSeverity::Deferred => { - Ok(()) + Ok(false) } _ => Err(EngineError::CriticalEngineTask(format!("{error:?}"))), - }; + } } } - Ok(()) } } @@ -474,7 +485,7 @@ where (SequencerKind::Shadow, _, _) | (_, BootstrapRole::ConductorFollower, false) => { SequencerEngineState::CatchingUp { - catchup: CanonicalUnsafeCatchup::default(), + catchup: CanonicalUnsafeCatchup::new(self.catchup_capacity), } } (kind, _, _) => { @@ -523,11 +534,11 @@ where } if let SequencerEngineState::CatchingUp { catchup } = &mut self.sequencer_state { - *catchup = CanonicalUnsafeCatchup::default(); + *catchup = CanonicalUnsafeCatchup::new(self.catchup_capacity); } } - self.advance_canonical_catchup().await?; + let more_catchup_pending = self.advance_canonical_catchup().await?; // If the unsafe head has updated, propagate it to the outbound channels. self.unsafe_head_tx.send_if_modified(|val| { @@ -543,7 +554,18 @@ where let recv_wait_timer = base_metrics::timed!( EngineMetrics::engine_processor_recv_wait_duration_seconds() ); - let recv_result = if self.processor.engine_state().el_sync_finished { + let recv_result = if more_catchup_pending { + // Serve any queued request, then continue the next catch-up batch without + // waiting for new traffic. + match request_channel.try_recv() { + Ok(request) => Some(request), + Err(mpsc::error::TryRecvError::Empty) => { + drop(recv_wait_timer); + continue; + } + Err(mpsc::error::TryRecvError::Disconnected) => None, + } + } else if self.processor.engine_state().el_sync_finished { request_channel.recv().await } else { tokio::select! { @@ -700,18 +722,19 @@ where let block_number = envelope.execution_payload.block_number(); let unsafe_head = self.processor.engine_state().sync_state.unsafe_head(); + let max_external_unsafe_gap = self.catchup_capacity as u64; let block_gap = block_number.checked_sub(unsafe_head.block_info.number); - if block_gap.is_some_and(|gap| { - gap > 0 && gap <= MAX_SEQUENCER_EXTERNAL_UNSAFE_GAP - }) { + if block_gap + .is_some_and(|gap| gap > 0 && gap <= max_external_unsafe_gap) + { info!( target: "engine", block_number, block_hash = %envelope.execution_payload.block_hash(), parent_hash = %envelope.execution_payload.parent_hash(), block_gap = ?block_gap, - max_external_unsafe_gap = MAX_SEQUENCER_EXTERNAL_UNSAFE_GAP, + max_external_unsafe_gap, "Sequencer enqueuing external unsafe payload within gap limit" ); self.processor @@ -723,7 +746,7 @@ where block_hash = %envelope.execution_payload.block_hash(), parent_hash = %envelope.execution_payload.parent_hash(), block_gap = ?block_gap, - max_external_unsafe_gap = MAX_SEQUENCER_EXTERNAL_UNSAFE_GAP, + max_external_unsafe_gap, unsafe_head_number = unsafe_head.block_info.number, unsafe_head_hash = %unsafe_head.block_info.hash, "Sequencer dropped external unsafe payload outside gap limit" @@ -800,25 +823,28 @@ where #[cfg(test)] mod tests { - use std::sync::Arc; + use std::{sync::Arc, time::Duration}; - use alloy_primitives::B256; + use alloy_eips::eip2718::Encodable2718; + use alloy_primitives::{B256, Bytes}; use alloy_rpc_types_engine::ExecutionPayloadV1; + use base_common_consensus::{BaseTxEnvelope, TxDeposit}; use base_common_genesis::RollupConfig; use base_common_rpc_types_engine::{BaseExecutionPayload, BaseExecutionPayloadEnvelope}; use base_consensus_engine::{ Engine, EngineState, test_utils::{MockEngineClient, TestEngineStateBuilder, test_engine_client_builder}, }; - use base_protocol::{BlockInfo, L2BlockInfo}; + use base_protocol::{BlockInfo, L1BlockInfoBedrock, L2BlockInfo}; use jsonrpsee::core::ClientError; use rstest::rstest; - use tokio::sync::watch; + use tokio::sync::{mpsc, watch}; use super::{BootstrapRole, SequencerEngineRequestCoordinator, SequencerEngineState}; use crate::{ - CanonicalUnsafeCatchup, Conductor, ConductorError, EngineProcessor, MockConductor, - MockEngineDerivationClient, NodeOperatingMode, ShadowReconciliationGate, + CanonicalUnsafeCatchup, Conductor, ConductorError, EngineActorRequest, EngineProcessor, + EngineRequestReceiver, MockConductor, MockEngineDerivationClient, NodeOperatingMode, + ShadowReconciliationGate, }; fn coordinator( @@ -844,6 +870,72 @@ mod tests { SequencerEngineRequestCoordinator::new(processor, mode, conductor, stopped, unsafe_head_tx) } + fn gossip_payload(number: u64) -> BaseExecutionPayloadEnvelope { + let deposit: Bytes = BaseTxEnvelope::from(TxDeposit { + input: L1BlockInfoBedrock::default().encode_calldata(), + ..Default::default() + }) + .encoded_2718() + .into(); + BaseExecutionPayloadEnvelope { + execution_payload: BaseExecutionPayload::V1(ExecutionPayloadV1 { + block_number: number, + block_hash: B256::with_last_byte(1), + parent_hash: B256::ZERO, + fee_recipient: Default::default(), + state_root: B256::ZERO, + receipts_root: B256::ZERO, + logs_bloom: Default::default(), + prev_randao: B256::ZERO, + gas_limit: 30_000_000, + gas_used: 0, + timestamp: number, + extra_data: Default::default(), + base_fee_per_gas: Default::default(), + transactions: vec![deposit], + }), + parent_beacon_block_root: None, + } + } + + #[rstest] + #[case::beyond_legacy_gap(301, true)] + #[case::within_catchup_window(2_999, true)] + #[case::beyond_catchup_window(3_001, false)] + #[tokio::test] + async fn regular_gossip_gap_spans_the_catchup_window(#[case] gap: u64, #[case] admitted: bool) { + let coordinator = coordinator(false, false, None); + let client = Arc::clone(coordinator.processor.client()); + let (request_tx, request_rx) = mpsc::channel(8); + let handle = coordinator.start(request_rx); + + request_tx + .send(EngineActorRequest::ProcessUnsafeL2BlockRequest(Box::new(gossip_payload(gap)))) + .await + .expect("gossip request"); + // Requests are handled in order, so a following barrier means the gossip was handled. + let (result_tx, mut result_rx) = mpsc::channel(1); + request_tx + .send(EngineActorRequest::PrepareSequencerStart { + expected_hash: B256::with_last_byte(1), + result_tx, + }) + .await + .expect("request barrier"); + // The reply may be deferred while an admitted insert is in flight; only ordering matters. + let _ = tokio::time::timeout(Duration::from_millis(500), result_rx.recv()).await; + // An admitted insert reaches the engine asynchronously. + for _ in 0..300 { + if client.last_new_payload_v2().await.is_some() { + break; + } + tokio::time::sleep(Duration::from_millis(10)).await; + } + handle.abort(); + + assert_eq!(client.last_new_payload_v2().await.is_some(), admitted); + } + #[tokio::test] async fn active_sequencer_without_conductor_starts_regular() { let coordinator = coordinator(false, false, None); diff --git a/crates/consensus/service/src/actors/sequencer/shadow_reconciliation.rs b/crates/consensus/service/src/actors/sequencer/shadow_reconciliation.rs index 5090adcd6e..2a11f5b75f 100644 --- a/crates/consensus/service/src/actors/sequencer/shadow_reconciliation.rs +++ b/crates/consensus/service/src/actors/sequencer/shadow_reconciliation.rs @@ -1,8 +1,12 @@ //! Request routing and buffered canonical inputs for shadow reconciliation. -use std::collections::{BTreeMap, VecDeque}; +use std::{ + collections::{BTreeMap, VecDeque}, + time::Duration, +}; use alloy_primitives::B256; +use base_common_genesis::{DenimTimestampSchedule, RollupConfig}; use base_common_rpc_types_engine::BaseExecutionPayloadEnvelope; use base_consensus_engine::ConsolidateInput; use base_protocol::L2BlockInfo; @@ -62,17 +66,67 @@ pub enum SequencerEngineState { } /// Rolling canonical unsafe payloads retained while safe derivation catches up. -#[derive(Debug, Default)] +#[derive(Debug)] pub struct CanonicalUnsafeCatchup { + max_payloads: usize, payloads: BTreeMap, observations: BTreeMap, highest_observed: Option<(u64, B256)>, faulted: bool, } +impl Default for CanonicalUnsafeCatchup { + fn default() -> Self { + Self::new(Self::MAX_PAYLOADS) + } +} + impl CanonicalUnsafeCatchup { - /// Maximum recent canonical unsafe payloads retained during catch-up. - pub const MAX_PAYLOADS: usize = SequencerConfig::MAX_SHADOW_BLOCKS_PER_CYCLE as usize; + /// Wall-clock span of recent canonical gossip retained during catch-up. + /// + /// A restarted follower only connects to the buffer once derived safe blocks reach the oldest + /// retained payload, so the window must exceed the unsafe-to-safe derivation latency (batch + /// submission, L1 inclusion, and derivation) with ample margin, independent of block time. + pub const CATCHUP_WINDOW: Duration = Duration::from_secs(600); + + /// Payloads retained during catch-up at the fastest supported block cadence. + pub const MAX_PAYLOADS: usize = (Self::CATCHUP_WINDOW.as_millis() + / DenimTimestampSchedule::DENIM_BLOCK_INTERVAL_MILLIS as u128) + as usize; + + /// Most buffered payloads applied to the engine before it serves other requests. + pub const MAX_APPLY_BATCH: usize = SequencerConfig::MAX_SHADOW_BLOCKS_PER_CYCLE as usize; + + /// Creates a catch-up buffer retaining at most `max_payloads` recent payloads. + pub const fn new(max_payloads: usize) -> Self { + Self { + max_payloads, + payloads: BTreeMap::new(), + observations: BTreeMap::new(), + highest_observed: None, + faulted: false, + } + } + + /// Returns the number of payloads that span [`Self::CATCHUP_WINDOW`] at `rollup`'s block time. + /// + /// A chain with Denim scheduled is sized for the Denim cadence so the window also holds + /// through the transition from whole-second blocks. + pub fn max_payloads_for(rollup: &RollupConfig) -> usize { + let legacy_interval_millis = rollup.block_time.saturating_mul(1_000); + let interval_millis = + if legacy_interval_millis == 0 || rollup.denim_timestamp_schedule().is_some() { + DenimTimestampSchedule::DENIM_BLOCK_INTERVAL_MILLIS + } else { + legacy_interval_millis + }; + ((Self::CATCHUP_WINDOW.as_millis() / u128::from(interval_millis)) as usize).max(1) + } + + /// Returns the maximum number of payloads retained. + pub const fn max_payloads(&self) -> usize { + self.max_payloads + } /// Retains an authenticated canonical payload in the rolling recent window. pub fn buffer_payload(&mut self, envelope: BaseExecutionPayloadEnvelope) { @@ -92,25 +146,31 @@ impl CanonicalUnsafeCatchup { self.highest_observed = Some((number, hash)); } self.payloads.insert(number, envelope); - while self.payloads.len() > Self::MAX_PAYLOADS { + while self.payloads.len() > self.max_payloads { self.payloads.pop_first(); } - while self.observations.len() > Self::MAX_PAYLOADS { + while self.observations.len() > self.max_payloads { self.observations.pop_first(); } } - /// Returns the contiguous canonical suffix that can extend `anchor` now. + /// Returns at most `limit` payloads of the contiguous canonical suffix that can extend + /// `anchor` now. + /// + /// Bounding the batch keeps the engine loop responsive and the cloned working set small while + /// a long run is applied; the remainder is returned by later calls as the anchor advances. pub fn contiguous_payloads( &mut self, anchor: L2BlockInfo, + limit: usize, ) -> Vec { self.payloads.retain(|number, _| *number > anchor.block_info.number); let mut number = anchor.block_info.number.saturating_add(1); let mut parent = anchor.block_info.hash; let mut payloads = Vec::new(); - while let Some(payload) = self.payloads.get(&number) { + while payloads.len() < limit { + let Some(payload) = self.payloads.get(&number) else { break }; if payload.execution_payload.parent_hash() != parent { break; } @@ -365,12 +425,14 @@ impl ShadowReconciliationGate { mod tests { use alloy_primitives::{Address, B256, Bloom, U256}; use alloy_rpc_types_engine::ExecutionPayloadV1; + use base_common_genesis::{BaseUpgradeConfig, RollupConfig, UpgradeConfig}; use base_common_rpc_types_engine::{BaseExecutionPayload, BaseExecutionPayloadEnvelope}; use base_consensus_engine::{ConsolidateInput, test_utils::TestAttributesBuilder}; use base_protocol::{BlockInfo, L2BlockInfo}; + use rstest::rstest; use super::{CanonicalUnsafeCatchup, ShadowReconciliationGate}; - use crate::EngineClientError; + use crate::{EngineClientError, SequencerConfig}; fn head(number: u64, hash: B256) -> L2BlockInfo { L2BlockInfo { @@ -408,7 +470,7 @@ mod tests { catchup.buffer_payload(payload(12, B256::with_last_byte(11), B256::with_last_byte(12))); catchup.buffer_payload(payload(11, safe.block_info.hash, B256::with_last_byte(11))); - let payloads = catchup.contiguous_payloads(safe); + let payloads = catchup.contiguous_payloads(safe, usize::MAX); assert_eq!(payloads.len(), 2); assert_eq!(payloads[0].execution_payload.block_number(), 11); assert_eq!(payloads[1].execution_payload.block_number(), 12); @@ -434,7 +496,118 @@ mod tests { assert_eq!(catchup.observations.len(), CanonicalUnsafeCatchup::MAX_PAYLOADS); assert_eq!(catchup.observations.first_key_value().map(|(number, _)| *number), Some(3)); let anchor = head(2, B256::from(U256::from(2))); - assert_eq!(catchup.contiguous_payloads(anchor).len(), CanonicalUnsafeCatchup::MAX_PAYLOADS); + assert_eq!( + catchup.contiguous_payloads(anchor, usize::MAX).len(), + CanonicalUnsafeCatchup::MAX_PAYLOADS + ); + } + + #[test] + fn catchup_connects_when_derivation_lags_beyond_the_shadow_cycle_limit() { + // Derivation leaves the safe head more than the shadow cycle limit behind the latest + // gossip, which is what happens to a restarted follower when blocks are fast. + let lag = SequencerConfig::MAX_SHADOW_BLOCKS_PER_CYCLE + 14; + let anchor_number = 50; + let tip = anchor_number + lag; + let mut catchup = CanonicalUnsafeCatchup::default(); + let mut parent = B256::from(U256::from(anchor_number)); + for number in anchor_number + 1..=tip { + let hash = B256::from(U256::from(number)); + catchup.buffer_payload(payload(number, parent, hash)); + parent = hash; + } + let anchor = head(anchor_number, B256::from(U256::from(anchor_number))); + assert!(!catchup.is_complete(anchor, anchor)); + + let suffix = catchup.contiguous_payloads(anchor, usize::MAX); + + assert_eq!(suffix.len() as u64, lag); + let applied = head(tip, B256::from(U256::from(tip))); + catchup.commit(applied); + assert!(catchup.is_complete(applied, applied)); + } + + #[rstest] + #[case::denim_default(RollupConfig::default(), 3_000)] + #[case::legacy_block_time(rollup_config(2, None), 300)] + #[case::denim_scheduled(rollup_config(2, Some(1_000)), 3_000)] + #[case::block_time_longer_than_window(rollup_config(601, None), 1)] + fn catchup_window_spans_the_same_time_at_any_block_time( + #[case] rollup: RollupConfig, + #[case] expected: usize, + ) { + assert_eq!(CanonicalUnsafeCatchup::max_payloads_for(&rollup), expected); + } + + #[test] + fn full_window_is_applied_in_bounded_batches() { + let total = CanonicalUnsafeCatchup::MAX_PAYLOADS as u64; + let mut catchup = CanonicalUnsafeCatchup::default(); + let mut parent = B256::ZERO; + for number in 1..=total { + let hash = B256::from(U256::from(number)); + catchup.buffer_payload(payload(number, parent, hash)); + parent = hash; + } + + let mut anchor = head(0, B256::ZERO); + let mut applied = 0; + while !catchup.is_complete(anchor, anchor) { + let batch = + catchup.contiguous_payloads(anchor, CanonicalUnsafeCatchup::MAX_APPLY_BATCH); + assert!(!batch.is_empty() && batch.len() <= CanonicalUnsafeCatchup::MAX_APPLY_BATCH); + applied += batch.len() as u64; + let last = batch.last().unwrap().execution_payload.block_number(); + anchor = head(last, B256::from(U256::from(last))); + catchup.commit(anchor); + } + + assert_eq!(applied, total); + } + + #[test] + fn failed_batch_resumes_after_applied_prefix() { + let mut catchup = CanonicalUnsafeCatchup::new(10); + let mut parent = B256::ZERO; + for number in 1..=10 { + let hash = B256::from(U256::from(number)); + catchup.buffer_payload(payload(number, parent, hash)); + parent = hash; + } + + // The engine applied block 4 before failing without a commit; the retry starts after it. + let anchor = head(4, B256::from(U256::from(4))); + let retry = catchup.contiguous_payloads(anchor, 3); + + let numbers: Vec<_> = + retry.iter().map(|payload| payload.execution_payload.block_number()).collect(); + assert_eq!(numbers, vec![5, 6, 7]); + } + + #[test] + fn catchup_retains_only_its_configured_capacity() { + let mut catchup = CanonicalUnsafeCatchup::new(4); + let mut parent = B256::ZERO; + for number in 1..=6 { + let hash = B256::from(U256::from(number)); + catchup.buffer_payload(payload(number, parent, hash)); + parent = hash; + } + + assert_eq!(catchup.max_payloads(), 4); + assert_eq!(catchup.payloads.keys().copied().collect::>(), vec![3, 4, 5, 6]); + assert_eq!(catchup.observations.len(), 4); + } + + fn rollup_config(block_time: u64, denim: Option) -> RollupConfig { + RollupConfig { + block_time, + upgrades: UpgradeConfig { + base: BaseUpgradeConfig { denim, ..Default::default() }, + ..Default::default() + }, + ..Default::default() + } } #[test] @@ -458,7 +631,7 @@ mod tests { let mut catchup = CanonicalUnsafeCatchup::default(); catchup.buffer_payload(payload(11, B256::with_last_byte(10), B256::with_last_byte(11))); let safe = head(12, B256::with_last_byte(12)); - assert!(catchup.contiguous_payloads(safe).is_empty()); + assert!(catchup.contiguous_payloads(safe, usize::MAX).is_empty()); assert!(catchup.is_complete(safe, safe)); } @@ -468,7 +641,7 @@ mod tests { let mut catchup = CanonicalUnsafeCatchup::default(); catchup.buffer_payload(payload(11, B256::with_last_byte(10), B256::with_last_byte(11))); let private = head(12, B256::with_last_byte(99)); - assert!(catchup.contiguous_payloads(private).is_empty()); + assert!(catchup.contiguous_payloads(private, usize::MAX).is_empty()); assert!(!catchup.is_complete(private, head(10, B256::with_last_byte(10)))); } diff --git a/crates/execution/cli/src/standard_node.rs b/crates/execution/cli/src/standard_node.rs index 024a1f6006..b1da9e6f32 100644 --- a/crates/execution/cli/src/standard_node.rs +++ b/crates/execution/cli/src/standard_node.rs @@ -52,10 +52,12 @@ use crate::upgrade_signal::{ pub struct MeteringArgs { /// Enable metering RPC for transaction bundle simulation. /// - /// Turns on `base_meterBundle`. The native payload builder throttles - /// transactions against resource-unit budgets only when this is set and - /// `--payload.resource-metering-schedule` is a non-empty file. The - /// Flashblocks builder uses `--builder.enable-resource-metering` instead. + /// Turns on `base_meterBundle` on standalone execution and RPC nodes + /// (`base-reth-node`, `base execution`, `base rpc`). Their native payload + /// builder throttles transactions against resource-unit budgets only when + /// this is set and `--payload.resource-metering-schedule` is a non-empty + /// file. `base-builder` and `base sequencer` gate both Flashblocks metering + /// and the native schedule on `--builder.enable-resource-metering` instead. #[arg(long = "enable-metering", env = "ENABLE_METERING", value_name = "ENABLE_METERING")] pub enable_metering: bool, @@ -97,11 +99,16 @@ pub struct MeteringArgs { /// CLI arguments for payload resource metering. #[derive(Debug, Clone, PartialEq, Eq, clap::Args)] pub struct ResourceMeteringArgs { - /// JSON file containing the startup resource-metering schedule. + /// JSON file containing the startup resource-metering schedule for + /// standalone execution and RPC nodes (`base-reth-node`, `base execution`, + /// `base rpc`). /// - /// The native payload builder throttles transactions against this schedule - /// when `--enable-metering` is set and the file is non-empty. Per-dimension - /// `dryRun` observes a budget without excluding transactions. + /// On those nodes, the native payload builder throttles transactions + /// against this schedule when `--enable-metering` is set and the file is + /// non-empty (`base-builder` and `base sequencer` gate builder metering as a + /// whole on `--builder.enable-resource-metering`). Each dimension must + /// explicitly set `dryRun` (`true` to observe a budget without excluding + /// transactions, `false` to enforce). #[arg(long = "payload.resource-metering-schedule", env = "PAYLOAD_RESOURCE_METERING_SCHEDULE")] pub resource_metering_schedule: Option, @@ -589,6 +596,7 @@ impl From<&StandardNodeArgs> for TxForwardingConfig { Self::new(args.rpc.builder_rpc_urls.clone()) .with_resend_after_ms(args.rpc.tx_forwarding_resend_after_ms) + .with_validity_lifetime_secs(args.rpc.validity_max_expiry_secs) .with_max_batch_size(args.rpc.tx_forwarding_batch_size) .with_max_rps(args.rpc.tx_forwarding_max_rps) .with_fifo_percent(args.rpc.tx_forwarding_fifo_percent) @@ -1667,7 +1675,8 @@ mod tests { {"name": "SSTORE", "countCost": 1}, {"name": "STATE_NEW_STORAGE_SLOT", "countCost": 1}, {"name": "NOT_AN_OPCODE", "countCost": 1} - ] + ], + "dryRun": true }] }"#, ) diff --git a/crates/execution/flashblocks-node/Cargo.toml b/crates/execution/flashblocks-node/Cargo.toml index 297e80cd29..1f590fa74c 100644 --- a/crates/execution/flashblocks-node/Cargo.toml +++ b/crates/execution/flashblocks-node/Cargo.toml @@ -15,6 +15,8 @@ workspace = true # workspace base-flashblocks.workspace = true base-node-runner.workspace = true +base-execution-txpool.workspace = true +base-common-flashblocks.workspace = true # reth reth-chain-state.workspace = true @@ -43,7 +45,6 @@ tracing = { workspace = true, features = ["std"] } # test-utils (optional) eyre = { workspace = true, optional = true } base-test-utils = { workspace = true, optional = true } -base-common-flashblocks = { workspace = true, optional = true } derive_more = { workspace = true, features = ["deref"], optional = true } [dev-dependencies] @@ -51,6 +52,7 @@ derive_more = { workspace = true, features = ["deref"], optional = true } base-test-utils = { workspace = true, features = ["contracts"] } base-flashblocks-node = { path = ".", features = ["test-utils"] } base-node-runner = { workspace = true, features = ["test-utils"] } +base-txpool-rpc.workspace = true # alloy (test) alloy-signer.workspace = true @@ -107,7 +109,6 @@ test-utils = [ "dep:alloy-primitives", "dep:alloy-rpc-types-engine", "dep:base-common-consensus", - "dep:base-common-flashblocks", "dep:base-execution-chainspec", "dep:base-test-utils", "dep:derive_more", diff --git a/crates/execution/flashblocks-node/README.md b/crates/execution/flashblocks-node/README.md index 7ce90fd46d..54303208c0 100644 --- a/crates/execution/flashblocks-node/README.md +++ b/crates/execution/flashblocks-node/README.md @@ -56,6 +56,10 @@ The `FlashblocksExtension` wires up: 2. **Canonical Subscription**: Reconciles pending state with canonical blocks 3. **RPC Extensions**: Provides extended Ethereum RPC methods with flashblock awareness 4. **WebSocket Subscriber**: Connects to and streams updates from the flashblock service +5. **Pool Deadline Expiry**: Records each received flashblock in the local transaction pool so + validity transactions whose `(block, flashblock)` deadline has been published are evicted + immediately, matching the builder. Transaction ingress is mempool-node first, so without this + an unbumped same-nonce replacement would be rejected as underpriced until the block commits. ## License diff --git a/crates/execution/flashblocks-node/src/extension.rs b/crates/execution/flashblocks-node/src/extension.rs index e18c2c9427..42885438bc 100644 --- a/crates/execution/flashblocks-node/src/extension.rs +++ b/crates/execution/flashblocks-node/src/extension.rs @@ -1,5 +1,6 @@ //! Contains the [`FlashblocksExtension`] which wires up the flashblocks feature -//! (canonical block subscription and RPC surface) on the Base node builder. +//! (flashblock subscription, pool deadline expiry, canonical block subscription and RPC +//! surface) on the Base node builder. use std::sync::Arc; @@ -12,6 +13,8 @@ use reth_chain_state::CanonStateSubscriptions; use tokio_stream::{StreamExt, wrappers::BroadcastStream}; use tracing::info; +use crate::PoolExpiringReceiver; + /// Helper struct that wires the Flashblocks feature (canonical subscription and RPC) into the node builder. #[derive(Debug)] pub struct FlashblocksExtension { @@ -35,20 +38,23 @@ impl BaseNodeExtension for FlashblocksExtension { }; let state = cfg.state; - let mut subscriber = FlashblocksSubscriber::new( - Arc::clone(&state), - cfg.websocket_url, - cfg.subscriber_ping_interval, - ); - let state_for_canonical = Arc::clone(&state); let state_for_rpc = Arc::clone(&state); let state_for_start = state; - // Start state processor, subscriber, and canonical subscription after node is started + // Start state processor, subscriber, and canonical subscription after node is started. + // The subscriber also records each received flashblock in the local pool so validity + // transactions expire at flashblock cadence on nodes that only follow the stream. let hooks = hooks.add_node_started_hook(move |ctx| { info!(message = "Starting Flashblocks state processor"); state_for_start.start(ctx.provider().clone()); + let receiver = + PoolExpiringReceiver::new(Arc::clone(&state_for_start), ctx.pool().clone()); + let mut subscriber = FlashblocksSubscriber::new( + Arc::new(receiver), + cfg.websocket_url, + cfg.subscriber_ping_interval, + ); subscriber.start(); let mut canonical_stream = diff --git a/crates/execution/flashblocks-node/src/lib.rs b/crates/execution/flashblocks-node/src/lib.rs index 99b3838e93..0b6f4a8f5f 100644 --- a/crates/execution/flashblocks-node/src/lib.rs +++ b/crates/execution/flashblocks-node/src/lib.rs @@ -10,5 +10,8 @@ mod extension; pub use extension::FlashblocksExtension; +mod pool_expiry; +pub use pool_expiry::PoolExpiringReceiver; + #[cfg(any(test, feature = "test-utils"))] pub mod test_harness; diff --git a/crates/execution/flashblocks-node/src/pool_expiry.rs b/crates/execution/flashblocks-node/src/pool_expiry.rs new file mode 100644 index 0000000000..a21bd01b4d --- /dev/null +++ b/crates/execution/flashblocks-node/src/pool_expiry.rs @@ -0,0 +1,115 @@ +//! Flashblock receiver that frees validity-transaction deadlines in the local pool. + +use std::sync::Arc; + +use base_common_flashblocks::Flashblock; +use base_execution_txpool::FlashblockExpiry; +use base_flashblocks::FlashblocksReceiver; +use tracing::debug; + +/// Forwards every received flashblock to the pending-state receiver and records its position in +/// the local transaction pool. +/// +/// The builder evicts a validity transaction the moment the flashblock holding its deadline is +/// published. A node that only follows that stream learned of the deadline no earlier than the +/// next block commit, so an unbumped same-nonce replacement submitted in between was rejected as +/// underpriced against a transaction the builder had already dropped. Recording the published +/// position here closes that gap for transaction ingress on mempool and RPC nodes. +#[derive(Debug)] +pub struct PoolExpiringReceiver { + receiver: Arc, + pool: Pool, +} + +impl PoolExpiringReceiver { + /// Wraps `receiver` so `pool` observes each flashblock before the pending state does. + pub const fn new(receiver: Arc, pool: Pool) -> Self { + Self { receiver, pool } + } +} + +impl FlashblocksReceiver for PoolExpiringReceiver +where + Receiver: FlashblocksReceiver, + Pool: FlashblockExpiry, +{ + fn on_flashblock_received(&self, flashblock: Flashblock) { + let block_number = flashblock.metadata.block_number; + let flashblock_index = flashblock.index; + // Expire before handing off: the pending-state processor executes asynchronously, and the + // same-nonce slot must be free by the time a client reacting to this flashblock resubmits. + let expired = self.pool.expire_published(block_number, flashblock_index); + if expired > 0 { + debug!( + block = block_number, + flashblock = flashblock_index, + expired, + "validity transactions expired at received flashblock" + ); + } + self.receiver.on_flashblock_received(flashblock); + } +} + +#[cfg(test)] +mod tests { + use std::sync::Mutex; + + use alloy_rpc_types_engine::PayloadId; + use base_common_flashblocks::{ExecutionPayloadFlashblockDeltaV1, Metadata}; + + use super::*; + + /// Records the positions the pool was asked to expire. `FlashblockExpiry::publish_and_expire` + /// is generic over its closure, which `mockall` cannot express. + #[derive(Clone, Default)] + struct RecordingPool(Arc>>); + + impl FlashblockExpiry for RecordingPool { + fn publish_and_expire( + &self, + _block_number: u64, + _flashblock_index: u64, + _publish: impl FnOnce() -> Result, + ) -> Result<(R, usize), E> { + unreachable!("followers never publish") + } + + fn expire_published(&self, block_number: u64, flashblock_index: u64) -> usize { + self.0.lock().unwrap().push((block_number, flashblock_index)); + 1 + } + } + + #[derive(Default)] + struct RecordingReceiver(Mutex>); + + impl FlashblocksReceiver for RecordingReceiver { + fn on_flashblock_received(&self, flashblock: Flashblock) { + self.0.lock().unwrap().push((flashblock.metadata.block_number, flashblock.index)); + } + } + + fn flashblock(block_number: u64, index: u64) -> Flashblock { + Flashblock { + payload_id: PayloadId::default(), + index, + base: None, + diff: ExecutionPayloadFlashblockDeltaV1::default(), + metadata: Metadata::new(block_number), + } + } + + #[test] + fn expires_pool_at_received_position_then_forwards() { + let pool = RecordingPool::default(); + let inner = Arc::new(RecordingReceiver::default()); + let receiver = PoolExpiringReceiver::new(Arc::clone(&inner), pool.clone()); + + receiver.on_flashblock_received(flashblock(100, 0)); + receiver.on_flashblock_received(flashblock(100, 3)); + + assert_eq!(*pool.0.lock().unwrap(), vec![(100, 0), (100, 3)]); + assert_eq!(*inner.0.lock().unwrap(), vec![(100, 0), (100, 3)]); + } +} diff --git a/crates/execution/flashblocks-node/src/test_harness.rs b/crates/execution/flashblocks-node/src/test_harness.rs index c9bcf688b9..7325021254 100644 --- a/crates/execution/flashblocks-node/src/test_harness.rs +++ b/crates/execution/flashblocks-node/src/test_harness.rs @@ -50,6 +50,8 @@ use tokio::{ }; use tokio_stream::{StreamExt, wrappers::BroadcastStream}; +use crate::PoolExpiringReceiver; + // The amount of time to wait (in milliseconds) after sending a new flashblock or canonical block // so it can be processed by the state processor const SLEEP_TIME: u64 = 10; @@ -198,7 +200,9 @@ impl BaseNodeExtension for FlashblocksTestExtension { ); ctx.modules.replace_configured(eth_pubsub.into_rpc())?; - let fb_for_task = fb; + // Mirror production: the local pool observes every flashblock before the pending + // state does, so validity deadlines expire at flashblock cadence on this node too. + let fb_for_task = PoolExpiringReceiver::new(fb, ctx.pool().clone()); let mut receiver = receiver .lock() .expect("flashblock receiver mutex poisoned") @@ -282,12 +286,20 @@ pub struct FlashblocksHarness { impl FlashblocksHarness { /// Launch a flashblocks-enabled harness with automatic canonical processing. pub async fn new() -> Result { - Self::with_options(true).await + Self::with_options(true, Vec::new()).await } /// Launch the harness configured for manual canonical progression. pub async fn manual_canonical() -> Result { - Self::with_options(false).await + Self::with_options(false, Vec::new()).await + } + + /// Launch the harness for manual canonical progression with additional node extensions, + /// such as the RPC ingress a mempool node exposes. + pub async fn manual_canonical_with_extensions( + extensions: Vec>, + ) -> Result { + Self::with_options(false, extensions).await } /// Get a reference to the inner [`TestHarness`] @@ -305,7 +317,10 @@ impl FlashblocksHarness { self.parts.send(flashblock).await } - async fn with_options(process_canonical: bool) -> Result { + async fn with_options( + process_canonical: bool, + mut extensions: Vec>, + ) -> Result { init_silenced_tracing(); // Build default chain spec programmatically @@ -316,8 +331,10 @@ impl FlashblocksHarness { let extension = FlashblocksTestExtension::new(process_canonical); let parts_source = extension.clone(); - // Launch the node with the flashblocks extension - let node = LocalNode::new(vec![Box::new(extension)], chain_spec).await?; + // Launch the node with the flashblocks extension. It replaces configured RPC modules, so + // it is installed last. + extensions.push(Box::new(extension)); + let node = LocalNode::new(extensions, chain_spec).await?; let engine = node.engine_api()?; tokio::time::sleep(Duration::from_millis(NODE_STARTUP_DELAY_MS)).await; @@ -346,9 +363,14 @@ pub struct FlashblocksBuilderTestHarness { impl FlashblocksBuilderTestHarness { /// Launch a new flashblocks builder test harness. pub async fn new() -> Self { + Self::with_extensions(Vec::new()).await + } + + /// Launch the harness with additional node extensions installed before flashblocks. + pub async fn with_extensions(extensions: Vec>) -> Self { // These tests simulate pathological timing (missing receipts, reorgs, etc.), so we disable // the automatic canonical listener and only apply blocks when the test explicitly requests it. - let node = FlashblocksHarness::manual_canonical() + let node = FlashblocksHarness::manual_canonical_with_extensions(extensions) .await .expect("able to launch flashblocks harness"); let provider = node.blockchain_provider(); diff --git a/crates/execution/flashblocks-node/tests/validity_expiry.rs b/crates/execution/flashblocks-node/tests/validity_expiry.rs new file mode 100644 index 0000000000..ea661f3745 --- /dev/null +++ b/crates/execution/flashblocks-node/tests/validity_expiry.rs @@ -0,0 +1,119 @@ +//! Regression test for validity-transaction expiry on nodes that follow the flashblock stream +//! instead of building it (mempool and RPC nodes). +//! +//! A validity transaction whose flashblock deadline has been published must release its +//! same-nonce slot as soon as this node receives that flashblock, so an unbumped replacement is +//! admitted before the block commits. Without follower-side expiry the replacement is rejected as +//! underpriced against a transaction the builder already dropped. + +use alloy_eips::Encodable2718; +use alloy_primitives::{B256, Bytes, TxHash, U256, hex::FromHex}; +use alloy_provider::Provider; +use base_common_consensus::BaseTransactionSigned; +use base_execution_txpool::{ValidityOperator, ValidityPredicate}; +use base_flashblocks_node::test_harness::{FlashblockBuilder, FlashblocksBuilderTestHarness}; +use base_test_utils::Account; +use base_txpool_rpc::{SendRawTransactionValidityExtension, SendRawTransactionValidityOptions}; +use reth_chainspec::EthChainSpec; +use reth_provider::ChainSpecProvider; +use reth_transaction_pool::test_utils::TransactionBuilder; + +/// Last flashblock index in which the validity transaction may be included. +const DEADLINE_FLASHBLOCK: u64 = 1; + +/// A transfer of `value` wei from Alice at `nonce` with fixed fees, so two calls with different +/// values are distinct transactions and the second is an unbumped replacement of the first. +fn alice_transfer( + harness: &FlashblocksBuilderTestHarness, + nonce: u64, + value: u128, +) -> BaseTransactionSigned { + let signer = B256::from_hex(Account::Alice.private_key()).expect("valid hex-encoded key"); + let tx = TransactionBuilder::default() + .signer(signer) + .chain_id(harness.provider.chain_spec().chain_id()) + .to(Account::Bob.address()) + .nonce(nonce) + .value(value) + .gas_limit(21_000) + .max_fee_per_gas(2_000_000_000) + .max_priority_fee_per_gas(1_000_000_000) + .into_eip1559() + .as_eip1559() + .expect("built as EIP-1559") + .clone(); + BaseTransactionSigned::Eip1559(tx) +} + +fn encoded(tx: &BaseTransactionSigned) -> Bytes { + tx.encoded_2718().into() +} + +async fn pooled(harness: &FlashblocksBuilderTestHarness, hash: TxHash) -> bool { + harness + .node + .provider() + .get_transaction_by_hash(hash) + .await + .expect("eth_getTransactionByHash succeeds") + .is_some() +} + +#[tokio::test] +async fn received_flashblock_deadline_releases_same_nonce_before_block_commits() { + let harness = FlashblocksBuilderTestHarness::with_extensions(vec![Box::new( + SendRawTransactionValidityExtension::default(), + )]) + .await; + let provider = harness.node.provider(); + let target_block = harness.node.latest_block().number + 1; + + // Alice's validity transaction may only land in `target_block` up to flashblock 1. + let original = alice_transfer(&harness, 0, 1); + let original_hash = *original.hash(); + let options = SendRawTransactionValidityOptions { + validity: vec![ + ValidityPredicate::BlockNumber { + op: ValidityOperator::LessThanOrEqual, + value: U256::from(target_block), + }, + ValidityPredicate::FlashblockIndex { + op: ValidityOperator::LessThanOrEqual, + value: U256::from(DEADLINE_FLASHBLOCK), + }, + ], + validity_signature: None, + }; + let accepted: TxHash = provider + .raw_request("base_sendRawTransactionValidity".into(), (encoded(&original), options)) + .await + .expect("validity transaction is admitted"); + assert_eq!(accepted, original_hash); + assert!(pooled(&harness, original_hash).await); + + // Same nonce, same fees: a plain replacement that reth prices as underpriced while the + // original still occupies the slot. + let replacement = alice_transfer(&harness, 0, 2); + let replacement_hash = *replacement.hash(); + let replacement_bytes = encoded(&replacement); + + // Flashblock 0 of the target block is published; the deadline has not passed yet. + harness.send_flashblock(FlashblockBuilder::new_base(&harness).build()).await; + let early = provider.send_raw_transaction(&replacement_bytes).await; + assert!( + early.as_ref().is_err_and(|error| error.to_string().contains("underpriced")), + "unbumped replacement must still be blocked before the deadline: {early:?}" + ); + assert!(pooled(&harness, original_hash).await); + + // Flashblock 1 is published without the original, so its deadline has passed. The block has + // not committed on this node, yet the nonce slot must already be free. + harness.send_flashblock(FlashblockBuilder::new(&harness, DEADLINE_FLASHBLOCK).build()).await; + assert!(!pooled(&harness, original_hash).await, "expired original must leave the pool"); + let admitted = provider + .send_raw_transaction(&replacement_bytes) + .await + .expect("unbumped replacement is admitted once the deadline flashblock is received"); + assert_eq!(*admitted.tx_hash(), replacement_hash); + assert!(pooled(&harness, replacement_hash).await); +} diff --git a/crates/execution/node/src/args.rs b/crates/execution/node/src/args.rs index 53eeb45155..d3aef0787b 100644 --- a/crates/execution/node/src/args.rs +++ b/crates/execution/node/src/args.rs @@ -9,7 +9,7 @@ use std::{ use alloy_primitives::Address; use base_execution_trie::{MdbxProofsStorageOptions, RocksdbProofsStorageOptions}; -use base_execution_txpool::{DEFAULT_PAYMENT_LIMIT, DEFAULT_SIGNATURE_LIMIT}; +use base_execution_txpool::{DEFAULT_PAYMENT_LIMIT, DEFAULT_SIGNATURE_LIMIT, ValidityPoolLimits}; use base_upgrade_signal::{UpgradeSignalArgs, UpgradeSignalL1RpcArgs}; use clap::{ArgAction, ValueEnum, builder::ArgPredicate}; @@ -391,6 +391,21 @@ pub struct RollupArgs { #[arg(long = "rollup.mempool-trusted-delegation-targets", value_delimiter = ',')] pub mempool_trusted_delegation_targets: Vec

, + /// Maximum number of transactions carrying validity predicates held by the + /// validity pool. Defaults to `--txpool.pending-max-count`. + #[arg(long = "rollup.validity-pool-max-count")] + pub validity_pool_max_count: Option, + + /// Maximum total size of the validity pool, in megabytes. Defaults to + /// `--txpool.pending-max-size`. + #[arg(long = "rollup.validity-pool-max-size")] + pub validity_pool_max_size: Option, + + /// Maximum validity transactions one external sender may hold in the validity + /// pool. Defaults to `--txpool.max-account-slots`. + #[arg(long = "rollup.validity-pool-max-account-slots")] + pub validity_pool_max_account_slots: Option, + /// If true, initialize external-proofs exex to save and serve trie nodes to provide proofs /// faster. #[arg( @@ -489,6 +504,19 @@ pub struct RollupArgs { pub upgrade_signal_l1_rpc: UpgradeSignalL1RpcArgs, } +impl RollupArgs { + /// Resolves the validity pool flags, converting the size cap from megabytes. + pub fn validity_pool_limits(&self) -> ValidityPoolLimits { + ValidityPoolLimits { + max_txs: self.validity_pool_max_count, + max_size: self + .validity_pool_max_size + .map(|megabytes| megabytes.saturating_mul(MIB as usize)), + max_account_slots: self.validity_pool_max_account_slots, + } + } +} + impl HasRollupArgs for RollupArgs { fn rollup_args(&self) -> &RollupArgs { self @@ -507,6 +535,9 @@ impl Default for RollupArgs { mempool_sender_limit: DEFAULT_SIGNATURE_LIMIT, mempool_payer_limit: DEFAULT_PAYMENT_LIMIT, mempool_trusted_delegation_targets: Vec::new(), + validity_pool_max_count: None, + validity_pool_max_size: None, + validity_pool_max_account_slots: None, proofs_history: false, proofs_history_storage_path: None, proofs_history_db: ProofsHistoryDbBackend::default(), @@ -591,6 +622,33 @@ mod tests { ); } + #[test] + fn validity_pool_limits_default_to_inherit() { + let args = CommandParser::::parse_from(["reth"]).args; + assert_eq!(args.validity_pool_limits(), ValidityPoolLimits::inherit()); + } + + #[test] + fn validity_pool_limits_parse_with_size_in_megabytes() { + let args = CommandParser::::parse_from([ + "reth", + "--rollup.validity-pool-max-count", + "5000", + "--rollup.validity-pool-max-size", + "3", + "--rollup.validity-pool-max-account-slots", + "32", + ]) + .args; + assert_eq!( + args.validity_pool_limits(), + ValidityPoolLimits::inherit() + .with_max_txs(5000) + .with_max_size(3 * 1024 * 1024) + .with_max_account_slots(32) + ); + } + #[test] fn test_parse_max_inflight_delegated_slots_override() { let expected_args = RollupArgs { max_inflight_delegated_slots: 7, ..Default::default() }; diff --git a/crates/execution/node/src/node.rs b/crates/execution/node/src/node.rs index 6b77066b13..375882d227 100644 --- a/crates/execution/node/src/node.rs +++ b/crates/execution/node/src/node.rs @@ -30,7 +30,7 @@ use base_execution_rpc::{ use base_execution_txpool::{ BaseOrdering, BasePooledTransaction, BasePooledTx, BaseTransactionPool, BaseTransactionValidator, GuardLimits, ParkableTransactionPool, TimestampedTransaction, - maintain_state_diff_invalidation, + ValidityPoolLimits, maintain_state_diff_invalidation, }; use reth_chain_state::CanonStateSubscriptions; use reth_chainspec::{BaseFeeParams, ChainSpecProvider, EthChainSpec, Hardforks}; @@ -273,6 +273,7 @@ impl BaseNode { signature_limit: mempool_sender_limit, payment_limit: mempool_payer_limit, }) + .with_validity_pool_limits(self.args.validity_pool_limits()) .with_additional_trusted_delegation_targets( self.args.mempool_trusted_delegation_targets.iter().copied(), ), @@ -888,6 +889,8 @@ pub struct BasePoolBuilder { pub max_inflight_delegated_slots: usize, /// Per-account EIP-8130 admission caps. pub guard_limits: GuardLimits, + /// Capacity of the validity pool; unset fields copy the pending subpool limit. + pub validity_pool_limits: ValidityPoolLimits, /// Additional trusted EIP-7702 delegation targets for locked payers. pub additional_trusted_delegation_targets: AddressSet, /// Marker for the pooled transaction type. @@ -901,6 +904,7 @@ impl Default for BasePoolBuilder { ordering: BaseOrdering::default(), max_inflight_delegated_slots: 4, guard_limits: GuardLimits::default(), + validity_pool_limits: ValidityPoolLimits::inherit(), additional_trusted_delegation_targets: AddressSet::default(), _pd: Default::default(), } @@ -914,6 +918,7 @@ impl Clone for BasePoolBuilder { ordering: self.ordering.clone(), max_inflight_delegated_slots: self.max_inflight_delegated_slots, guard_limits: self.guard_limits, + validity_pool_limits: self.validity_pool_limits, additional_trusted_delegation_targets: self .additional_trusted_delegation_targets .clone(), @@ -950,6 +955,12 @@ impl BasePoolBuilder { self } + /// Sets the validity pool's capacity. Unset fields copy the pending subpool limit. + pub const fn with_validity_pool_limits(mut self, limits: ValidityPoolLimits) -> Self { + self.validity_pool_limits = limits; + self + } + /// Sets additional trusted delegation targets for balance-bounded locked payers. pub fn with_additional_trusted_delegation_targets( mut self, @@ -978,6 +989,7 @@ where ordering, max_inflight_delegated_slots, guard_limits, + validity_pool_limits, additional_trusted_delegation_targets, .. } = self; @@ -1016,8 +1028,9 @@ where blob_store, final_pool_config.clone(), ); - let transaction_pool = - BaseTransactionPool::new(transaction_pool, ordering).with_guard_limits(guard_limits); + let transaction_pool = BaseTransactionPool::new(transaction_pool, ordering) + .with_guard_limits(guard_limits) + .with_validity_pool_limits(validity_pool_limits); spawn_maintenance_tasks(ctx, transaction_pool.clone(), &final_pool_config)?; let state_diff_events = BroadcastStream::new(ctx.provider().subscribe_to_canonical_state()); ctx.task_executor().spawn_critical_task( @@ -1030,6 +1043,12 @@ where max_inflight_delegated_slots = max_inflight_delegated_slots, sender_limit = guard_limits.signature_limit, payer_limit = guard_limits.payment_limit, + validity_pool_max_txs = validity_pool_limits + .max_txs + .unwrap_or(final_pool_config.pending_limit.max_txs), + validity_pool_max_size = validity_pool_limits + .max_size + .unwrap_or(final_pool_config.pending_limit.max_size), "Transaction pool initialized" ); debug!(target: "reth::cli", "Spawned txpool maintenance tasks"); @@ -1172,8 +1191,10 @@ where resource_metering: self.resource_metering, rejection_cache: self.rejection_cache, state_provider_metrics: ctx.config().engine.state_provider_metrics, + resting_predicate_mode: Default::default(), }, - ); + ) + .with_task_executor(ctx.task_executor().clone()); Ok(payload_builder) } } diff --git a/crates/execution/node/src/proof_history.rs b/crates/execution/node/src/proof_history.rs index 6bfcf3cb41..424f8a2479 100644 --- a/crates/execution/node/src/proof_history.rs +++ b/crates/execution/node/src/proof_history.rs @@ -54,6 +54,9 @@ pub async fn launch_node_with_proof_history( mempool_sender_limit, mempool_payer_limit, mempool_trusted_delegation_targets, + validity_pool_max_count, + validity_pool_max_size, + validity_pool_max_account_slots, proofs_history, proofs_history_storage_path, proofs_history_db, @@ -77,6 +80,9 @@ pub async fn launch_node_with_proof_history( mempool_sender_limit, mempool_payer_limit, mempool_trusted_delegation_targets, + validity_pool_max_count, + validity_pool_max_size, + validity_pool_max_account_slots, proofs_history: false, proofs_history_storage_path: None, proofs_history_db: ProofsHistoryDbBackend::default(), diff --git a/crates/execution/payload/Cargo.toml b/crates/execution/payload/Cargo.toml index 4c53c98b58..c289eeab25 100644 --- a/crates/execution/payload/Cargo.toml +++ b/crates/execution/payload/Cargo.toml @@ -14,6 +14,7 @@ workspace = true [dependencies] # reth reth-evm.workspace = true +reth-tasks.workspace = true reth-chainspec.workspace = true reth-storage-api.workspace = true reth-engine-tree.workspace = true @@ -62,6 +63,8 @@ derive_more.workspace = true serde_json.workspace = true moka = { workspace = true, features = ["sync"] } metrics = { workspace = true, optional = true } +parking_lot.workspace = true +clap = { workspace = true, optional = true, features = ["derive"] } [dev-dependencies] reth-provider.workspace = true @@ -70,9 +73,23 @@ metrics-exporter-prometheus.workspace = true base-execution-chainspec.workspace = true reth-revm = { workspace = true, features = ["test-utils"] } base-observability-events = { workspace = true, features = ["test-utils"] } +criterion.workspace = true + +[[bench]] +name = "validity_conditions" +harness = false + +[[bench]] +name = "resting_carry" +harness = false + +[[bench]] +name = "native_resting_sweep" +harness = false [features] default = [ "metrics" ] +clap = [ "dep:clap" ] metrics = [ "base-common-evm/metrics", "base-metrics/metrics", diff --git a/crates/execution/payload/benches/native_resting_sweep.rs b/crates/execution/payload/benches/native_resting_sweep.rs new file mode 100644 index 0000000000..8d237373ab --- /dev/null +++ b/crates/execution/payload/benches/native_resting_sweep.rs @@ -0,0 +1,314 @@ +//! Per-block latency of the native (post-Denim) payload builder carrying a large resting set, +//! including the blocks that sweep members no build yielded. +//! +//! The pool holds [`LIVE_RESTING`] validity transactions whose predicates never become satisfied, +//! so every block yields them and holds them back, plus one block of plain transfers. Each block +//! also rests [`CHURN_PER_BLOCK`] members whose transactions then leave the pool, so the sweep +//! always has members to drop and the set stays at a steady size. +//! +//! Blocks run back to back through the same store calls as the native builder: take the set, +//! carry it into the block, run the selection loop, then finish the block and store the set. By +//! default the next block starts immediately, so a sweep still running when it takes the set is +//! counted against that block rather than hidden behind a state root. +//! `RESTING_SWEEP_BENCH_GAP_MS` adds an untimed pause between blocks to model the state root. The +//! table reports the build path of each block from taking the set to the end of the selection +//! loop. +//! +//! ```sh +//! cargo bench -p base-execution-payload-builder --bench native_resting_sweep +//! ``` + +use std::{ + sync::Arc, + time::{Duration, Instant}, +}; + +use alloy_consensus::{Header, SignableTransaction, TxEip1559}; +use alloy_eips::eip2718::Encodable2718; +use alloy_primitives::{Address, B256, Bytes, Signature, TxKind, U256, keccak256}; +use base_common_chains::BaseUpgrade; +use base_common_consensus::{BaseTransactionSigned, BaseTxEnvelope, Predeploys}; +use base_execution_chainspec::{BaseChainSpec, BaseChainSpecBuilder}; +use base_execution_evm::BaseEvmConfig; +use base_execution_payload_builder::{ + BasePayloadBuilderAttributes, DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, + ParkableBestPayloadTransactions, RESTING_UNSEEN_RETAINED_BLOCKS, RestingPredicateMode, + SharedRestingPredicates, + builder::{BasePayloadBuilderCtx, ExecutionInfo}, + config::BaseBuilderConfig, + payload::EthPayloadBuilderAttributes, +}; +use base_execution_txpool::{ + BaseOrdering, BasePooledTransaction, ParkedBestTransactions, ValidityOperator, + ValidityPredicate, +}; +use reth_basic_payload_builder::PayloadConfig; +use reth_ethereum_forks::ForkCondition; +use reth_evm::execute::{BlockBuilder, BlockExecutor}; +use reth_payload_builder::PayloadId; +use reth_primitives_traits::{Recovered, SealedHeader}; +use reth_revm::db::State; +use reth_transaction_pool::{ + TransactionOrigin, ValidPoolTransaction, identifier::TransactionId, pool::PendingPool, +}; +use revm::{ + database::{InMemoryDB, TransitionState}, + state::AccountInfo, +}; + +type Ordering = BaseOrdering; +type Pool = PendingPool; +type Ctx = BasePayloadBuilderCtx; + +const CHAIN_ID: u64 = 8_453; +/// Resting validity transactions the pool yields in every block. +const LIVE_RESTING: usize = 100_000; +/// Members rested in each block whose transactions leave the pool afterwards. +const CHURN_PER_BLOCK: u64 = 4_800; +/// Plain transfers that fill each block. +const FILLER_PER_BLOCK: usize = 120; +/// Blocks built before measuring, so the set reaches its steady-state size. +const WARMUP_BLOCKS: u64 = 3 * RESTING_UNSEEN_RETAINED_BLOCKS; +/// Blocks measured. +const MEASURED_BLOCKS: u64 = 4 * RESTING_UNSEEN_RETAINED_BLOCKS; +/// Long enough that a block always waits for the previous block's sweep, so carry includes it. +const SWEEP_WAIT_LIMIT: Duration = Duration::from_secs(1); +const TRANSFER_GAS: u64 = 21_000; +const SENDER_OFFSET: usize = 1_000_000; +const DENIM_TIMESTAMP: u64 = 1; + +fn address(index: usize) -> Address { + Address::from_word(U256::from(index + 1).into()) +} + +fn transaction( + sender_index: usize, + priority_fee: u128, + predicates: Vec, +) -> Arc> { + let tx = TxEip1559 { + chain_id: CHAIN_ID, + nonce: 0, + gas_limit: TRANSFER_GAS, + max_fee_per_gas: 1_000_000_000_000, + max_priority_fee_per_gas: priority_fee, + to: TxKind::Call(Address::repeat_byte(0xee)), + value: U256::from(sender_index), + access_list: Default::default(), + input: Bytes::new(), + }; + let envelope = BaseTxEnvelope::Eip1559(tx.into_signed(Signature::test_signature())); + let encoded_length = envelope.encode_2718_len(); + let sender = address(SENDER_OFFSET + sender_index); + let transaction = BasePooledTransaction::new( + Recovered::new_unchecked(BaseTransactionSigned::from(envelope), sender), + encoded_length, + ) + .with_validity_predicates(predicates); + Arc::new(ValidPoolTransaction { + transaction_id: TransactionId::new(((SENDER_OFFSET + sender_index) as u64).into(), 0), + transaction, + propagate: true, + timestamp: Instant::now(), + origin: TransactionOrigin::External, + authority_ids: None, + }) +} + +/// [`LIVE_RESTING`] never-satisfied validity transactions, each watching its own balance, plus +/// one block of filler, and the parent state that funds every sender. +fn resting_pool() -> (Pool, InMemoryDB) { + let mut pool = PendingPool::new(Ordering::coinbase_tip()); + let mut db = InMemoryDB::default(); + for sender_index in 0..LIVE_RESTING + FILLER_PER_BLOCK { + let (priority_fee, predicates) = if sender_index < LIVE_RESTING { + ( + 10, + vec![ValidityPredicate::Balance { + address: address(sender_index), + op: ValidityOperator::Equal, + value: U256::ONE, + }], + ) + } else { + (1, Vec::new()) + }; + pool.add_transaction(transaction(sender_index, priority_fee, predicates), 0); + db.insert_account_info( + address(SENDER_OFFSET + sender_index), + AccountInfo { balance: U256::from(u128::MAX), ..Default::default() }, + ); + } + (pool, db) +} + +fn payload_context() -> Ctx { + let chain_spec = Arc::new( + BaseChainSpecBuilder::base_mainnet() + .with_fork(BaseUpgrade::Denim, ForkCondition::Timestamp(DENIM_TIMESTAMP)) + .build(), + ); + let gas_limit = FILLER_PER_BLOCK as u64 * TRANSFER_GAS; + let parent = Arc::new(SealedHeader::seal_slow(Header { gas_limit, ..Default::default() })); + let payload_id = PayloadId::new([0; 8]); + let attributes = BasePayloadBuilderAttributes:: { + payload_attributes: EthPayloadBuilderAttributes { + id: payload_id, + parent: parent.hash(), + timestamp: DENIM_TIMESTAMP, + parent_beacon_block_root: Some(B256::ZERO), + ..Default::default() + }, + gas_limit: Some(gas_limit), + ..Default::default() + }; + BasePayloadBuilderCtx { + evm_config: BaseEvmConfig::base(Arc::clone(&chain_spec)), + builder_config: BaseBuilderConfig::default() + .with_resting_predicate_mode(RestingPredicateMode::Enforce), + chain_spec, + config: PayloadConfig::new(parent, attributes, payload_id), + cancel: Default::default(), + best_payload: None, + } +} + +/// A storage predicate on its own slot, the shape of a smart-account nonce check. +fn churn_predicate(index: u64) -> ValidityPredicate { + ValidityPredicate::Storage { + address: Address::repeat_byte(0xcc), + slot: U256::from(index), + mask: U256::MAX, + op: ValidityOperator::Equal, + value: U256::from(index + 1), + } +} + +fn block_hash(block: u64) -> B256 { + keccak256(block.to_be_bytes()) +} + +/// Store calls that differ between builder versions. +mod store { + use alloy_primitives::B256; + use base_execution_payload_builder::{RestingPredicateStore, RestingPredicates}; + + pub(super) fn new() -> RestingPredicateStore { + RestingPredicateStore::new(reth_tasks::Runtime::test()) + } + + pub(super) fn finish_and_put( + store: &RestingPredicateStore, + resting: RestingPredicates, + block_hash: B256, + ) { + store.put(store.finish(resting), block_hash); + } +} + +/// Untimed pause between finishing a block and starting the next, from +/// `RESTING_SWEEP_BENCH_GAP_MS`, standing in for the state root and leeway an end-of-block +/// sweep overlaps in the builder. Unset, the next block starts immediately. +fn block_gap() -> Option { + std::env::var("RESTING_SWEEP_BENCH_GAP_MS").ok().map(|ms| { + Duration::from_millis(ms.parse().expect("RESTING_SWEEP_BENCH_GAP_MS is a number")) + }) +} + +struct BlockTiming { + block: u64, + carry: Duration, + selection: Duration, + carried: usize, +} + +fn main() { + let (pool, db) = resting_pool(); + let ctx = payload_context(); + let store = store::new(); + let gap = block_gap(); + let mut churn = 0u64; + let mut timings = Vec::new(); + + for block in 1..=WARMUP_BLOCKS + MEASURED_BLOCKS { + let start = Instant::now(); + let mut resting = store.take(DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, SWEEP_WAIT_LIMIT); + resting.begin_block(block_hash(block - 1), block, Some(&TransitionState::default())); + let carry = start.elapsed(); + let carried = resting.len(); + + let mut state = State::builder().with_database(db.clone()).with_bundle_update().build(); + state.load_cache_account(Predeploys::L1_BLOCK_INFO).expect("in-memory load cannot fail"); + let mut builder = ctx.block_builder(&mut state).expect("block builder"); + let mut info = ExecutionInfo::default(); + // The production builder hands the pool iterator a hold over the shared set, so resting + // members are parked at the source instead of reaching the build loop. + let shared = SharedRestingPredicates::new(resting); + let best = ParkableBestPayloadTransactions::new(Box::new( + ParkedBestTransactions::new(pool.best(), Ordering::coinbase_tip(), 0) + .with_hold(Some(shared.iterator_hold())), + )); + ctx.execute_best_transactions(&mut info, &mut builder, best, Some(&shared)) + .expect("in-memory execution cannot fail"); + let mut resting = shared.take_predicates(); + assert_eq!(builder.executor().receipts().len(), FILLER_PER_BLOCK); + let selection = start.elapsed() - carry; + + for _ in 0..CHURN_PER_BLOCK { + resting.rest(keccak256(churn.to_be_bytes()), churn_predicate(churn)); + churn += 1; + } + store::finish_and_put(&store, resting, block_hash(block)); + if let Some(gap) = gap { + std::thread::sleep(gap); + } + + if block > WARMUP_BLOCKS { + timings.push(BlockTiming { block, carry, selection, carried }); + } + } + + println!("gap_ms={}", gap.map_or(0, |gap| gap.as_millis())); + println!( + "{:>6} {:>9} {:>12} {:>14} {:>12}", + "block", "carried", "carry (ms)", "selection (ms)", "total (ms)" + ); + for t in &timings { + println!( + "{:>6} {:>9} {:>12.3} {:>14.3} {:>12.3}", + t.block, + t.carried, + t.carry.as_secs_f64() * 1e3, + t.selection.as_secs_f64() * 1e3, + (t.carry + t.selection).as_secs_f64() * 1e3 + ); + } + let summarize = |label: &str, filter: &dyn Fn(&BlockTiming) -> bool| { + let mut carry: Vec = + timings.iter().filter(|t| filter(t)).map(|t| t.carry.as_secs_f64() * 1e3).collect(); + let mut total: Vec = timings + .iter() + .filter(|t| filter(t)) + .map(|t| (t.carry + t.selection).as_secs_f64() * 1e3) + .collect(); + carry.sort_by(f64::total_cmp); + total.sort_by(f64::total_cmp); + let mean = |v: &[f64]| v.iter().sum::() / v.len() as f64; + let quantile = |v: &[f64], q: f64| v[((v.len() - 1) as f64 * q).round() as usize]; + println!( + "SUMMARY {label} blocks={} carry_mean_ms={:.3} carry_p50_ms={:.3} carry_p90_ms={:.3} carry_max_ms={:.3} total_mean_ms={:.3} total_p50_ms={:.3} total_p90_ms={:.3} total_max_ms={:.3}", + carry.len(), + mean(&carry), + quantile(&carry, 0.5), + quantile(&carry, 0.9), + quantile(&carry, 1.0), + mean(&total), + quantile(&total, 0.5), + quantile(&total, 0.9), + quantile(&total, 1.0), + ); + }; + summarize("all", &|_| true); + summarize("sweep_blocks", &|t| t.block.is_multiple_of(RESTING_UNSEEN_RETAINED_BLOCKS)); + summarize("other_blocks", &|t| !t.block.is_multiple_of(RESTING_UNSEEN_RETAINED_BLOCKS)); +} diff --git a/crates/execution/payload/benches/resting_carry.rs b/crates/execution/payload/benches/resting_carry.rs new file mode 100644 index 0000000000..de5a0e979b --- /dev/null +++ b/crates/execution/payload/benches/resting_carry.rs @@ -0,0 +1,157 @@ +//! Benchmarks carrying a large set of resting validity transactions across blocks. +//! +//! Under sustained validity traffic a builder can carry over 100k resting members, and an unseen +//! sweep can drop tens of thousands of them at once. Work done in +//! `RestingPredicates::begin_block` runs between the forkchoice update and flashblock +//! scheduling, so every millisecond spent there delays the first flashblock and can cost whole +//! flashblocks. This benchmark measures the block-start path and the unseen sweep separately. +//! Before the sweep moved to the end of the block, `begin_block/sweep_block` included it. +//! +//! ```sh +//! cargo bench -p base-execution-payload-builder --bench resting_carry +//! ``` + +use std::{hint::black_box, time::Duration}; + +use alloy_primitives::{Address, B256, U256, keccak256}; +use base_execution_payload_builder::{ + DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, RESTING_UNSEEN_RETAINED_BLOCKS, RestingPredicates, +}; +use base_execution_txpool::{ValidityOperator, ValidityPredicate}; +use criterion::{BatchSize, BenchmarkId, Criterion}; +use revm::database::TransitionState; + +/// Members still yielded by the pool, kept by the sweep. +const LIVE_MEMBERS: usize = 75_000; +/// Members no build yielded for longer than the retention window, dropped by the sweep. +const STALE_MEMBERS: usize = 45_000; +/// Distinct contracts the storage predicates read. +const CONTRACTS: u64 = 64; +/// Distinct accounts the balance predicates read, so their buckets convert to the ordered form. +const ACCOUNTS: u64 = 256; + +/// First block whose start sweeps unseen members. +const SWEEP_BLOCK: u64 = 2 * RESTING_UNSEEN_RETAINED_BLOCKS; + +fn tx_hash(i: u64) -> B256 { + keccak256(i.to_be_bytes()) +} + +fn block_hash(block: u64) -> B256 { + keccak256((u64::MAX - block).to_be_bytes()) +} + +/// Mix of storage-equality predicates (one slot per sender, like a smart-account nonce) and +/// balance thresholds shared by many members. +fn predicate(i: u64) -> ValidityPredicate { + if i.is_multiple_of(4) { + ValidityPredicate::Balance { + address: Address::with_last_byte((i % ACCOUNTS) as u8), + op: ValidityOperator::GreaterThanOrEqual, + value: U256::from(1_000_000 + i), + } + } else { + let mut address = [0u8; 20]; + address[..8].copy_from_slice(&(i % CONTRACTS).to_be_bytes()); + ValidityPredicate::Storage { + address: Address::from(address), + slot: U256::from(i), + mask: U256::MAX, + op: ValidityOperator::Equal, + value: U256::from(i + 1), + } + } +} + +/// Members rested and released again during a past burst, which leave the set's maps at their +/// peak capacity. +const BURST_MEMBERS: u64 = 1_000_000; + +/// Returns a set that finished `last_block`, holding `STALE_MEMBERS` last seen in block 1 and +/// `LIVE_MEMBERS` last seen in `last_block`. With `burst`, the set first grew to +/// [`BURST_MEMBERS`] in block 1. +fn large_set(last_block: u64, burst: bool) -> RestingPredicates { + let empty = TransitionState::default(); + let mut resting = RestingPredicates::new(DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD); + resting.begin_block(B256::ZERO, 1, Some(&empty)); + if burst { + let first = (STALE_MEMBERS + LIVE_MEMBERS) as u64; + for i in first..first + BURST_MEMBERS { + resting.rest(tx_hash(i), predicate(i)); + } + for i in first..first + BURST_MEMBERS { + resting.remove(tx_hash(i)); + } + } + for i in 0..STALE_MEMBERS as u64 { + resting.rest(tx_hash(i), predicate(i)); + } + resting.end_block(block_hash(1)); + for block in 2..=last_block { + resting.begin_block(block_hash(block - 1), block, Some(&empty)); + if block == last_block { + for i in STALE_MEMBERS as u64..(STALE_MEMBERS + LIVE_MEMBERS) as u64 { + resting.rest(tx_hash(i), predicate(i)); + } + } + resting.end_block(block_hash(block)); + } + assert_eq!(resting.len(), STALE_MEMBERS + LIVE_MEMBERS); + resting +} + +fn bench_begin_block(c: &mut Criterion) { + let empty = TransitionState::default(); + let mut group = c.benchmark_group("resting_carry/begin_block"); + group.sample_size(10).measurement_time(Duration::from_secs(20)); + for (label, block, burst) in [ + ("regular", SWEEP_BLOCK - 1, false), + ("sweep_block", SWEEP_BLOCK, false), + ("sweep_block_after_burst", SWEEP_BLOCK, true), + ] { + group.bench_function(BenchmarkId::from_parameter(label), |b| { + b.iter_batched( + || large_set(block - 1, burst), + |mut resting| { + let carry_over = + resting.begin_block(block_hash(block - 1), block, Some(&empty)); + black_box(carry_over); + resting + }, + BatchSize::PerIteration, + ); + }); + } + group.finish(); +} + +/// The unseen sweep itself, which now runs when a sweep block finishes, off the build path. +fn bench_end_of_block_sweep(c: &mut Criterion) { + let empty = TransitionState::default(); + let mut group = c.benchmark_group("resting_carry/end_of_block_sweep"); + group.sample_size(10).measurement_time(Duration::from_secs(20)); + for (label, burst) in [("steady", false), ("after_burst", true)] { + group.bench_function(BenchmarkId::from_parameter(label), |b| { + b.iter_batched( + || { + let mut resting = large_set(SWEEP_BLOCK - 1, burst); + resting.begin_block(block_hash(SWEEP_BLOCK - 1), SWEEP_BLOCK, Some(&empty)); + resting + }, + |mut resting| { + assert_eq!(resting.sweep_unseen(), STALE_MEMBERS); + resting + }, + BatchSize::PerIteration, + ); + }); + } + group.finish(); +} + +fn main() { + let mut criterion = Criterion::default().configure_from_args(); + bench_begin_block(&mut criterion); + bench_end_of_block_sweep(&mut criterion); + criterion.final_summary(); +} diff --git a/crates/execution/payload/benches/validity_conditions.rs b/crates/execution/payload/benches/validity_conditions.rs new file mode 100644 index 0000000000..bc340f2267 --- /dev/null +++ b/crates/execution/payload/benches/validity_conditions.rs @@ -0,0 +1,479 @@ +//! Benchmarks for compiled validity conditions (validity predicate batches): ingestion, +//! repeated evaluation in the candidate loop, resting/parking membership checks, and +//! builder forwarding, plus a retained-memory report per predicate mix. +//! +//! Timing predicates are interleaved with state predicates rather than leading the batch, +//! so the benchmark exercises wire orders where the block-number predicate is not first. +//! +//! The memory table is printed before the timed benchmarks. It reports +//! `size_of::() + heap_size()` next to the bytes requested from the +//! allocator while compiling the batch, and the pool-accounted size of a transfer carrying +//! that batch after JSON decoding, which is how the RPC path ingests it. +//! +//! ```sh +//! cargo bench -p base-execution-payload-builder --bench validity_conditions +//! ``` + +use std::{ + alloc::{GlobalAlloc, Layout, System}, + hint::black_box, + sync::atomic::{AtomicUsize, Ordering}, + time::Instant, +}; + +use alloy_consensus::{InMemorySize, SignableTransaction, TxEip1559}; +use alloy_eips::eip2718::Encodable2718; +use alloy_primitives::{Address, Bytes, Signature, TxKind, U256}; +use base_common_consensus::{BaseTransactionSigned, BaseTxEnvelope}; +use base_execution_payload_builder::ValidityPredicateKey; +use base_execution_txpool::{ + BasePooledTransaction, PredicateContext, TransactionValidity, ValidatedTransactionExtensions, + ValidityConditions, ValidityOperator, ValidityPredicate, +}; +use criterion::{BatchSize, BenchmarkId, Criterion, Throughput}; +use reth_primitives_traits::Recovered; +use reth_transaction_pool::{TransactionOrigin, ValidPoolTransaction, identifier::TransactionId}; +use revm::{database::InMemoryDB, state::AccountInfo}; + +/// Counts allocations so the report can show real allocator traffic next to `heap_size`. +struct CountingAllocator; + +static ALLOCATIONS: AtomicUsize = AtomicUsize::new(0); +static ALLOCATED_BYTES: AtomicUsize = AtomicUsize::new(0); + +// SAFETY: every method forwards to `System` unchanged and only bumps relaxed counters, so the +// allocator contract is exactly that of `System`. +unsafe impl GlobalAlloc for CountingAllocator { + unsafe fn alloc(&self, layout: Layout) -> *mut u8 { + ALLOCATIONS.fetch_add(1, Ordering::Relaxed); + ALLOCATED_BYTES.fetch_add(layout.size(), Ordering::Relaxed); + // SAFETY: the caller upholds the `GlobalAlloc::alloc` contract for `layout`. + unsafe { System.alloc(layout) } + } + + unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) { + // SAFETY: `ptr` was returned by `System` through this allocator with the same `layout`. + unsafe { System.dealloc(ptr, layout) } + } + + unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 { + ALLOCATIONS.fetch_add(1, Ordering::Relaxed); + ALLOCATED_BYTES.fetch_add(new_size, Ordering::Relaxed); + // SAFETY: `ptr` was returned by `System` through this allocator with `layout`, and the + // caller upholds the `GlobalAlloc::realloc` contract for `new_size`. + unsafe { System.realloc(ptr, layout, new_size) } + } +} + +#[global_allocator] +static GLOBAL: CountingAllocator = CountingAllocator; + +fn allocations_during(f: impl FnOnce() -> T) -> (T, usize, usize) { + let count = ALLOCATIONS.load(Ordering::Relaxed); + let bytes = ALLOCATED_BYTES.load(Ordering::Relaxed); + let value = f(); + ( + value, + ALLOCATIONS.load(Ordering::Relaxed) - count, + ALLOCATED_BYTES.load(Ordering::Relaxed) - bytes, + ) +} + +const EXPIRY_BLOCK: u64 = 1_000; +const NONCE: u64 = 7; +const BALANCE: u64 = 1_000; +const CONTEXT: PredicateContext = PredicateContext { block_number: 500, flashblock_index: 3 }; +const EXPIRED_CONTEXT: PredicateContext = + PredicateContext { block_number: EXPIRY_BLOCK + 1, flashblock_index: 3 }; +/// Wire ceiling for one batch. +const MAX_PREDICATES: usize = 64; +/// Transactions in the pool-scale evaluation benchmark. +const POOL_TRANSACTIONS: usize = 10_000; + +fn address(index: usize) -> Address { + Address::from_word(U256::from(index + 1).into()) +} + +fn state_predicate(kind: usize, address: Address, satisfied: bool) -> ValidityPredicate { + match kind % 3 { + 0 => ValidityPredicate::Storage { + address, + slot: U256::from(kind), + mask: U256::MAX, + op: ValidityOperator::Equal, + value: if satisfied { U256::from(kind) } else { U256::MAX }, + }, + 1 => ValidityPredicate::Balance { + address, + op: ValidityOperator::GreaterThanOrEqual, + value: U256::from(if satisfied { 1 } else { BALANCE + 1 }), + }, + _ => ValidityPredicate::Nonce { + address, + op: ValidityOperator::Equal, + value: if satisfied { U256::from(NONCE) } else { U256::from(NONCE + 1) }, + }, + } +} + +/// A batch with `state` state predicates rotating through storage, balance, and nonce, and +/// `timing` timing predicates (one block expiry, then flashblock lower bounds) interleaved +/// among them. +/// +/// The block expiry is inserted about two thirds of the way through the state predicates and +/// the flashblock bounds about one third of the way, both shifted by `rotation`, so the +/// block-number predicate is never simply first and pool-scale workloads vary the position. +/// +/// State addresses start at `address_base` so pool-scale workloads can use distinct accounts. +/// When `last_fails` is set, the final state predicate is unsatisfied against [`database`]. +fn mixed( + timing: usize, + state: usize, + address_base: usize, + rotation: usize, + last_fails: bool, +) -> Vec { + let mut predicates = Vec::with_capacity(timing + state); + for kind in 0..state { + let satisfied = !(last_fails && kind + 1 == state); + predicates.push(state_predicate(kind, address(address_base + kind), satisfied)); + } + if timing > 0 { + let at = ((2 * state).div_ceil(3) + rotation) % (predicates.len() + 1); + predicates.insert( + at, + ValidityPredicate::BlockNumber { + op: ValidityOperator::LessThanOrEqual, + value: U256::from(EXPIRY_BLOCK), + }, + ); + } + for _ in 1..timing { + let at = (state / 3 + rotation) % (predicates.len() + 1); + predicates.insert( + at, + ValidityPredicate::FlashblockIndex { + op: ValidityOperator::GreaterThanOrEqual, + value: U256::ONE, + }, + ); + } + predicates +} + +/// State under which every satisfied predicate from [`mixed`] holds. +fn database(predicates: impl IntoIterator) -> InMemoryDB { + let mut db = InMemoryDB::default(); + for predicate in predicates { + match predicate { + ValidityPredicate::Balance { address, .. } + | ValidityPredicate::Nonce { address, .. } => { + db.insert_account_info( + address, + AccountInfo { + balance: U256::from(BALANCE), + nonce: NONCE, + ..Default::default() + }, + ); + } + ValidityPredicate::Storage { address, slot, .. } => { + db.insert_account_info( + address, + AccountInfo { + balance: U256::from(BALANCE), + nonce: NONCE, + ..Default::default() + }, + ); + db.insert_account_storage(address, slot, slot).expect("in-memory insert"); + } + ValidityPredicate::BlockNumber { .. } | ValidityPredicate::FlashblockIndex { .. } => {} + } + } + db +} + +struct Mix { + name: &'static str, + timing: usize, + state: usize, +} + +impl Mix { + fn predicates(&self) -> Vec { + mixed(self.timing, self.state, 0, 0, false) + } +} + +const MIXES: &[Mix] = &[ + Mix { name: "expiry", timing: 1, state: 0 }, + Mix { name: "nonce+expiry", timing: 1, state: 1 }, + Mix { name: "3state+expiry", timing: 1, state: 3 }, + Mix { name: "6state+2timing", timing: 2, state: 6 }, + Mix { name: "14state+2timing", timing: 2, state: 14 }, + Mix { name: "62state+2timing", timing: 2, state: MAX_PREDICATES - 2 }, +]; + +fn transfer(predicates: Vec) -> BasePooledTransaction { + let tx = TxEip1559 { + chain_id: 8453, + nonce: NONCE, + gas_limit: 21_000, + max_fee_per_gas: 100, + max_priority_fee_per_gas: 1, + to: TxKind::Call(Address::ZERO), + value: U256::ONE, + access_list: Default::default(), + input: Bytes::new(), + }; + let envelope = BaseTxEnvelope::Eip1559(tx.into_signed(Signature::test_signature())); + let encoded_length = envelope.encode_2718_len(); + BasePooledTransaction::new( + Recovered::new_unchecked(BaseTransactionSigned::from(envelope), address(0)), + encoded_length, + ) + .with_validity_predicates(predicates) +} + +/// Decodes the batch from JSON the way the RPC path does, so spare `Vec` capacity left by +/// the decoder is part of what the pool ingests. +fn json_decoded(predicates: Vec) -> TransactionValidity { + let json = serde_json::to_string(&TransactionValidity { + validity: predicates, + validity_signature: None, + }) + .expect("serializes"); + serde_json::from_str(&json).expect("deserializes") +} + +fn pooled(transaction: BasePooledTransaction) -> ValidPoolTransaction { + ValidPoolTransaction { + transaction, + transaction_id: TransactionId::new(0.into(), NONCE), + propagate: false, + timestamp: Instant::now(), + origin: TransactionOrigin::Private, + authority_ids: None, + } +} + +fn report_memory() { + let plain_size = transfer(Vec::new()).size(); + println!( + "retained memory per compiled batch (bytes); size_of::() = {}, size_of::() = {}", + core::mem::size_of::(), + core::mem::size_of::() + ); + println!( + "{:<16} {:>5} {:>9} {:>9} {:>6} {:>7} {:>8} {:>10} {:>8} {:>8} {:>9}", + "mix", + "preds", + "heap_size", + "total", + "allocs", + "alloc_B", + "json_cap", + "json_total", + "tx_exact", + "tx_json", + "tx_growth" + ); + for mix in MIXES { + let exact = mix.predicates(); + let count = exact.len(); + let (conditions, allocations, allocated_bytes) = + allocations_during(|| ValidityConditions::new(exact)); + let total = core::mem::size_of::() + conditions.heap_size(); + + let decoded = json_decoded(mix.predicates()); + let json_capacity = decoded.validity.capacity(); + let json_transaction = transfer(Vec::new()).with_validity(decoded); + let json_total = core::mem::size_of::() + + json_transaction.validity_conditions().heap_size(); + let json_tx_size = json_transaction.size(); + let tx_size = transfer(mix.predicates()).size(); + + let mut db = database(mix.predicates()); + let (passed, eval_allocations, _) = allocations_during(|| { + ValidityPredicateKey::first_unsatisfied(&conditions, &mut db, &CONTEXT) + .expect("in-memory reads cannot fail") + }); + assert!(passed.is_none(), "{}: all predicates must pass", mix.name); + assert_eq!(eval_allocations, 0, "{}: successful evaluation must not allocate", mix.name); + if let Some(last) = conditions.state_predicates().last() { + let (hit, membership_allocations, _) = allocations_during(|| conditions.contains(last)); + assert!(hit); + assert_eq!(membership_allocations, 0, "{}: membership must not allocate", mix.name); + } + + println!( + "{:<16} {:>5} {:>9} {:>9} {:>6} {:>7} {:>8} {:>10} {:>8} {:>8} {:>8.1}%", + mix.name, + count, + conditions.heap_size(), + total, + allocations, + allocated_bytes, + json_capacity, + json_total, + tx_size, + json_tx_size, + (json_tx_size as f64 / plain_size as f64 - 1.0) * 100.0, + ); + } + println!( + "plain transfer pool size = {plain_size} B; tx_growth = accounted growth of the JSON-decoded transaction over that transfer" + ); + println!(); +} + +fn ingest(c: &mut Criterion) { + let mut group = c.benchmark_group("validity_conditions/ingest"); + for mix in MIXES { + group.bench_function(BenchmarkId::new("exact_capacity", mix.name), |b| { + b.iter_batched(|| mix.predicates(), ValidityConditions::new, BatchSize::SmallInput); + }); + group.bench_function(BenchmarkId::new("json_decoded", mix.name), |b| { + b.iter_batched( + || json_decoded(mix.predicates()).validity, + ValidityConditions::new, + BatchSize::SmallInput, + ); + }); + } + group.finish(); +} + +/// Evaluation before predicates were compiled at ingestion: scan the submitted batch in order. +fn full_scan( + predicates: &[ValidityPredicate], + db: &mut InMemoryDB, + context: &PredicateContext, +) -> Option { + predicates + .iter() + .position(|predicate| !predicate.matches(db, context).expect("in-memory reads cannot fail")) +} + +fn evaluate(c: &mut Criterion) { + let mut group = c.benchmark_group("validity_conditions/evaluate"); + for mix in MIXES { + let submitted = mix.predicates(); + let conditions = ValidityConditions::new(mix.predicates()); + let mut db = database(mix.predicates()); + group.bench_function(BenchmarkId::new("pass", mix.name), |b| { + b.iter(|| { + let result = + ValidityPredicateKey::first_unsatisfied(&conditions, &mut db, &CONTEXT) + .expect("in-memory reads cannot fail"); + assert!(result.is_none()); + }); + }); + group.bench_function(BenchmarkId::new("timing_reject", mix.name), |b| { + b.iter(|| { + let result = + ValidityPredicateKey::first_unsatisfied(&conditions, &mut db, &EXPIRED_CONTEXT) + .expect("in-memory reads cannot fail"); + assert!(result.is_some()); + }); + }); + // Baseline: the pre-compilation evaluation, which scanned the submitted batch in wire + // order and evaluated timing predicates as ordinary predicates. + group.bench_function(BenchmarkId::new("full_scan_pass", mix.name), |b| { + b.iter(|| { + let result = full_scan(&submitted, &mut db, &CONTEXT); + assert!(result.is_none()); + }); + }); + group.bench_function(BenchmarkId::new("full_scan_timing_reject", mix.name), |b| { + b.iter(|| { + let result = full_scan(&submitted, &mut db, &EXPIRED_CONTEXT); + assert!(result.is_some()); + }); + }); + if mix.state > 0 { + let failing = ValidityConditions::new(mixed(mix.timing, mix.state, 0, 0, true)); + group.bench_function(BenchmarkId::new("last_state_fails", mix.name), |b| { + b.iter(|| { + let result = + ValidityPredicateKey::first_unsatisfied(&failing, &mut db, &CONTEXT) + .expect("in-memory reads cannot fail"); + assert!(result.is_some()); + }); + }); + } + } + group.finish(); +} + +/// Evaluates every transaction of a pool-sized set once per iteration, so the working set is +/// many compiled batches rather than one hot batch. Each batch uses distinct accounts and a +/// different timing-predicate position. +fn evaluate_pool(c: &mut Criterion) { + let mut group = c.benchmark_group("validity_conditions/evaluate_pool"); + group.sample_size(20); + group.throughput(Throughput::Elements(POOL_TRANSACTIONS as u64)); + for mix in MIXES.iter().filter(|mix| mix.state > 0 && mix.state <= 14) { + let batches: Vec> = (0..POOL_TRANSACTIONS) + .map(|index| mixed(mix.timing, mix.state, index * mix.state, index, false)) + .collect(); + let mut db = database(batches.iter().flatten().cloned()); + let conditions: Vec = + batches.into_iter().map(ValidityConditions::new).collect(); + group.bench_function(BenchmarkId::new("pass", mix.name), |b| { + b.iter(|| { + for conditions in &conditions { + let result = + ValidityPredicateKey::first_unsatisfied(conditions, &mut db, &CONTEXT) + .expect("in-memory reads cannot fail"); + black_box(result.is_none()); + } + }); + }); + } + group.finish(); +} + +fn membership(c: &mut Criterion) { + let mut group = c.benchmark_group("validity_conditions/contains"); + let missing = ValidityPredicate::Balance { + address: Address::repeat_byte(0xff), + op: ValidityOperator::Equal, + value: U256::ZERO, + }; + for mix in MIXES.iter().filter(|mix| mix.state > 0) { + let conditions = ValidityConditions::new(mix.predicates()); + let last = conditions.state_predicates().last().expect("state predicate").clone(); + group.bench_function(BenchmarkId::new("hit_last_state", mix.name), |b| { + b.iter(|| assert!(conditions.contains(black_box(&last)))); + }); + group.bench_function(BenchmarkId::new("miss", mix.name), |b| { + b.iter(|| assert!(!conditions.contains(black_box(&missing)))); + }); + } + group.finish(); +} + +/// Builder forwarding: extracting the wire sidecar from a pooled transaction. +fn forward(c: &mut Criterion) { + let mut group = c.benchmark_group("validity_conditions/forward"); + for mix in MIXES { + let transaction = pooled(transfer(mix.predicates())); + group.bench_function(BenchmarkId::new("extract", mix.name), |b| { + b.iter(|| black_box(TransactionValidity::extract(&transaction))); + }); + } + group.finish(); +} + +fn main() { + report_memory(); + let mut criterion = Criterion::default().configure_from_args(); + ingest(&mut criterion); + evaluate(&mut criterion); + evaluate_pool(&mut criterion); + membership(&mut criterion); + forward(&mut criterion); + criterion.final_summary(); +} diff --git a/crates/execution/payload/src/builder.rs b/crates/execution/payload/src/builder.rs index 87f6c22415..cb549063f7 100644 --- a/crates/execution/payload/src/builder.rs +++ b/crates/execution/payload/src/builder.rs @@ -48,6 +48,7 @@ use reth_revm::{ witness::ExecutionWitnessRecord, }; use reth_storage_api::{BlockReader, StateProvider, StateProviderFactory, errors::ProviderError}; +use reth_tasks::Runtime; use reth_transaction_pool::{BestTransactionsAttributes, PoolTransaction, TransactionPool}; use reth_trie_common::ExecutionWitnessMode; use reth_trie_parallel::state_root_task::PayloadStateRootHandle; @@ -56,9 +57,10 @@ use tracing::{debug, debug_span, info, instrument, trace, warn}; use crate::{ Attributes, BasePayloadBuilderAttributes, BlockDeferrals, BuilderMetrics, - CoinbaseTipAffordability, InclusionTracker, MeteringProvider, ParkableBestPayloadTransactions, - ParkablePayloadTransactions, ParkedPredicateIndex, PayloadPrimitives, PredicateLoadTracker, - PredicateReadRecorder, RejectionCacheMetrics, StateChangeEffects, ValidityMetrics, + CoinbaseTipAffordability, HoldInterrupt, InclusionTracker, MeteringProvider, + ParkableBestPayloadTransactions, ParkablePayloadTransactions, ParkedPredicateIndex, + PayloadPrimitives, PredicateLoadTracker, PredicateReadRecorder, RejectionCacheMetrics, + RestingPredicateStore, SharedRestingPredicates, StateChangeEffects, ValidityMetrics, ValidityPredicateEvaluation, config::BaseBuilderConfig, error::BasePayloadBuilderError, payload::BaseBuiltPayload, }; @@ -112,6 +114,8 @@ pub struct BasePayloadBuilder< pub client: Client, /// Settings for the builder, e.g. DA settings. pub config: BaseBuilderConfig, + /// Validity transactions resting from the last block this builder finished. + pub resting_predicates: RestingPredicateStore, /// Marker for the payload attributes type. _pd: PhantomData, } @@ -128,11 +132,20 @@ where pool: self.pool.clone(), client: self.client.clone(), config: self.config.clone(), + resting_predicates: self.resting_predicates.clone(), _pd: PhantomData, } } } +/// Longest a native build waits for the previous block's resting sweep before starting with an +/// empty set. +/// +/// The native builder has no leeway: the sweep overlaps only the previous block's state root, and +/// any wait comes out of a short block's selection time, where re-evaluating every validity +/// transaction is cheaper than a long wait. +const RESTING_SWEEP_WAIT_LIMIT: Duration = Duration::from_millis(25); + impl BasePayloadBuilder { /// `BasePayloadBuilder` constructor. /// @@ -142,13 +155,27 @@ impl BasePayloadBuilder { } /// Configures the builder with the given [`BaseBuilderConfig`]. - pub const fn with_builder_config( + pub fn with_builder_config( pool: Pool, client: Client, evm_config: Evm, config: BaseBuilderConfig, ) -> Self { - Self { pool, client, evm_config, config, _pd: PhantomData } + Self { + pool, + client, + evm_config, + config, + resting_predicates: RestingPredicateStore::default(), + _pd: PhantomData, + } + } + + /// Runs end-of-block maintenance of the resting validity set, such as the unseen sweep, on + /// `executor` instead of inline. + pub fn with_task_executor(mut self, executor: Runtime) -> Self { + self.resting_predicates = RestingPredicateStore::new(executor); + self } } @@ -178,7 +205,10 @@ where fn build_payload<'a, Txs>( &self, args: BuildArguments>, - best: impl FnOnce(BestTransactionsAttributes) -> Txs + Send + Sync + 'a, + best: impl FnOnce(BestTransactionsAttributes, Option<&SharedRestingPredicates>) -> Txs + + Send + + Sync + + 'a, ) -> Result>, PayloadBuilderError> where Txs: ParkablePayloadTransactions< @@ -206,9 +236,11 @@ where tracing::Span::current().record("parent_num", ctx.parent().number()); let pool = self.pool.clone(); - let builder = Builder::new(best).with_permanent_eviction(move |hashes| { - let _ = pool.remove_transactions(hashes); - }); + let builder = Builder::new(best) + .with_permanent_eviction(move |hashes| { + let _ = pool.remove_transactions(hashes); + }) + .with_resting_predicates(self.resting_predicates.clone()); let state_provider = crate::BuilderStateProvider::new( self.client.state_by_block_hash(ctx.parent().hash())?, @@ -259,7 +291,7 @@ where let state_provider = self.client.state_by_block_hash(ctx.parent().hash())?; - let builder = Builder::new(|_| NoopPayloadTransactions::::default()); + let builder = Builder::new(|_, _| NoopPayloadTransactions::::default()); builder.witness(state_provider, &self.client, &ctx) } } @@ -284,9 +316,12 @@ where args: BuildArguments, ) -> Result, PayloadBuilderError> { let no_tx_pool = args.config.attributes.no_tx_pool(); - let outcome = self.build_payload(args, |attrs| { + let outcome = self.build_payload(args, |attrs, resting| { ParkableBestPayloadTransactions::new( - self.pool.best_transactions_with_attributes_and_parking(attrs), + self.pool.best_transactions_with_attributes_and_parking( + attrs, + resting.map(SharedRestingPredicates::iterator_hold), + ), ) })?; @@ -326,12 +361,25 @@ where cancel: Default::default(), best_payload: None, }; - self.build_payload(args, |_| NoopPayloadTransactions::::default())? + self.build_payload(args, |_, _| NoopPayloadTransactions::::default())? .into_payload() .ok_or_else(|| PayloadBuilderError::MissingPayload) } } +/// Interrupts a pool iterator's walk past held resting transactions once its job is cancelled or +/// asked to finalize, as the build loop checks before each candidate. Resting transactions are +/// only held from Denim on, where the build loop honours finalization requests. +fn job_interrupt(cancel: &Arc) -> HoldInterrupt { + let cancel = Arc::downgrade(cancel); + Arc::new(move || cancel.upgrade().is_none_or(|cancel| cancel.is_interrupted())) +} + +/// Creates the candidate iterator of one build from the fee attributes and, for an enforcing +/// build, the resting set whose hold the iterator parks members with. +pub type BestTransactionsFactory<'a, Txs> = + dyn FnOnce(BestTransactionsAttributes, Option<&SharedRestingPredicates>) -> Txs + 'a; + /// The type that builds the payload. /// /// Payload building for Base is composed of several steps. @@ -351,16 +399,27 @@ where pub struct Builder<'a, Txs> { /// Yields the best transaction to include if transactions from the mempool are allowed. #[debug(skip)] - best: Box Txs + 'a>, + best: Box>, /// Permanently removes mempool transactions that exceeded a per-transaction resource limit. #[debug(skip)] evict_permanently_rejected: Box) + 'a>, + /// Carries resting validity transactions between builds when the carried mode is enabled. + resting_predicates: Option, } impl<'a, Txs> Builder<'a, Txs> { /// Creates a new [`Builder`]. - pub fn new(best: impl FnOnce(BestTransactionsAttributes) -> Txs + Send + Sync + 'a) -> Self { - Self { best: Box::new(best), evict_permanently_rejected: Box::new(|_| {}) } + pub fn new( + best: impl FnOnce(BestTransactionsAttributes, Option<&SharedRestingPredicates>) -> Txs + + Send + + Sync + + 'a, + ) -> Self { + Self { + best: Box::new(best), + evict_permanently_rejected: Box::new(|_| {}), + resting_predicates: None, + } } /// Sets the callback used to permanently remove mempool transactions that exceeded a @@ -369,6 +428,13 @@ impl<'a, Txs> Builder<'a, Txs> { self.evict_permanently_rejected = Box::new(evict); self } + + /// Sets the store that carries resting validity transactions from one finished block to the + /// next. + pub fn with_resting_predicates(mut self, store: RestingPredicateStore) -> Self { + self.resting_predicates = Some(store); + self + } } impl Builder<'_, Txs> { @@ -392,7 +458,11 @@ impl Builder<'_, Txs> { >, Attrs: Attributes, { - let Self { best, evict_permanently_rejected } = self; + let Self { best, evict_permanently_rejected, resting_predicates } = self; + // The pool iterator's hold watches the job's cancellation through a weak reference. A + // dropped `CancelOnDrop` clone cancels the job, so only this owner, dropped with `ctx` when + // the build returns, keeps one. + let cancel_watch = Arc::new(ctx.cancel.clone()); debug!(target: "payload_builder", id=%ctx.payload_id(), parent_header = ?ctx.parent().hash(), parent_number = ctx.parent().number(), "building new payload"); let mut db = State::builder().with_database(db).with_bundle_update().build(); @@ -418,9 +488,39 @@ impl Builder<'_, Txs> { let mut info = ctx.execute_sequencer_transactions(&mut builder)?; // 3. if mem pool transactions are requested we execute them + let mut resting = None; if !ctx.attributes().no_tx_pool() { - let best_txs = best(ctx.best_transaction_attributes(builder.evm_mut().block())); - let cancelled = ctx.execute_best_transactions(&mut info, &mut builder, best_txs)?; + // Without flashblocks, resting transactions only carry across blocks, from Denim on. + resting = resting_predicates + .as_ref() + .filter(|_| { + ctx.builder_config.resting_predicate_mode.is_enabled() && ctx.is_denim_active() + }) + .map(|store| { + let start = Instant::now(); + let mut resting = store.take( + ctx.builder_config.predicate_bucket_ordered_threshold, + RESTING_SWEEP_WAIT_LIMIT, + ); + // The transitions hold the system calls and deposits applied on top of the + // parent state. + let carry_over = resting.begin_block( + ctx.parent().hash(), + ctx.parent().number().saturating_add(1), + builder.evm_mut().db_mut().transition_state.as_ref(), + ); + ValidityMetrics::record_resting_carry_over(&carry_over, start.elapsed()); + SharedRestingPredicates::new(resting) + .with_interrupt(job_interrupt(&cancel_watch)) + }); + // Only an enforcing build hands the pool iterator a hold; a shadow build evaluates + // resting members to count mismatches. + let hold = resting + .as_ref() + .filter(|_| ctx.builder_config.resting_predicate_mode.is_enforced()); + let best_txs = best(ctx.best_transaction_attributes(builder.evm_mut().block()), hold); + let cancelled = + ctx.execute_best_transactions(&mut info, &mut builder, best_txs, resting.as_ref())?; if !info.permanently_rejected_txs.is_empty() { let rejected = std::mem::take(&mut info.permanently_rejected_txs); let count = rejected.len(); @@ -449,6 +549,10 @@ impl Builder<'_, Txs> { return Ok(BuildOutcomeKind::Aborted { fees: info.total_fees }); } } + // Start the end-of-block sweep now so it overlaps the state root. + let resting = resting + .zip(resting_predicates.as_ref()) + .map(|(resting, store)| store.finish(resting.take_predicates())); let block_num = ctx.parent().number().saturating_add(1); let state_root = state_root_handle.and_then(|mut task| { @@ -487,6 +591,9 @@ impl Builder<'_, Txs> { let sealed_block = Arc::new(block.sealed_block().clone()); debug!(target: "payload_builder", id=%ctx.payload_id(), sealed_block_header = ?sealed_block.header(), "sealed built block"); + if let (Some(resting), Some(store)) = (resting, resting_predicates) { + store.put(resting, sealed_block.hash()); + } let execution_outcome = BlockExecutionOutput { state: db.take_bundle(), result: execution_result }; @@ -909,6 +1016,7 @@ where mut best_txs: impl ParkablePayloadTransactions< Transaction: PoolTransaction> + BasePooledTx, >, + resting: Option<&SharedRestingPredicates>, ) -> Result, PayloadBuilderError> where Builder: BlockBuilder, @@ -941,14 +1049,21 @@ where let mut validity_candidates_evaluated = 0_u64; let mut validity_candidates_deferred = 0_u64; let mut predicate_eval_cutoff_hit = false; + let resting_mode = self.builder_config.resting_predicate_mode; + // Counted locally and recorded once per build, so the loop does no metric lookups. + let mut carried_shadow_mismatches = 0_u64; let block_timestamp = self.attributes().timestamp(); let can_finalize_early = self.is_denim_active(); let resource_metering = &self.builder_config.resource_metering; let mut resource_throttled = 0u64; + // Set when the job is cancelled mid-scan. The loop breaks instead of returning so the + // per-build validity metrics below are still recorded for scans cut short. + let mut cancelled = false; while let Some(tx) = best_txs.next(()) { if self.cancel.is_cancelled() { - return Ok(Some(())); + cancelled = true; + break; } if can_finalize_early && self.cancel.is_finalization_requested() { break; @@ -1046,6 +1161,25 @@ where }; *predicate_eval_duration.get_or_insert(Duration::ZERO) += evaluation_start.elapsed(); + if let Some(resting) = resting { + match &evaluation { + Ok(ValidityPredicateEvaluation::Unsatisfied { + predicate, + expired: false, + .. + }) => resting.rest(tx_hash, predicate.clone()), + Ok(ValidityPredicateEvaluation::Matched) + if !resting_mode.is_enforced() + && resting.member(tx_hash, tx.validity_conditions()).is_some() => + { + carried_shadow_mismatches += 1; + resting.remove(tx_hash); + } + _ => { + resting.remove(tx_hash); + } + } + } match evaluation { Ok(ValidityPredicateEvaluation::Matched) => { ValidityMetrics::validity_predicate_evaluations_total("matched") @@ -1300,10 +1434,12 @@ where } if self.cancel.is_cancelled() { - return Ok(Some(())); + cancelled = true; + break; } let mut state_change_effects = StateChangeEffects::default(); + let mut resting_woken = Vec::new(); let mut pending_resource_usage = None; let mut executed_decision = None; let gas_output = match builder.execute_transaction_with_commit_condition( @@ -1327,6 +1463,9 @@ where predicate_index.affected_by_state(&result_and_state.state); predicate_bucket_wakeups += state_change_effects.woken_buckets as u64; } + if let Some(resting) = resting { + resting_woken = resting.wake_by_state(&result_and_state.state); + } CommitChanges::Yes } }, @@ -1408,6 +1547,12 @@ where } best_txs.mark_current_committed(); + // Members the pool iterator held are parked in it; a commit that changed the state + // their predicate reads promotes them so the scan evaluates them at their priority + // position. + for woken in resting_woken { + best_txs.promote(woken); + } let predicates_need_rescan = !state_change_effects.affected_transactions.is_empty(); let predicate_rescan_start = Instant::now(); for (rescanned, parked_hash) in @@ -1446,6 +1591,9 @@ where ValidityMetrics::validity_predicate_evaluations_total("rescan_matched") .increment(1); predicate_index.remove(parked_hash); + if let Some(resting) = resting { + resting.remove(parked_hash); + } best_txs.promote(parked_hash); } Ok(ValidityPredicateEvaluation::Unsatisfied { predicate, .. }) => { @@ -1453,6 +1601,9 @@ where "rescan_not_satisfied", ) .increment(1); + if let Some(resting) = resting { + resting.rest(parked_hash, predicate.clone()); + } predicate_index.reindex(parked_hash, predicate); } Err(error) => { @@ -1465,6 +1616,9 @@ where "failed to re-read validity predicate state" ); predicate_index.remove(parked_hash); + if let Some(resting) = resting { + resting.remove(parked_hash); + } best_txs.discard_parked(parked_hash); } } @@ -1517,6 +1671,18 @@ where predicate_bucket_wakeups, &predicate_index, ); + if let Some(stats) = resting.map(SharedRestingPredicates::take_stats) + && stats.parked > 0 + { + ValidityMetrics::resting_predicate_parked_total().increment(stats.parked); + ValidityMetrics::resting_predicate_carried_parked_total() + .increment(stats.carried_parked); + ValidityMetrics::record_resting_hold_duration(stats.duration); + } + if carried_shadow_mismatches > 0 { + ValidityMetrics::resting_predicate_shadow_mismatches_total() + .increment(carried_shadow_mismatches); + } if resource_throttled > 0 { info!( target: "payload_builder", @@ -1526,6 +1692,9 @@ where ); } + if cancelled { + return Ok(Some(())); + } // A cancellation that raced the finalization break (or an empty iterator) must still // win, so re-check it before the finalized payload is assembled. Gated on Denim so // pre-Denim control flow is unchanged. @@ -1556,7 +1725,9 @@ mod tests { use base_common_evm::BaseTime; use base_execution_chainspec::{BaseChainSpec, BaseChainSpecBuilder}; use base_execution_evm::BaseEvmConfig; - use base_execution_txpool::{BasePooledTransaction, ValidityOperator, ValidityPredicate}; + use base_execution_txpool::{ + BasePooledTransaction, BestTransactionsHold, ValidityOperator, ValidityPredicate, + }; use base_observability_events::{TransactionEventCapture, TransactionEventType}; use reth_basic_payload_builder::{BuildOutcomeKind, PayloadConfig}; use reth_chainspec::ChainSpec; @@ -1570,7 +1741,9 @@ mod tests { cancelled::CancelOnDrop, database::StateProviderDatabase, db::State, test_utils::StateProviderTest, }; - use reth_transaction_pool::PoolTransaction; + use reth_transaction_pool::{ + PoolTransaction, TransactionOrigin, ValidPoolTransaction, identifier::TransactionId, + }; use reth_trie_common::{HashedPostState, updates::TrieUpdates}; use reth_trie_parallel::{ error::StateRootTaskError, @@ -1578,14 +1751,17 @@ mod tests { PayloadStateRootHandle, StateRootComputeOutcome, StateRootSink, StateRootUpdateStream, }, }; - use revm::{Database, state::EvmState}; + use revm::{Database, database::TransitionState, state::EvmState}; - use super::{BasePayloadBuilderCtx, Builder, ExecutionInfo}; + use super::{ + BasePayloadBuilderCtx, Builder, ExecutionInfo, RESTING_SWEEP_WAIT_LIMIT, job_interrupt, + }; use crate::{ - BasePayloadBuilderAttributes, MeteringProvider, NoopMeteringProvider, - ParkablePayloadTransactions, ResourceMeteringConfig, ResourceMeteringDimension, - ResourceMeteringOperation, ResourceMeteringSchedule, SharedMeteringProvider, - config::BaseBuilderConfig, payload::EthPayloadBuilderAttributes, + BasePayloadBuilderAttributes, DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, MeteringProvider, + NoopMeteringProvider, ParkablePayloadTransactions, ResourceMeteringConfig, + ResourceMeteringDimension, ResourceMeteringOperation, ResourceMeteringSchedule, + RestingPredicateMode, RestingPredicateStore, SharedMeteringProvider, + SharedRestingPredicates, config::BaseBuilderConfig, payload::EthPayloadBuilderAttributes, }; #[derive(Debug)] @@ -1642,7 +1818,8 @@ mod tests { best_payload: None, }; let provider = NoopProvider::default(); - let builder = Builder::new(|_| NoopPayloadTransactions::::default()); + let builder = + Builder::new(|_, _| NoopPayloadTransactions::::default()); let outcome = builder .build(StateProviderDatabase::new(&provider), &provider, Some(state_root_handle), ctx) .expect("empty payload must build"); @@ -1749,6 +1926,15 @@ mod tests { where Txs: ParkablePayloadTransactions + Send + Sync, { + let provider = funded_provider(funded_senders); + Builder::new(|_, _| transactions) + .build(StateProviderDatabase::new(&provider), &provider, Some(state_root_handle()), ctx) + .expect("payload must build") + } + + /// A parent state with the `BaseTime` predeploy and `funded_senders` holding the maximum + /// balance. + fn funded_provider(funded_senders: &[Address]) -> StateProviderTest { let mut storage = HashMap::default(); storage.insert( StorageKey::from(BaseTime::ADMIN_SLOT.to_be_bytes::<32>()), @@ -1769,9 +1955,7 @@ mod tests { HashMap::default(), ); } - Builder::new(|_| transactions) - .build(StateProviderDatabase::new(&provider), &provider, Some(state_root_handle()), ctx) - .expect("payload must build") + provider } fn build_pool_payload_with( @@ -1795,27 +1979,8 @@ mod tests { where Txs: ParkablePayloadTransactions + Send + Sync, { - let mut storage = HashMap::default(); - storage.insert( - StorageKey::from(BaseTime::ADMIN_SLOT.to_be_bytes::<32>()), - U256::from_be_slice(Predeploys::PROXY_ADMIN.as_slice()), - ); - let mut provider = StateProviderTest::default(); - provider.insert_account( - Predeploys::BASE_TIME, - Account::default(), - Some(BaseTime::proxy_bytecode()), - storage, - ); - for sender in funded_senders { - provider.insert_account( - *sender, - Account { balance: U256::MAX, ..Default::default() }, - None, - HashMap::default(), - ); - } - Builder::new(|_| transactions) + let provider = funded_provider(funded_senders); + Builder::new(|_, _| transactions) .with_permanent_eviction(evict) .build(StateProviderDatabase::new(&provider), &provider, Some(state_root_handle()), ctx) .expect("payload must build") @@ -1857,6 +2022,8 @@ mod tests { /// they outbid them. promoted_first: bool, invalid: Arc>>, + /// Parks held candidates before yielding them, as the pool's parking iterator does. + hold: Option>>, } impl TestParkableTransactions { @@ -1876,9 +2043,19 @@ mod tests { current: None, promoted_first: true, invalid, + hold: None, } } + /// Parks the candidates `hold` holds before yielding them. + fn with_hold( + mut self, + hold: Option>>, + ) -> Self { + self.hold = hold; + self + } + /// Yields promoted transactions only after every queued one, so queued transactions can /// change state between a promotion and the re-yield. fn with_promoted_last(transactions: Vec) -> Self { @@ -1891,13 +2068,29 @@ mod tests { fn next(&mut self, _ctx: ()) -> Option { assert!(self.current.is_none(), "current transaction was not lifecycle-managed"); - let transaction = if self.promoted_first { - self.ready.pop_front().or_else(|| self.queued.pop_front()) - } else { - self.queued.pop_front().or_else(|| self.ready.pop_front()) - }?; - self.current = Some(transaction.clone()); - Some(transaction) + loop { + let transaction = if self.promoted_first { + self.ready.pop_front().or_else(|| self.queued.pop_front()) + } else { + self.queued.pop_front().or_else(|| self.ready.pop_front()) + }?; + if let Some(hold) = self.hold.as_mut() + && !transaction.validity_conditions().is_empty() + && hold.hold(&ValidPoolTransaction { + transaction_id: TransactionId::new(0.into(), 0), + transaction: transaction.clone(), + propagate: false, + timestamp: std::time::Instant::now(), + origin: TransactionOrigin::External, + authority_ids: None, + }) + { + self.parked.insert(*transaction.hash(), transaction); + continue; + } + self.current = Some(transaction.clone()); + return Some(transaction); + } } fn mark_invalid(&mut self, sender: Address, nonce: u64) { @@ -2008,8 +2201,10 @@ mod tests { #[test] fn native_builder_includes_transaction_with_matching_block_predicate() { let event_capture = TransactionEventCapture::install(); - let transaction = - pool_transaction(0).with_validity_predicates(vec![ValidityPredicate::BlockNumber { + // Event capture is process-global, so this transaction must not share a hash with other + // tests, such as the flashblock-index rejection test that uses `pool_transaction(0)`. + let transaction = pool_transaction_to(0, Address::repeat_byte(0x4d), U256::ZERO) + .with_validity_predicates(vec![ValidityPredicate::BlockNumber { op: ValidityOperator::Equal, value: U256::ONE, }]); @@ -2215,6 +2410,180 @@ mod tests { assert_eq!(included_hashes, vec![trigger_hash, gated_hash]); } + /// A store holding a set finished as `tip` in which `transaction` rests under `predicate`. + fn resting_store( + tip: B256, + transaction: &BasePooledTransaction, + predicate: ValidityPredicate, + ) -> RestingPredicateStore { + let store = RestingPredicateStore::default(); + let mut resting = + store.take(DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, RESTING_SWEEP_WAIT_LIMIT); + resting.begin_block(B256::ZERO, 0, Some(&TransitionState::default())); + resting.rest(*transaction.hash(), predicate); + store.put(store.finish(resting), tip); + store + } + + /// Builds one block at `timestamp` with the resting `mode`, returning the included hashes and + /// the block hash. + fn build_carrying_resting_at( + timestamp: u64, + mode: RestingPredicateMode, + store: &RestingPredicateStore, + transactions: Vec, + ) -> (Vec, B256) { + let mut ctx = pool_payload_context(timestamp); + ctx.builder_config.resting_predicate_mode = mode; + let funded_senders = + transactions.iter().map(|transaction| transaction.sender()).collect::>(); + let provider = funded_provider(&funded_senders); + let outcome = Builder::new(|_, resting| { + TestParkableTransactions::new(transactions) + .with_hold(resting.map(SharedRestingPredicates::iterator_hold)) + }) + .with_resting_predicates(store.clone()) + .build(StateProviderDatabase::new(&provider), &provider, Some(state_root_handle()), ctx) + .expect("payload must build"); + let (BuildOutcomeKind::Better { payload } | BuildOutcomeKind::Freeze(payload)) = outcome + else { + panic!("expected a built payload, got {outcome:?}") + }; + let included = payload.block().body().transactions.iter().map(|tx| tx.tx_hash()).collect(); + (included, payload.block().hash()) + } + + /// Builds one Denim block with the resting `mode`, returning the included hashes and the + /// block hash. + fn build_carrying_resting( + mode: RestingPredicateMode, + store: &RestingPredicateStore, + transactions: Vec, + ) -> (Vec, B256) { + build_carrying_resting_at(DENIM_TIMESTAMP, mode, store, transactions) + } + + fn pool_parent_hash() -> B256 { + pool_payload_context(DENIM_TIMESTAMP).parent().hash() + } + + /// The set records `gated` as resting although its predicate holds, so whether it is included + /// shows whether the build evaluated it. + #[test] + fn native_builder_skips_evaluating_carried_resting_transactions_only_when_enforced() { + let watched_address = Address::repeat_byte(0x47); + let holds = ValidityPredicate::Balance { + address: watched_address, + op: ValidityOperator::Equal, + value: U256::ZERO, + }; + let gated = pool_transaction_to(0, Address::repeat_byte(0x58), U256::ZERO) + .with_validity_predicates(vec![holds.clone()]); + let gated_hash = *gated.hash(); + + for (mode, evaluated) in [ + (RestingPredicateMode::Off, true), + (RestingPredicateMode::Shadow, true), + (RestingPredicateMode::Enforce, false), + ] { + let store = resting_store(pool_parent_hash(), &gated, holds.clone()); + let (included, _) = build_carrying_resting(mode, &store, vec![gated.clone()]); + assert_eq!(included.contains(&gated_hash), evaluated, "mode {mode:?}"); + } + } + + #[test] + fn native_builder_discards_a_resting_set_finished_on_another_parent() { + let watched_address = Address::repeat_byte(0x48); + let holds = ValidityPredicate::Balance { + address: watched_address, + op: ValidityOperator::Equal, + value: U256::ZERO, + }; + let gated = pool_transaction_to(0, Address::repeat_byte(0x59), U256::ZERO) + .with_validity_predicates(vec![holds.clone()]); + let gated_hash = *gated.hash(); + let store = resting_store(B256::repeat_byte(0x99), &gated, holds); + + let (included, _) = + build_carrying_resting(RestingPredicateMode::Enforce, &store, vec![gated]); + + assert_eq!(included, [gated_hash]); + } + + /// A commit that satisfies a carried predicate includes the transaction exactly where the + /// build without carried resting includes it. + #[test] + fn native_builder_includes_a_carried_resting_transaction_once_its_state_changes() { + let watched_address = Address::repeat_byte(0x49); + let blocker = ValidityPredicate::Balance { + address: watched_address, + op: ValidityOperator::Equal, + value: U256::ONE, + }; + let gated = pool_transaction_to(0, Address::repeat_byte(0x5a), U256::ZERO) + .with_validity_predicates(vec![blocker.clone()]); + let trigger = pool_transaction_to(0, watched_address, U256::ONE); + let expected = vec![*trigger.hash(), *gated.hash()]; + + for mode in [RestingPredicateMode::Off, RestingPredicateMode::Enforce] { + let store = resting_store(pool_parent_hash(), &gated, blocker.clone()); + let (included, _) = + build_carrying_resting(mode, &store, vec![gated.clone(), trigger.clone()]); + assert_eq!(included, expected, "mode {mode:?}"); + } + } + + #[test] + fn native_builder_carries_unsatisfied_transactions_into_the_next_block() { + let watched_address = Address::repeat_byte(0x4a); + let blocker = ValidityPredicate::Balance { + address: watched_address, + op: ValidityOperator::Equal, + value: U256::ONE, + }; + let gated = pool_transaction_to(0, Address::repeat_byte(0x5b), U256::ZERO) + .with_validity_predicates(vec![blocker]); + let store = RestingPredicateStore::default(); + + let (included, block_hash) = + build_carrying_resting(RestingPredicateMode::Enforce, &store, vec![gated.clone()]); + + assert!(included.is_empty()); + let mut resting = + store.take(DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, RESTING_SWEEP_WAIT_LIMIT); + let carry_over = resting.begin_block(block_hash, 2, Some(&TransitionState::default())); + assert_eq!(carry_over.carried, 1); + assert!(resting.member(*gated.hash(), gated.validity_conditions()).is_some()); + } + + /// Without flashblocks, resting transactions only carry across blocks, which needs Denim. + #[test] + fn native_builder_ignores_the_resting_set_when_off_or_before_denim() { + let watched_address = Address::repeat_byte(0x4b); + let blocker = ValidityPredicate::Balance { + address: watched_address, + op: ValidityOperator::Equal, + value: U256::ONE, + }; + let gated = pool_transaction_to(0, Address::repeat_byte(0x5c), U256::ZERO) + .with_validity_predicates(vec![blocker]); + + for (timestamp, mode) in [ + (DENIM_TIMESTAMP, RestingPredicateMode::Off), + (DENIM_TIMESTAMP - 1, RestingPredicateMode::Enforce), + ] { + let store = RestingPredicateStore::default(); + build_carrying_resting_at(timestamp, mode, &store, vec![gated.clone()]); + assert!( + store + .take(DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, RESTING_SWEEP_WAIT_LIMIT) + .is_empty(), + "timestamp {timestamp}, mode {mode:?}" + ); + } + } + #[test] fn native_builder_bounds_initial_and_rescan_predicate_evaluation() { let watched_address = Address::repeat_byte(0x44); @@ -2253,6 +2622,23 @@ mod tests { assert_eq!(included_hashes, vec![trigger_hash]); } + #[test] + fn job_interrupt_observes_the_job_without_cancelling_it() { + let cancel = CancelOnDrop::default(); + let watch = Arc::new(cancel.clone()); + + let interrupt = job_interrupt(&watch); + assert!(!interrupt()); + drop(interrupt); + assert!(!cancel.is_interrupted(), "dropping the hold's interrupt must not cancel the job"); + + let interrupt = job_interrupt(&watch); + cancel.request_finalization(); + assert!(interrupt(), "a finalization request interrupts the walk"); + drop(watch); + assert!(interrupt(), "a finished build interrupts the walk"); + } + #[test] fn cancellation_takes_precedence_over_finalization() { let ctx = pool_payload_context(DENIM_TIMESTAMP); @@ -2271,7 +2657,7 @@ mod tests { ctx.config.attributes.transactions = vec![sequencer_attribute_tx(&pool_transaction(0))]; drop(ctx.cancel.clone()); - let err = Builder::new(|_| NoopPayloadTransactions::::default()) + let err = Builder::new(|_, _| NoopPayloadTransactions::::default()) .witness(test_state_provider(), NoopProvider::default(), &ctx) .unwrap_err(); assert!(err.to_string().contains("cancelled"), "{err}"); @@ -2799,7 +3185,8 @@ mod tests { let invalid = Arc::new(Mutex::new(Vec::new())); let transactions = TestParkableTransactions::recording(vec![tx], invalid); - ctx.execute_best_transactions(&mut info, &mut builder, transactions).expect("mempool scan"); + ctx.execute_best_transactions(&mut info, &mut builder, transactions, None) + .expect("mempool scan"); let nonce_after = builder .evm_mut() .db_mut() diff --git a/crates/execution/payload/src/config.rs b/crates/execution/payload/src/config.rs index a0a0296dd3..63f5f83305 100644 --- a/crates/execution/payload/src/config.rs +++ b/crates/execution/payload/src/config.rs @@ -19,7 +19,8 @@ use tracing::{debug, warn}; use crate::{ DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD, MeteringProvider, NoopMeteringProvider, RejectionCache, ResourceMeteringError, ResourceMeteringMetrics, ResourceMeteringSchedule, - ResourceMeteringUsage, ResourceSample, ResourceThrottlingDecision, SharedMeteringProvider, + ResourceMeteringUsage, ResourceSample, ResourceThrottlingDecision, RestingPredicateMode, + SharedMeteringProvider, }; /// Settings for the Base payload builder. @@ -53,6 +54,9 @@ pub struct BaseBuilderConfig { /// label. Adds overhead to every state read, so it follows reth's /// `--engine.state-provider-metrics` and stays off by default. pub state_provider_metrics: bool, + /// Whether validity transactions whose predicate was found unsatisfied in an earlier block skip + /// evaluation until a state change wakes them. Applies from Denim onward. + pub resting_predicate_mode: RestingPredicateMode, } impl Default for BaseBuilderConfig { @@ -66,6 +70,7 @@ impl Default for BaseBuilderConfig { resource_metering: ResourceMeteringConfig::default(), rejection_cache: RejectionCache::default(), state_provider_metrics: false, + resting_predicate_mode: RestingPredicateMode::Off, } } } @@ -86,9 +91,17 @@ impl BaseBuilderConfig { resource_metering: ResourceMeteringConfig::default(), rejection_cache: RejectionCache::default(), state_provider_metrics: false, + resting_predicate_mode: RestingPredicateMode::Off, } } + /// Sets the mode for validity transactions resting from an earlier block. + #[must_use] + pub const fn with_resting_predicate_mode(mut self, mode: RestingPredicateMode) -> Self { + self.resting_predicate_mode = mode; + self + } + /// Sets whether build-loop state reads are timed. #[must_use] pub const fn with_state_provider_metrics(mut self, enabled: bool) -> Self { @@ -146,13 +159,15 @@ impl Default for ResourceMeteringConfig { impl ResourceMeteringConfig { /// Builds a shared config from startup flags. /// - /// `--enable-metering` turns on `base_meterBundle`. The native payload - /// builder throttles transactions against resource-unit budgets only when - /// that flag is set and a non-empty schedule file is loaded. A missing - /// schedule leaves those budgets inactive so mempool clients can serve - /// meterBundle without a builder schedule. An empty schedule file fails - /// closed because the operator asked to load a schedule and it has no - /// dimensions. + /// `--builder.enable-resource-metering` (`BUILDER_ENABLE_RESOURCE_METERING`) + /// gates builder metering on `base-builder` and `base sequencer`; + /// `--enable-metering` (`ENABLE_METERING`) gates `base_meterBundle` and the + /// standalone execution node's native schedule. The native payload builder + /// throttles transactions against resource-unit budgets only when `enabled` + /// is set and a non-empty schedule file is loaded. A missing schedule leaves + /// those budgets inactive so mempool clients can serve `base_meterBundle` + /// without a builder schedule. An empty schedule file fails closed because + /// the operator asked to load a schedule and it has no dimensions. pub fn from_parts( enabled: bool, schedule_path: Option<&Path>, diff --git a/crates/execution/payload/src/lib.rs b/crates/execution/payload/src/lib.rs index 3dd57ae6c2..5777492585 100644 --- a/crates/execution/payload/src/lib.rs +++ b/crates/execution/payload/src/lib.rs @@ -43,6 +43,13 @@ pub use metrics::{BuilderMetrics, ValidityMetrics}; mod deferrals; pub use deferrals::BlockDeferrals; +mod resting; +pub use resting::{ + FinishingRestingPredicates, HoldInterrupt, RESTING_UNSEEN_RETAINED_BLOCKS, RestingCarryOver, + RestingMember, RestingPredicateMode, RestingPredicateStore, RestingPredicates, RestingStats, + SharedRestingPredicates, +}; + mod inclusion; pub use inclusion::{FLOW_STANDARD, FLOW_VALIDITY, InclusionFlow, InclusionTracker}; diff --git a/crates/execution/payload/src/metrics.rs b/crates/execution/payload/src/metrics.rs index a0cd1ac843..bc0ae451fb 100644 --- a/crates/execution/payload/src/metrics.rs +++ b/crates/execution/payload/src/metrics.rs @@ -4,6 +4,7 @@ use std::time::Duration; use crate::{ FLOW_STANDARD, FLOW_VALIDITY, InclusionTracker, ParkedPredicateIndex, PredicateLoadTracker, + RestingCarryOver, }; base_metrics::define_metrics! { @@ -58,6 +59,29 @@ base_metrics::define_metrics! { #[describe("Validity predicate evaluation attempts")] #[label(outcome)] validity_predicate_evaluations_total: counter, + #[describe("Resting validity transactions carried into a block build from the previous block")] + resting_predicate_carried: histogram, + #[describe("Resting validity transactions released from the carried set, by reason")] + #[label(reason)] + resting_predicate_carry_released_total: counter, + #[describe("Block builds that discarded carried resting validity transactions because the parent was not the last block the builder finished")] + resting_predicate_carry_resets_total: counter, + #[describe("Resting validity transactions parked in the transaction iterator without reaching the build loop")] + resting_predicate_parked_total: counter, + #[describe("Time one native block build's transaction iterator spent advancing past held resting validity transactions, in seconds")] + resting_predicate_hold_duration: histogram, + #[describe("Resting validity transactions whose predicates matched when evaluated in shadow mode")] + resting_predicate_shadow_mismatches_total: counter, + #[describe("Of the parked resting validity transactions, those carried over from an earlier block")] + resting_predicate_carried_parked_total: counter, + #[describe("Duration of carrying resting validity transactions into a block build, on the build path before the selection loop, in seconds")] + resting_predicate_begin_block_duration: histogram, + #[describe("Duration of the periodic sweep of resting validity transactions no build yielded, run off the build path, in seconds")] + resting_predicate_sweep_duration: histogram, + #[describe("Time a block build waited for the previous block's resting sweep to finish, in seconds")] + resting_predicate_sweep_wait_duration: histogram, + #[describe("Block builds that stopped waiting for the previous block's resting sweep and started with an empty resting set")] + resting_predicate_sweep_wait_timeouts_total: counter, } base_metrics::define_metrics! { @@ -123,6 +147,48 @@ impl ValidityMetrics { Self::predicate_slots_loaded_unique().record(tracker.unique_slots() as f64); } + /// Records what carrying resting validity transactions into a new block did, and how long it + /// took. + pub fn record_resting_carry_over(carry_over: &RestingCarryOver, duration: Duration) { + Self::resting_predicate_begin_block_duration().record(duration.as_secs_f64()); + if carry_over.reset { + Self::resting_predicate_carry_resets_total().increment(1); + } + for (reason, released) in [ + ("pre_block_state", carry_over.released_by_state), + ("block_number", carry_over.released_by_block_number), + ] { + if released > 0 { + Self::resting_predicate_carry_released_total(reason).increment(released as u64); + } + } + Self::resting_predicate_carried().record(carry_over.carried as f64); + } + + /// Records one sweep of resting validity transactions no build yielded. + pub fn record_resting_unseen_sweep(released: usize, duration: Duration) { + if released > 0 { + Self::resting_predicate_carry_released_total("unseen").increment(released as u64); + } + Self::resting_predicate_sweep_duration().record(duration.as_secs_f64()); + } + + /// Records the time one native block build's transaction iterator spent advancing past held + /// resting validity transactions. + pub fn record_resting_hold_duration(duration: Duration) { + Self::resting_predicate_hold_duration().record(duration.as_secs_f64()); + } + + /// Records how long a block build waited for the previous block's resting sweep. + pub fn record_resting_sweep_wait(waited: Duration) { + Self::resting_predicate_sweep_wait_duration().record(waited.as_secs_f64()); + } + + /// Records a block build that gave up waiting for the previous block's resting sweep. + pub fn record_resting_sweep_wait_timeout() { + Self::resting_predicate_sweep_wait_timeouts_total().increment(1); + } + /// Records validity-predicate index bucket wakeups and depth distribution for one build. pub fn record_predicate_index_diagnostics(wakeups: u64, index: &ParkedPredicateIndex) { Self::predicate_bucket_wakeups().record(wakeups as f64); diff --git a/crates/execution/payload/src/resource_metering.rs b/crates/execution/payload/src/resource_metering.rs index 3edd174260..77f5ac8bd5 100644 --- a/crates/execution/payload/src/resource_metering.rs +++ b/crates/execution/payload/src/resource_metering.rs @@ -70,8 +70,8 @@ pub struct ResourceMeteringDimension { pub operations: Vec, /// Observe over-budget usage without excluding the transaction. /// - /// Limits take effect unless this flag is set. Dry-run is the explicit - /// opt-out used while rolling out a dimension. + /// Required in JSON (`dryRun`): set `true` to observe budgets without + /// excluding transactions, or `false` to enforce. pub dry_run: bool, } @@ -94,7 +94,6 @@ struct ResourceMeteringDimensionFile { base_gas_weight: u64, #[serde(default)] operations: Vec, - #[serde(default)] dry_run: bool, } @@ -1013,7 +1012,8 @@ mod tests { "name": "SSTORE", "gasUsedWeight": 3, "countCost": 4 - }] + }], + "dryRun": false }] }"#, ) @@ -1042,6 +1042,51 @@ mod tests { assert!(schedule.dimensions[0].dry_run); } + #[test] + fn dimension_requires_explicit_dry_run() { + let missing_dry_run = ResourceMeteringSchedule::from_json( + r#"{ + "version": 1, + "dimensions": [{ + "name": "execution", + "blockLimit": 100, + "baseGasWeight": 1 + }] + }"#, + ); + assert!(matches!(missing_dry_run, Err(ResourceMeteringError::ParseJson(_)))); + let schedule = ResourceMeteringSchedule::from_json( + r#"{ + "version": 1, + "dimensions": [ + { + "name": "compute", + "blockLimit": 1000, + "transactionLimit": 500, + "baseGasWeight": 1, + "operations": [ + {"name": "SSTORE", "gasUsedWeight": 2, "countCost": 10} + ], + "dryRun": true + }, + { + "name": "state_updates", + "blockLimit": 800, + "transactionLimit": 400, + "operations": [ + {"name": "STATE_CHANGED_STORAGE_SLOT", "countCost": 25} + ], + "dryRun": false + } + ] + }"#, + ) + .expect("schedule with explicit dryRun on every dimension must parse"); + assert_eq!(schedule.dimensions.len(), 2); + assert!(schedule.dimensions[0].dry_run); + assert!(!schedule.dimensions[1].dry_run); + } + #[test] fn rejects_unknown_json_fields() { let error = ResourceMeteringSchedule::from_json( @@ -1063,7 +1108,8 @@ mod tests { "name": "cpu", "blockLimit": 100, "transactionLimit": 0, - "baseGasWeight": 1 + "baseGasWeight": 1, + "dryRun": false }] }"#, ) @@ -1396,7 +1442,8 @@ mod tests { "dimensions": [{ "name": "cpu", "blockLimit": 100, - "baseGasWeight": 1 + "baseGasWeight": 1, + "dryRun": false }] }"#, ) diff --git a/crates/execution/payload/src/resting.rs b/crates/execution/payload/src/resting.rs new file mode 100644 index 0000000000..5a2bead8f8 --- /dev/null +++ b/crates/execution/payload/src/resting.rs @@ -0,0 +1,972 @@ +//! Validity transactions resting under an unsatisfied predicate, carried across block builds. + +use std::{ + collections::BTreeMap, + sync::{Arc, mpsc}, + time::{Duration, Instant}, +}; + +use alloy_primitives::{B256, TxHash, map::B256Set}; +use base_execution_txpool::{ + BasePooledTx, BestTransactionsHold, PredicateContext, ValidityConditions, ValidityPredicate, +}; +use parking_lot::Mutex; +use reth_tasks::Runtime; +use reth_transaction_pool::ValidPoolTransaction; +use revm::{database::TransitionState, state::EvmState}; +use tracing::debug_span; + +use crate::{ParkedPredicateIndex, ValidityMetrics, ValidityPredicateKey}; + +/// Blocks a member stays in the set after the last build that yielded it. +/// +/// Every state change wakes members whether or not the pool still yields them, so keeping a member +/// longer only costs memory. A member that left the pool is dropped when the block this many +/// blocks after the last build that yielded it finishes, two seconds at the 200ms block time Denim +/// activates. The sweep that drops it only visits the members filed that many blocks ago, and runs +/// off the build path; see [`RestingPredicateStore::finish`]. +pub const RESTING_UNSEEN_RETAINED_BLOCKS: u64 = 10; + +/// Mode for holding back validity transactions that rest under an unchanged predicate. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +#[cfg_attr(feature = "clap", derive(clap::ValueEnum))] +pub enum RestingPredicateMode { + /// Resting transactions are not tracked. + #[default] + Off, + /// Resting transactions are tracked and still evaluated. A resting transaction whose + /// predicates match is counted as a mismatch, which would be a missed inclusion under + /// enforcement. + Shadow, + /// Resting transactions are parked without being evaluated until a state change that can + /// satisfy their predicate wakes them. + Enforce, +} + +impl RestingPredicateMode { + /// Returns whether resting transactions are tracked. + pub const fn is_enabled(&self) -> bool { + matches!(self, Self::Shadow | Self::Enforce) + } + + /// Returns whether resting transactions are held back. + pub const fn is_enforced(&self) -> bool { + matches!(self, Self::Enforce) + } +} + +/// Blocks in which a resting transaction was last evaluated and last yielded. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct RestingMember { + /// Block whose build last found the recorded predicate unsatisfied. + pub rested_block: u64, + /// Block whose build last yielded the transaction. + pub seen_block: u64, + /// Block whose generation lists the member, so the sweep visits it once that block is + /// [`RESTING_UNSEEN_RETAINED_BLOCKS`] old. + filed_block: u64, +} + +/// What [`RestingPredicates::begin_block`] did to the set. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct RestingCarryOver { + /// The set was emptied because the new block does not build on the last block it followed. + pub reset: bool, + /// Members released because a pre-block write changed the state their predicate reads. + pub released_by_state: usize, + /// Members released because their predicate reads the block number. + pub released_by_block_number: usize, + /// Members carried into the new block. + pub carried: usize, +} + +/// Validity transactions whose recorded predicate is unsatisfied at the current build position. +/// +/// A member's predicate stays false until state it reads changes. Builders report every state +/// change: commits in the selection loop through [`Self::woken_by_state`], and the parent state +/// plus pre-block writes through [`Self::begin_block`]. Skipping a member is therefore the same as +/// evaluating it and parking it again. The set may over-wake but must never miss a wakeup. +/// +/// Members are only valid against the state they were checked on. The set records the block whose +/// final state it last followed, and empties itself when the next block builds on anything else, +/// such as a block built by another sequencer or a different payload version for the same height. +#[derive(Debug)] +pub struct RestingPredicates { + index: ParkedPredicateIndex, + ordered_threshold: usize, + block_number: u64, + /// Hash of the block whose final state the members were last checked against. `None` while a + /// block is being built, so a build that does not finish cannot hand its members on. + tip: Option, + /// Members by the block they are filed under. A member is filed under the block that rested + /// it, and re-filed under its latest seen block when the sweep finds it was seen since, so the + /// sweep only visits members old enough to drop. Entries for members that were removed or + /// re-filed since are stale and skipped. + generations: BTreeMap>, +} + +impl RestingPredicates { + /// Creates an empty set whose predicate buckets convert at `ordered_threshold` entries. + pub fn new(ordered_threshold: usize) -> Self { + Self { + index: ParkedPredicateIndex::new(ordered_threshold), + ordered_threshold, + block_number: 0, + tip: None, + generations: BTreeMap::new(), + } + } + + /// Starts building `block_number` on `parent_hash`. + /// + /// `pre_block_changes` must hold every write made on top of the parent state before the + /// selection loop starts, such as system calls and deposits. Members are kept only when the + /// parent is the block this set last finished and those writes are known; otherwise the set + /// is emptied. Members blocked by a block-number predicate are always released so the new + /// block re-evaluates them. + pub fn begin_block( + &mut self, + parent_hash: B256, + block_number: u64, + pre_block_changes: Option<&TransitionState>, + ) -> RestingCarryOver { + let mut carry_over = RestingCarryOver::default(); + let follows_parent = self.tip.take() == Some(parent_hash); + match pre_block_changes.filter(|_| follows_parent) { + None => { + carry_over.reset = !self.index.is_empty(); + self.index = ParkedPredicateIndex::new(self.ordered_threshold); + self.generations.clear(); + } + Some(changes) => { + for hash in self.index.affected_by_transitions(changes).affected_transactions { + carry_over.released_by_state += usize::from(self.index.remove(hash).is_some()); + } + for hash in self.index.transactions_blocked_by(ValidityPredicateKey::BlockNumber) { + carry_over.released_by_block_number += + usize::from(self.index.remove(hash).is_some()); + } + } + } + self.block_number = block_number; + carry_over.carried = self.index.len(); + carry_over + } + + /// Records that the block started by [`Self::begin_block`] was finished as `block_hash`, with + /// every commit reported to the set. + pub const fn end_block(&mut self, block_hash: B256) { + self.tip = Some(block_hash); + } + + /// Returns whether finishing the block being built is due an unseen sweep, because some + /// members are filed under a block at least [`RESTING_UNSEEN_RETAINED_BLOCKS`] old. + pub fn sweep_due(&self) -> bool { + self.due_generation().is_some_and(|due| { + self.generations.first_key_value().is_some_and(|(filed, _)| *filed <= due) + }) + } + + /// Latest block whose generation is old enough to sweep. + const fn due_generation(&self) -> Option { + self.block_number.checked_sub(RESTING_UNSEEN_RETAINED_BLOCKS) + } + + /// Drops members no build yielded in the last [`RESTING_UNSEEN_RETAINED_BLOCKS`] blocks and + /// returns how many were dropped. + /// + /// Only visits members filed under a block that is old enough: those not seen since are + /// dropped and the rest are re-filed under the block that last saw them. Run every block, it + /// costs about the members rested or re-filed in one block rather than the whole set. + /// Builders run it through [`RestingPredicateStore::finish`], off the path between the + /// forkchoice update and flashblock scheduling. + pub fn sweep_unseen(&mut self) -> usize { + let Some(due) = self.due_generation() else { return 0 }; + let span = debug_span!( + target: "payload_builder", + "resting_unseen_sweep", + block_number = self.block_number, + visited = tracing::field::Empty, + released = tracing::field::Empty, + ); + let _entered = span.enter(); + let start = Instant::now(); + let (mut visited, mut released) = (0, 0); + while let Some(generation) = self.generations.first_entry() { + if *generation.key() > due { + break; + } + let (filed, hashes) = generation.remove_entry(); + visited += hashes.len(); + for hash in hashes { + let Some(member) = self.index.transaction_mut(hash) else { continue }; + if member.filed_block != filed { + continue; + } + if member.seen_block > due { + member.filed_block = member.seen_block; + self.generations.entry(member.seen_block).or_default().push(hash); + } else { + self.index.remove(hash); + released += 1; + } + } + } + ValidityMetrics::record_resting_unseen_sweep(released, start.elapsed()); + span.record("visited", visited); + span.record("released", released); + released + } + + /// Records that `predicate` was unsatisfied for `transaction_hash` at the current build + /// position. + /// + /// Flashblock-index predicates are not recorded because the index changes without any state + /// change. + pub fn rest(&mut self, transaction_hash: TxHash, predicate: ValidityPredicate) { + if matches!(predicate, ValidityPredicate::FlashblockIndex { .. }) { + return; + } + // A member resting again in the block that filed it is already listed in that block's + // generation; listing it twice would only give the sweep a stale entry to skip. + let filed = self + .index + .entry(transaction_hash) + .is_some_and(|(member, _)| member.filed_block == self.block_number); + let member = RestingMember { + rested_block: self.block_number, + seen_block: self.block_number, + filed_block: self.block_number, + }; + self.index.park(transaction_hash, member, predicate); + if !filed { + self.generations.entry(self.block_number).or_default().push(transaction_hash); + } + } + + /// Removes a member, returning whether it was present. + pub fn remove(&mut self, transaction_hash: TxHash) -> bool { + !self.index.is_empty() && self.index.remove(transaction_hash).is_some() + } + + /// Returns the member for a transaction with `conditions`. + /// + /// A hash re-added to the pool with a batch that no longer contains the recorded predicate is + /// not a member, because sidecar predicates are not covered by the hash. Neither is a + /// transaction whose batch expired, so the builder evaluates and drops it. + pub fn member( + &self, + transaction_hash: TxHash, + conditions: &ValidityConditions, + ) -> Option { + if self.index.is_empty() { + return None; + } + let (member, predicate) = self.index.entry(transaction_hash)?; + Self::applies(*member, predicate, conditions, self.block_number).then_some(*member) + } + + /// Holds back a member of the set: records that the current build yielded it and returns + /// whether it was carried over from an earlier block. Uses one lookup, because builders call + /// it for every validity candidate. + pub fn hold( + &mut self, + transaction_hash: TxHash, + conditions: &ValidityConditions, + ) -> Option { + if self.index.is_empty() { + return None; + } + let block_number = self.block_number; + let (member, predicate) = self.index.entry_mut(transaction_hash)?; + if !Self::applies(*member, predicate, conditions, block_number) { + return None; + } + let carried = member.rested_block < block_number; + // Members that rested in this block were already seen in it. + if carried { + member.seen_block = block_number; + } + Some(carried) + } + + /// Whether `member`, resting under `predicate`, applies to a transaction with `conditions`. + /// + /// Only carried members are checked for expiry: one that rested in the block being built was + /// evaluated, and found unexpired, at the same block number. + fn applies( + member: RestingMember, + predicate: &ValidityPredicate, + conditions: &ValidityConditions, + block_number: u64, + ) -> bool { + conditions.contains(predicate) + && (member.rested_block == block_number + || !conditions.is_expired(&PredicateContext { block_number, flashblock_index: 0 })) + } + + /// Returns members whose predicate may have changed in `state`, one committed transaction's + /// changes. Callers remove them with [`Self::remove`] once the commit is applied. + pub fn woken_by_state(&self, state: &EvmState) -> Vec { + if self.index.is_empty() { + return Vec::new(); + } + self.index.affected_by_state(state).affected_transactions + } + + /// Returns whether the set has no members. + pub fn is_empty(&self) -> bool { + self.index.is_empty() + } + + /// Returns the number of members. + pub fn len(&self) -> usize { + self.index.len() + } +} + +/// A finished block's [`RestingPredicates`], possibly still being swept in the background. +#[derive(Debug)] +pub struct FinishingRestingPredicates(Finishing); + +#[derive(Debug)] +enum Finishing { + Ready(RestingPredicates), + Sweeping(mpsc::Receiver), +} + +impl FinishingRestingPredicates { + /// Returns the set, waiting up to `limit` for an unfinished sweep. + /// + /// Returns `None` when the sweep does not finish in time or its task was dropped without + /// returning the set; the caller then starts with an empty set. Time spent waiting is recorded, + /// because it delays the build that takes the set. + pub fn wait(self, limit: Duration) -> Option { + match self.0 { + Finishing::Ready(resting) => Some(resting), + Finishing::Sweeping(receiver) => match receiver.try_recv() { + Ok(resting) => { + ValidityMetrics::record_resting_sweep_wait(Duration::ZERO); + Some(resting) + } + Err(mpsc::TryRecvError::Empty) => { + let start = Instant::now(); + let resting = receiver.recv_timeout(limit); + ValidityMetrics::record_resting_sweep_wait(start.elapsed()); + match resting { + Ok(resting) => Some(resting), + // The sweep finishes on its worker and drops the set there. + Err(mpsc::RecvTimeoutError::Timeout) => { + ValidityMetrics::record_resting_sweep_wait_timeout(); + None + } + Err(mpsc::RecvTimeoutError::Disconnected) => None, + } + } + Err(mpsc::TryRecvError::Disconnected) => None, + }, + } + } +} + +/// Work spent holding back resting transactions since the stats were last taken. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct RestingStats { + /// Resting transactions parked without being yielded to the build loop. + pub parked: u64, + /// Of [`Self::parked`], transactions whose predicate was found unsatisfied in an earlier block. + pub carried_parked: u64, + /// Time the pool iterators spent advancing past held transactions, from the first hold of + /// each `next` call until it found a candidate. + pub duration: Duration, +} + +/// Reports whether the build driving a pool iterator has stopped selecting transactions. +pub type HoldInterrupt = Arc bool + Send + Sync>; + +/// A resting set shared between one build and the pool iterators it drives. +/// +/// The build loop populates the set through [`Self::rest`] and reports commits through +/// [`Self::wake_by_state`]. Each pool iterator the build creates carries a [`Self::iterator_hold`] +/// that parks members at the source, so the build loop never sees a transaction whose recorded +/// predicate is still unsatisfied. The iterator and the build loop run on the build thread, so +/// the lock is uncontended; it exists because the iterator is a boxed trait object the build loop +/// does not own. +#[derive(Clone)] +pub struct SharedRestingPredicates { + inner: Arc>, + /// Interrupts the walks of the iterators this handle's holds park members in. + interrupt: Option, +} + +impl std::fmt::Debug for SharedRestingPredicates { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("SharedRestingPredicates") + .field("inner", &self.inner) + .field("interrupt", &self.interrupt.is_some()) + .finish() + } +} + +#[derive(Debug)] +struct SharedRestingState { + predicates: RestingPredicates, + /// Members parked by the current iterator's hold. The build loop promotes these when a commit + /// wakes them; members the build loop parked itself are woken by its own predicate index. + held: B256Set, + stats: RestingStats, +} + +impl SharedRestingPredicates { + /// Shares `predicates` with the pool iterators of one build. + pub fn new(predicates: RestingPredicates) -> Self { + Self { + inner: Arc::new(Mutex::new(SharedRestingState { + predicates, + held: B256Set::default(), + stats: RestingStats::default(), + })), + interrupt: None, + } + } + + /// Interrupts the holds this handle creates once `interrupt` returns true, so a pool iterator + /// stops advancing past held members when its build is cancelled. Clones share the resting + /// set, so a builder can give each iterator a handle that observes that iterator's build. + #[must_use] + pub fn with_interrupt(mut self, interrupt: HoldInterrupt) -> Self { + self.interrupt = Some(interrupt); + self + } + + /// Returns a hold for a new pool iterator. Members the previous iterator held are forgotten, + /// because they were parked in that iterator. + pub fn iterator_hold(&self) -> Box> { + self.inner.lock().held.clear(); + Box::new(RestingHold { shared: self.clone() }) + } + + /// Records that `predicate` was unsatisfied for `transaction_hash` at the current build + /// position. + pub fn rest(&self, transaction_hash: TxHash, predicate: ValidityPredicate) { + self.inner.lock().predicates.rest(transaction_hash, predicate); + } + + /// Removes a member, returning whether it was present. + pub fn remove(&self, transaction_hash: TxHash) -> bool { + self.inner.lock().predicates.remove(transaction_hash) + } + + /// Returns the member for a transaction with `conditions`. + pub fn member( + &self, + transaction_hash: TxHash, + conditions: &ValidityConditions, + ) -> Option { + self.inner.lock().predicates.member(transaction_hash, conditions) + } + + /// Removes the members whose predicate may have changed in `state`, one committed + /// transaction's changes, and returns those the current iterator held so the caller can + /// promote them. + pub fn wake_by_state(&self, state: &EvmState) -> Vec { + let mut inner = self.inner.lock(); + if inner.predicates.is_empty() { + return Vec::new(); + } + let mut held = Vec::new(); + for transaction_hash in inner.predicates.woken_by_state(state) { + inner.predicates.remove(transaction_hash); + if inner.held.remove(&transaction_hash) { + held.push(transaction_hash); + } + } + held + } + + /// Returns and resets the work spent holding back resting transactions. + pub fn take_stats(&self) -> RestingStats { + std::mem::take(&mut self.inner.lock().stats) + } + + /// Takes the resting set, leaving an empty one, so it can be carried into the next block. + pub fn take_predicates(&self) -> RestingPredicates { + let mut inner = self.inner.lock(); + let empty = RestingPredicates::new(inner.predicates.ordered_threshold); + std::mem::replace(&mut inner.predicates, empty) + } + + /// Returns the number of members. + pub fn len(&self) -> usize { + self.inner.lock().predicates.len() + } + + /// Returns whether the set has no members. + pub fn is_empty(&self) -> bool { + self.inner.lock().predicates.is_empty() + } +} + +/// Parks members of a [`SharedRestingPredicates`] inside a pool iterator. +#[derive(Debug)] +struct RestingHold { + shared: SharedRestingPredicates, +} + +impl BestTransactionsHold for RestingHold { + fn hold(&mut self, transaction: &ValidPoolTransaction) -> bool { + let mut inner = self.shared.inner.lock(); + let Some(carried) = inner + .predicates + .hold(*transaction.hash(), transaction.transaction.validity_conditions()) + else { + return false; + }; + inner.held.insert(*transaction.hash()); + inner.stats.parked += 1; + inner.stats.carried_parked += u64::from(carried); + true + } + + fn record_duration(&mut self, duration: Duration) { + self.shared.inner.lock().stats.duration += duration; + } + + fn is_interrupted(&self) -> bool { + self.shared.interrupt.as_ref().is_some_and(|interrupt| interrupt()) + } +} + +/// Holds one builder's [`RestingPredicates`] between payload jobs. +/// +/// A job takes the set when it starts and puts it back once it finishes a block. A job that fails +/// or is abandoned never puts it back, so the next job starts empty. Concurrent jobs each see at +/// most one of them holding the set. +#[derive(Debug, Clone, Default)] +pub struct RestingPredicateStore { + members: Arc>>, + /// Runs end-of-block sweeps in the background. Without one, sweeps run inline in + /// [`Self::finish`]. + executor: Option, +} + +impl RestingPredicateStore { + /// Name of the task-executor worker that runs end-of-block sweeps. + const SWEEP_WORKER: &'static str = "resting-sweep"; + + /// Creates an empty store whose end-of-block sweeps run on `executor`. + pub fn new(executor: Runtime) -> Self { + Self { members: Arc::default(), executor: Some(executor) } + } + + /// Takes the stored set, or creates an empty one with `ordered_threshold`. + /// + /// Waits up to `sweep_wait_limit` for a sweep the previous job started. A sweep still running + /// after that is abandoned and the set returned empty: the build evaluates every validity + /// transaction again, as after a reset, and the sweep drops the old set on its worker. + pub fn take(&self, ordered_threshold: usize, sweep_wait_limit: Duration) -> RestingPredicates { + let stored = self.members.lock().take(); + stored + .and_then(|(finishing, block_hash)| { + let mut resting = finishing.wait(sweep_wait_limit)?; + resting.end_block(block_hash); + Some(resting) + }) + .unwrap_or_else(|| RestingPredicates::new(ordered_threshold)) + } + + /// Finishes the selection loop of the block being built. + /// + /// When an unseen sweep is due it runs on the task executor, so it overlaps the state root and + /// the leeway before the next build instead of delaying that build's flashblocks. Hand the + /// result to [`Self::put`] once the block is sealed. + pub fn finish(&self, mut resting: RestingPredicates) -> FinishingRestingPredicates { + if resting.is_empty() { + // Every listed member was removed, so every generation entry is stale. + resting.generations.clear(); + return FinishingRestingPredicates(Finishing::Ready(resting)); + } + if !resting.sweep_due() { + return FinishingRestingPredicates(Finishing::Ready(resting)); + } + let Some(executor) = &self.executor else { + resting.sweep_unseen(); + return FinishingRestingPredicates(Finishing::Ready(resting)); + }; + let (sender, receiver) = mpsc::sync_channel(1); + // A persistent named worker, like `Runtime::spawn_drop`, avoids creating a thread per + // sweep. The set comes back over a std channel because `take` may wait for it from inside + // a payload job's `block_on`, where blocking on a tokio channel panics. + let _ = executor.spawn_blocking_named(Self::SWEEP_WORKER, move || { + resting.sweep_unseen(); + // The receiver is gone if the block failed to seal; the set is dropped with it. + let _ = sender.send(resting); + }); + FinishingRestingPredicates(Finishing::Sweeping(receiver)) + } + + /// Stores the set of the block finished as `block_hash` for the next job. + pub fn put(&self, resting: FinishingRestingPredicates, block_hash: B256) { + *self.members.lock() = Some((resting, block_hash)); + } +} + +#[cfg(test)] +mod tests { + use alloy_primitives::{Address, U256}; + use base_execution_txpool::ValidityOperator; + use revm::{ + database::{AccountStatus, TransitionAccount, TransitionState, states::StorageSlot}, + state::{Account, AccountInfo}, + }; + + use super::*; + + const WATCHED: Address = Address::repeat_byte(0xaa); + const UNRELATED: Address = Address::repeat_byte(0xbb); + const PARENT: B256 = B256::repeat_byte(0x01); + /// Long enough for any test sweep to finish. + const WAIT: Duration = Duration::from_secs(10); + const BUILT: B256 = B256::repeat_byte(0x02); + + fn balance_at_least(address: Address, value: u64) -> ValidityPredicate { + ValidityPredicate::Balance { + address, + op: ValidityOperator::GreaterThanOrEqual, + value: U256::from(value), + } + } + + fn storage_equals(address: Address, slot: u64, value: u64) -> ValidityPredicate { + ValidityPredicate::Storage { + address, + slot: U256::from(slot), + mask: U256::MAX, + op: ValidityOperator::Equal, + value: U256::from(value), + } + } + + fn conditions(predicates: &[ValidityPredicate]) -> ValidityConditions { + ValidityConditions::new(predicates.to_vec()) + } + + fn hash(byte: u8) -> TxHash { + TxHash::repeat_byte(byte) + } + + fn balance_transition(address: Address, old: u64, new: u64) -> TransitionState { + let account = + |balance: u64| AccountInfo { balance: U256::from(balance), ..Default::default() }; + TransitionState::single( + address, + TransitionAccount { + info: Some(account(new)), + status: AccountStatus::Changed, + previous_info: Some(account(old)), + previous_status: AccountStatus::Loaded, + storage: Default::default(), + storage_was_destroyed: false, + }, + ) + } + + fn storage_transition(address: Address, slot: u64, old: u64, new: u64) -> TransitionState { + let mut storage = alloy_primitives::map::HashMap::default(); + storage + .insert(U256::from(slot), StorageSlot::new_changed(U256::from(old), U256::from(new))); + TransitionState::single( + address, + TransitionAccount { + info: Some(AccountInfo::default()), + status: AccountStatus::Changed, + previous_info: Some(AccountInfo::default()), + previous_status: AccountStatus::Loaded, + storage, + storage_was_destroyed: false, + }, + ) + } + + /// A set that rested `predicates` (one transaction each) in block 1 and finished it as + /// [`PARENT`]. + fn rested_in_parent(predicates: &[ValidityPredicate]) -> RestingPredicates { + let mut resting = RestingPredicates::new(32); + resting.begin_block(B256::ZERO, 1, Some(&TransitionState::default())); + for (index, predicate) in predicates.iter().enumerate() { + resting.rest(hash(index as u8), predicate.clone()); + } + resting.end_block(PARENT); + resting + } + + #[test] + fn members_carry_into_a_block_built_on_the_finished_block() { + let predicate = balance_at_least(WATCHED, 1); + let mut resting = rested_in_parent(std::slice::from_ref(&predicate)); + + let carry_over = resting.begin_block(PARENT, 2, Some(&TransitionState::default())); + + assert_eq!(carry_over, RestingCarryOver { carried: 1, ..Default::default() }); + assert_eq!(resting.hold(hash(0), &conditions(&[predicate])), Some(true)); + } + + #[test] + fn set_is_emptied_when_the_parent_is_not_the_finished_block() { + let predicate = balance_at_least(WATCHED, 1); + let mut resting = rested_in_parent(std::slice::from_ref(&predicate)); + + let carry_over = resting.begin_block(BUILT, 2, Some(&TransitionState::default())); + + assert!(carry_over.reset); + assert_eq!(carry_over.carried, 0); + assert!(resting.member(hash(0), &conditions(&[predicate])).is_none()); + } + + #[test] + fn set_is_emptied_when_pre_block_writes_are_unknown() { + let mut resting = rested_in_parent(&[balance_at_least(WATCHED, 1)]); + + assert!(resting.begin_block(PARENT, 2, None).reset); + assert!(resting.is_empty()); + } + + #[test] + fn unfinished_block_does_not_hand_on_its_members() { + let mut resting = rested_in_parent(&[balance_at_least(WATCHED, 1)]); + resting.begin_block(PARENT, 2, Some(&TransitionState::default())); + + // Block 2 is never finished, so block 3 cannot trust the set even on a matching hash. + assert!(resting.begin_block(PARENT, 3, Some(&TransitionState::default())).reset); + } + + #[test] + fn pre_block_writes_release_members_reading_the_written_state() { + let watched = balance_at_least(WATCHED, 1); + let unrelated = storage_equals(UNRELATED, 7, 1); + let mut resting = rested_in_parent(&[watched.clone(), unrelated.clone()]); + let mut changes = balance_transition(WATCHED, 0, 5); + changes.transitions.extend(storage_transition(UNRELATED, 8, 0, 1).transitions); + + let carry_over = resting.begin_block(PARENT, 2, Some(&changes)); + + assert_eq!(carry_over.released_by_state, 1); + assert!(resting.member(hash(0), &conditions(&[watched])).is_none()); + assert!(resting.member(hash(1), &conditions(&[unrelated])).is_some()); + } + + #[test] + fn pre_block_storage_write_releases_members_on_that_slot() { + let predicate = storage_equals(WATCHED, 7, 1); + let mut resting = rested_in_parent(std::slice::from_ref(&predicate)); + + let carry_over = + resting.begin_block(PARENT, 2, Some(&storage_transition(WATCHED, 7, 0, 1))); + + assert_eq!(carry_over.released_by_state, 1); + assert!(resting.is_empty()); + } + + #[test] + fn block_number_blockers_are_released_every_block() { + let not_yet = ValidityPredicate::BlockNumber { + op: ValidityOperator::GreaterThanOrEqual, + value: U256::from(5), + }; + let mut resting = rested_in_parent(&[not_yet]); + + let carry_over = resting.begin_block(PARENT, 2, Some(&TransitionState::default())); + + assert_eq!(carry_over.released_by_block_number, 1); + assert!(resting.is_empty()); + } + + /// Builds blocks `2..=last_block` on top of [`rested_in_parent`], holding `held` in blocks up + /// to `held_through`, and finishes each one through `store`. Returns the set after + /// `last_block`. + fn carry_through( + store: &RestingPredicateStore, + resting: RestingPredicates, + held: &ValidityConditions, + held_through: u64, + last_block: u64, + ) -> RestingPredicates { + store.put(store.finish(resting), PARENT); + let mut parent = PARENT; + for block in 2..=last_block { + let mut resting = store.take(32, WAIT); + let carry_over = resting.begin_block(parent, block, Some(&TransitionState::default())); + assert!(!carry_over.reset, "block {block} must follow the finished block"); + if block <= held_through { + assert_eq!(resting.hold(hash(0), held), Some(true)); + } + parent = B256::with_last_byte(block as u8); + store.put(store.finish(resting), parent); + } + store.take(32, WAIT) + } + + #[test] + fn members_not_yielded_for_the_retention_window_are_dropped() { + let seen = balance_at_least(WATCHED, 1); + let unseen = balance_at_least(UNRELATED, 1); + let held = conditions(std::slice::from_ref(&seen)); + let store = RestingPredicateStore::default(); + let window = RESTING_UNSEEN_RETAINED_BLOCKS; + // A member last seen in block 1 is dropped when block 1 + window finishes. + let rested = || rested_in_parent(&[seen.clone(), unseen.clone()]); + assert_eq!(carry_through(&store, rested(), &held, u64::MAX, window).len(), 2); + let resting = carry_through(&store, rested(), &held, u64::MAX, 1 + window); + assert_eq!(resting.len(), 1); + assert!(resting.member(hash(0), &held).is_some()); + } + + #[test] + fn member_seen_after_it_was_filed_is_dropped_a_window_after_it_was_last_seen() { + let predicate = balance_at_least(WATCHED, 1); + let held = conditions(std::slice::from_ref(&predicate)); + let store = RestingPredicateStore::default(); + let window = RESTING_UNSEEN_RETAINED_BLOCKS; + let last_seen = 5; + let rested = || rested_in_parent(std::slice::from_ref(&predicate)); + // The sweep at block 1 + window re-files the member under block 5 instead of dropping it. + assert_eq!( + carry_through(&store, rested(), &held, last_seen, last_seen + window - 1).len(), + 1 + ); + assert!(carry_through(&store, rested(), &held, last_seen, last_seen + window).is_empty()); + } + + #[test] + fn resting_again_in_the_same_block_lists_the_member_once() { + let predicate = balance_at_least(WATCHED, 1); + let mut resting = RestingPredicates::new(32); + resting.begin_block(PARENT, 2, Some(&TransitionState::default())); + resting.rest(hash(0), predicate.clone()); + resting.rest(hash(0), predicate); + assert_eq!(resting.len(), 1); + assert_eq!(resting.generations.get(&2).map(Vec::len), Some(1)); + } + + #[test] + fn block_start_never_sweeps_unseen_members() { + let mut resting = rested_in_parent(&[balance_at_least(WATCHED, 1)]); + let block = 2 * RESTING_UNSEEN_RETAINED_BLOCKS; + // The member was last seen in block 1, long enough ago to be swept, but block start is on + // the path to flashblock scheduling and must leave the sweep to the end of a block. + let carry_over = resting.begin_block(PARENT, block, Some(&TransitionState::default())); + assert_eq!(carry_over, RestingCarryOver { carried: 1, ..Default::default() }); + assert!(resting.sweep_due()); + assert_eq!(resting.sweep_unseen(), 1); + } + + #[test] + fn sweep_runs_on_the_task_executor_and_hands_the_set_on() { + let runtime = Runtime::test(); + let store = RestingPredicateStore::new(runtime); + let mut resting = rested_in_parent(&[balance_at_least(WATCHED, 1)]); + let block = 2 * RESTING_UNSEEN_RETAINED_BLOCKS; + resting.begin_block(PARENT, block, Some(&TransitionState::default())); + resting.rest(hash(1), balance_at_least(UNRELATED, 1)); + + let finishing = store.finish(resting); + assert!(matches!(finishing.0, Finishing::Sweeping(_))); + store.put(finishing, BUILT); + + let mut resting = store.take(32, WAIT); + assert_eq!(resting.len(), 1, "the member unseen since block 1 is swept"); + let carry_over = resting.begin_block(BUILT, block + 1, Some(&TransitionState::default())); + assert_eq!(carry_over, RestingCarryOver { carried: 1, ..Default::default() }); + } + + #[test] + fn finishing_a_block_without_a_due_sweep_does_not_spawn() { + let store = RestingPredicateStore::new(Runtime::test()); + let mut resting = rested_in_parent(&[balance_at_least(WATCHED, 1)]); + resting.begin_block(PARENT, 2, Some(&TransitionState::default())); + assert!(matches!(store.finish(resting).0, Finishing::Ready(_))); + } + + #[test] + fn finishing_an_emptied_set_drops_its_stale_generations() { + let store = RestingPredicateStore::new(Runtime::test()); + let mut resting = RestingPredicates::new(32); + resting.begin_block(PARENT, 2, Some(&TransitionState::default())); + resting.rest(hash(0), balance_at_least(WATCHED, 1)); + assert!(resting.remove(hash(0))); + + let Finishing::Ready(resting) = store.finish(resting).0 else { + panic!("an empty set must not spawn a sweep"); + }; + assert!(resting.generations.is_empty()); + } + + #[test] + fn build_stops_waiting_for_a_slow_sweep_and_starts_empty() { + let limit = Duration::from_millis(25); + let store = RestingPredicateStore::new(Runtime::test()); + let (sender, receiver) = mpsc::sync_channel(1); + store.put(FinishingRestingPredicates(Finishing::Sweeping(receiver)), PARENT); + + let start = Instant::now(); + let mut resting = store.take(32, limit); + assert!(start.elapsed() >= limit); + assert!(resting.is_empty()); + let carry_over = resting.begin_block(PARENT, 2, Some(&TransitionState::default())); + assert_eq!(carry_over, RestingCarryOver::default()); + drop(sender); + } + + #[test] + fn readded_hash_without_the_recorded_predicate_is_not_a_member() { + let mut resting = rested_in_parent(&[balance_at_least(WATCHED, 1)]); + resting.begin_block(PARENT, 2, Some(&TransitionState::default())); + + assert!(resting.member(hash(0), &conditions(&[balance_at_least(UNRELATED, 1)])).is_none()); + } + + #[test] + fn expired_batch_is_not_a_member() { + let blocker = balance_at_least(WATCHED, 1); + let deadline = ValidityPredicate::BlockNumber { + op: ValidityOperator::LessThanOrEqual, + value: U256::ONE, + }; + let mut resting = rested_in_parent(std::slice::from_ref(&blocker)); + resting.begin_block(PARENT, 2, Some(&TransitionState::default())); + + assert!(resting.member(hash(0), &conditions(&[deadline, blocker])).is_none()); + } + + #[test] + fn committed_state_wakes_members_reading_it() { + let predicate = balance_at_least(WATCHED, 1); + let resting = rested_in_parent(std::slice::from_ref(&predicate)); + let mut account = Account::default(); + account.info.balance = U256::from(3); + + assert_eq!(resting.woken_by_state(&EvmState::from_iter([(WATCHED, account)])), [hash(0)]); + } + + #[test] + fn flashblock_index_predicates_do_not_rest() { + let mut resting = RestingPredicates::new(32); + resting.rest( + hash(0), + ValidityPredicate::FlashblockIndex { op: ValidityOperator::Equal, value: U256::ONE }, + ); + assert!(resting.is_empty()); + } + + #[test] + fn store_hands_the_set_to_the_next_job() { + let store = RestingPredicateStore::default(); + let mut resting = store.take(32, WAIT); + resting.begin_block(B256::ZERO, 1, Some(&TransitionState::default())); + resting.rest(hash(0), balance_at_least(WATCHED, 1)); + store.put(store.finish(resting), PARENT); + + assert_eq!(store.take(32, WAIT).len(), 1); + assert!(store.take(32, WAIT).is_empty(), "a taken set leaves the store empty"); + } +} diff --git a/crates/execution/payload/src/validity.rs b/crates/execution/payload/src/validity.rs index b1e847db7c..cb92d86d44 100644 --- a/crates/execution/payload/src/validity.rs +++ b/crates/execution/payload/src/validity.rs @@ -10,7 +10,7 @@ use alloy_primitives::{ use base_execution_txpool::{ PredicateContext, ValidityConditions, ValidityOperator, ValidityPredicate, }; -use revm::{Database, state::EvmState}; +use revm::{Database, database::TransitionState, state::EvmState}; /// The number of parked predicates at which a flat bucket becomes ordered by default. pub const DEFAULT_PREDICATE_BUCKET_ORDERED_THRESHOLD: usize = 32; @@ -192,6 +192,23 @@ impl ParkedPredicateIndex { self.transactions.get(&transaction_hash).map(|entry| &entry.predicate) } + /// Returns an indexed transaction and the predicate it is parked under, with one lookup. + pub fn entry(&self, transaction_hash: TxHash) -> Option<(&T, &ValidityPredicate)> { + self.transactions.get(&transaction_hash).map(|entry| (&entry.transaction, &entry.predicate)) + } + + /// Returns a mutable indexed parked transaction. + pub fn transaction_mut(&mut self, transaction_hash: TxHash) -> Option<&mut T> { + self.transactions.get_mut(&transaction_hash).map(|entry| &mut entry.transaction) + } + + /// Returns a mutable indexed transaction and the predicate it is parked under, with one lookup. + pub fn entry_mut(&mut self, transaction_hash: TxHash) -> Option<(&mut T, &ValidityPredicate)> { + self.transactions + .get_mut(&transaction_hash) + .map(|entry| (&mut entry.transaction, &entry.predicate)) + } + /// Replaces a parked transaction's currently unsatisfied predicate. pub fn reindex(&mut self, transaction_hash: TxHash, predicate: ValidityPredicate) -> bool { let Some(previous) = self.transactions.get(&transaction_hash) else { @@ -219,48 +236,88 @@ impl ParkedPredicateIndex { pub fn affected_by_state(&self, state: &EvmState) -> StateChangeEffects { let mut effects = StateChangeEffects::default(); for (address, account) in state { + let destroyed = account.is_selfdestructed(); // Deleted accounts read as nonce zero, regardless of the retained account info. - let new_nonce = if account.is_selfdestructed() { 0 } else { account.info.nonce }; - if new_nonce != account.original_info().nonce { - self.wake_bucket( - &mut effects, - ValidityPredicateKey::Nonce(*address), - U256::from(account.original_info().nonce), - U256::from(new_nonce), - ); - } - if account.info.balance != account.original_info().balance { - self.wake_bucket( - &mut effects, - ValidityPredicateKey::Balance(*address), - account.original_info().balance, - account.info.balance, - ); - } - if account.is_selfdestructed() { - for (key, bucket) in &self.blockers { - if matches!(key, ValidityPredicateKey::Storage(key_address, _) if key_address == address) - { - bucket.extend_all(&mut effects.affected_transactions); - effects.woken_buckets += 1; - } - } - } else { - for (slot, value) in &account.storage { - if value.is_changed() { - self.wake_bucket( - &mut effects, - ValidityPredicateKey::Storage(*address, *slot), - value.original_value(), - value.present_value(), - ); - } - } - } + let new_nonce = if destroyed { 0 } else { account.info.nonce }; + self.wake_account( + &mut effects, + *address, + AccountChange { + old_nonce: account.original_info().nonce, + new_nonce, + old_balance: account.original_info().balance, + new_balance: account.info.balance, + storage_destroyed: destroyed, + }, + account + .storage + .iter() + .filter(|(_, value)| value.is_changed()) + .map(|(slot, value)| (*slot, value.original_value(), value.present_value())), + ); + } + effects + } + + /// Returns parked transactions and index-bucket wakeups triggered by the net changes a + /// [`TransitionState`] holds relative to the state it started from. + /// + /// Builders use it for writes made outside the selection loop, such as pre-execution system + /// calls and deposits, which never pass through [`Self::affected_by_state`]. + pub fn affected_by_transitions(&self, transitions: &TransitionState) -> StateChangeEffects { + let mut effects = StateChangeEffects::default(); + for (address, transition) in &transitions.transitions { + let old = transition.previous_info.as_ref(); + let new = transition.info.as_ref(); + self.wake_account( + &mut effects, + *address, + AccountChange { + old_nonce: old.map_or(0, |info| info.nonce), + new_nonce: new.map_or(0, |info| info.nonce), + old_balance: old.map_or(U256::ZERO, |info| info.balance), + new_balance: new.map_or(U256::ZERO, |info| info.balance), + storage_destroyed: transition.storage_was_destroyed + || transition.status.was_destroyed(), + }, + transition + .storage + .iter() + .filter(|(_, slot)| slot.is_changed()) + .map(|(key, slot)| (*key, slot.previous_or_original_value, slot.present_value)), + ); } effects } + /// Returns the number of indexed transactions. + pub fn len(&self) -> usize { + self.transactions.len() + } + + /// Returns the transactions parked under `blocker`. + pub fn transactions_blocked_by(&self, blocker: ValidityPredicateKey) -> Vec { + let mut hashes = Vec::new(); + if let Some(bucket) = self.blockers.get(&blocker) { + bucket.extend_all(&mut hashes); + } + hashes + } + + /// Removes every entry for which `keep` returns `false` and returns how many were removed. + pub fn retain(&mut self, mut keep: impl FnMut(&T) -> bool) -> usize { + let removed = self + .transactions + .iter() + .filter(|(_, entry)| !keep(&entry.transaction)) + .map(|(hash, _)| *hash) + .collect::>(); + for hash in &removed { + self.remove(*hash); + } + removed.len() + } + /// Returns the number of parked transactions blocked on each distinct index bucket. pub fn bucket_depths(&self) -> impl Iterator + '_ { self.blockers.values().map(PredicateBucket::len) @@ -300,6 +357,44 @@ impl ParkedPredicateIndex { } } + fn wake_account( + &self, + effects: &mut StateChangeEffects, + address: Address, + change: AccountChange, + changed_slots: impl Iterator, + ) { + if change.new_nonce != change.old_nonce { + self.wake_bucket( + effects, + ValidityPredicateKey::Nonce(address), + U256::from(change.old_nonce), + U256::from(change.new_nonce), + ); + } + if change.new_balance != change.old_balance { + self.wake_bucket( + effects, + ValidityPredicateKey::Balance(address), + change.old_balance, + change.new_balance, + ); + } + if change.storage_destroyed { + for (key, bucket) in &self.blockers { + if matches!(key, ValidityPredicateKey::Storage(key_address, _) if *key_address == address) + { + bucket.extend_all(&mut effects.affected_transactions); + effects.woken_buckets += 1; + } + } + } else { + for (slot, old, new) in changed_slots { + self.wake_bucket(effects, ValidityPredicateKey::Storage(address, slot), old, new); + } + } + } + fn wake_bucket( &self, effects: &mut StateChangeEffects, @@ -313,6 +408,17 @@ impl ParkedPredicateIndex { } } +/// Account fields that validity predicates read, before and after a state change. +#[derive(Debug, Clone, Copy)] +struct AccountChange { + old_nonce: u64, + new_nonce: u64, + old_balance: U256, + new_balance: U256, + /// Whether every storage slot of the account may have changed. + storage_destroyed: bool, +} + impl ValidityPredicateKey { const fn is_state(self) -> bool { matches!(self, Self::Balance(_) | Self::Nonce(_) | Self::Storage(_, _)) diff --git a/crates/execution/rpc/src/debug.rs b/crates/execution/rpc/src/debug.rs index 962aeb5c88..a300e302b7 100644 --- a/crates/execution/rpc/src/debug.rs +++ b/crates/execution/rpc/src/debug.rs @@ -232,7 +232,7 @@ where .await .map_err(PayloadBuilderError::other)?; - let builder = Builder::new(|_| { + let builder = Builder::new(|_, _| { NoopPayloadTransactions::::default() }); diff --git a/crates/execution/runner/src/node.rs b/crates/execution/runner/src/node.rs index ecd3689f83..cc0f1ddbe2 100644 --- a/crates/execution/runner/src/node.rs +++ b/crates/execution/runner/src/node.rs @@ -120,6 +120,7 @@ impl BaseNode { signature_limit: mempool_sender_limit, payment_limit: mempool_payer_limit, }) + .with_validity_pool_limits(self.args.validity_pool_limits()) .with_additional_trusted_delegation_targets( self.args.mempool_trusted_delegation_targets.iter().copied(), ), diff --git a/crates/execution/tx-forwarding/README.md b/crates/execution/tx-forwarding/README.md index b0154c55f0..23d2aef45f 100644 --- a/crates/execution/tx-forwarding/README.md +++ b/crates/execution/tx-forwarding/README.md @@ -9,7 +9,7 @@ This crate provides: - **`TxForwardingService`**: Starts one reader and forwarder pipeline per destination - **`TxForwardingHandle`**: Gracefully stops readers and drains destination queues - **Per-destination delivery**: Each builder has an independent bounded queue and deduplication cache -- **Resend logic**: Automatically resends transactions that haven't been included after a configurable window +- **Resend logic**: Automatically resends transactions that haven't been included after a configurable window; validity transactions are resent only after a failed forward, because they are expected to wait in the pool until their predicates hold and the builder keeps them until their block expiry ## Forwarding order diff --git a/crates/execution/tx-forwarding/src/config.rs b/crates/execution/tx-forwarding/src/config.rs index e5d00c8818..e74d904290 100644 --- a/crates/execution/tx-forwarding/src/config.rs +++ b/crates/execution/tx-forwarding/src/config.rs @@ -2,6 +2,7 @@ use std::time::Duration; +use base_execution_txpool::DEFAULT_MAX_VALIDITY_EXPIRY_SECS; use url::Url; use crate::{forwarder::ForwarderConfig, reader::ReaderConfig}; @@ -35,6 +36,14 @@ pub struct TxForwardingConfig { pub builder_urls: Vec, /// Resend transactions that haven't been included after this duration in milliseconds. pub resend_after_ms: u64, + /// Longest lifetime of a validity transaction in seconds, as enforced at ingress by + /// `--validity-max-expiry-secs`. + /// + /// A validity transaction is resent only after a failed forward. One forwarded without a + /// reported failure is held for twice this long (at least `resend_after_ms`), which outlives + /// it, so it is never resent on a timer: staying in the pool until its predicates hold is its expected state, and the + /// builder keeps it parked until its block expiry. + pub validity_lifetime_secs: u64, /// Maximum number of transactions per batch (0 = unlimited). pub max_batch_size: usize, /// Maximum RPC requests per second per forwarder (0 = unlimited). @@ -73,6 +82,7 @@ impl Default for TxForwardingConfig { builder_urls: Vec::new(), // Default: 2 blocks (~4 seconds on Base) resend_after_ms: DEFAULT_RESEND_AFTER_MS, + validity_lifetime_secs: DEFAULT_MAX_VALIDITY_EXPIRY_SECS, max_batch_size: DEFAULT_MAX_BATCH_SIZE, max_rps: DEFAULT_MAX_RPS, fifo_percent: DEFAULT_FIFO_PERCENT, @@ -103,6 +113,13 @@ impl TxForwardingConfig { self } + /// Sets the longest lifetime of a validity transaction in seconds; see + /// [`Self::validity_lifetime_secs`]. + pub const fn with_validity_lifetime_secs(mut self, secs: u64) -> Self { + self.validity_lifetime_secs = secs; + self + } + /// Sets the maximum batch size per RPC request. pub const fn with_max_batch_size(mut self, size: usize) -> Self { self.max_batch_size = size; @@ -167,6 +184,10 @@ impl TxForwardingConfig { .max(1); ReaderConfig { resend_after: Duration::from_millis(self.resend_after_ms), + // At least the standard window, so a short or zero lifetime cannot make held + // validity transactions resend faster than standard ones. + validity_hold: Duration::from_secs(self.validity_lifetime_secs.saturating_mul(2)) + .max(Duration::from_millis(self.resend_after_ms)), channel_capacity, poll_interval: ReaderConfig::POLL_INTERVAL, fifo_percent: self.fifo_percent, @@ -188,6 +209,25 @@ impl TxForwardingConfig { mod tests { use super::*; + #[test] + fn validity_hold_is_twice_the_validity_lifetime() { + let config = TxForwardingConfig::new(vec!["http://builder.test".parse().unwrap()]); + let reader = config.reader_config(); + assert_eq!(reader.resend_after, Duration::from_millis(DEFAULT_RESEND_AFTER_MS)); + assert_eq!(reader.validity_hold, Duration::from_secs(2 * DEFAULT_MAX_VALIDITY_EXPIRY_SECS)); + + let reader = config + .clone() + .with_resend_after_ms(2_000) + .with_validity_lifetime_secs(15) + .reader_config(); + assert_eq!(reader.resend_after, Duration::from_secs(2)); + assert_eq!(reader.validity_hold, Duration::from_secs(30)); + + let reader = config.with_validity_lifetime_secs(0).reader_config(); + assert_eq!(reader.validity_hold, reader.resend_after, "never shorter than resend_after"); + } + #[test] fn queue_capacity_defaults_to_two_batches() { let config = TxForwardingConfig::new(vec!["http://builder.test".parse().unwrap()]) diff --git a/crates/execution/tx-forwarding/src/extension.rs b/crates/execution/tx-forwarding/src/extension.rs index bfcd2b32f7..770666b8f0 100644 --- a/crates/execution/tx-forwarding/src/extension.rs +++ b/crates/execution/tx-forwarding/src/extension.rs @@ -34,6 +34,7 @@ impl BaseNodeExtension for TxForwardingExtension { info!( builder_urls = ?config.builder_urls, resend_after_ms = config.resend_after_ms, + validity_lifetime_secs = config.validity_lifetime_secs, max_batch_size = config.max_batch_size, max_rps = config.max_rps, "starting transaction forwarding pipeline" diff --git a/crates/execution/tx-forwarding/src/forwarder/task.rs b/crates/execution/tx-forwarding/src/forwarder/task.rs index 6bdeda6253..f1a4598220 100644 --- a/crates/execution/tx-forwarding/src/forwarder/task.rs +++ b/crates/execution/tx-forwarding/src/forwarder/task.rs @@ -1,7 +1,7 @@ use std::{collections::VecDeque, sync::Arc, time::Instant}; use alloy_primitives::TxHash; -use base_execution_txpool::MAX_VALIDATED_INSERT_BATCH; +use base_execution_txpool::{MAX_VALIDATED_INSERT_BATCH, is_already_imported_rejection}; use base_observability_events::{ TransactionEventProducer, TransactionEventType, transaction_event, }; @@ -94,6 +94,8 @@ pub(crate) struct DestinationForwarder { limiter: RateLimiter, buffer: Vec, buffer_limit: usize, + /// Where to report hashes whose forward failed, so the reader can resend them. + failures: Option>, } impl DestinationForwarder { @@ -108,7 +110,31 @@ impl DestinationForwarder { let buffer_limit = config.batch_limit(); let buffer = Vec::with_capacity(buffer_limit); let url_label: Arc = builder_url.to_string().into(); - Self { builder_url, url_label, client, receiver, config, limiter, buffer, buffer_limit } + Self { + builder_url, + url_label, + client, + receiver, + config, + limiter, + buffer, + buffer_limit, + failures: None, + } + } + + /// Reports the hash of every request whose forward failed on `failures`: those in a batch + /// dropped after its retries, and those the destination rejected for any reason other than + /// already holding the transaction. Reports are dropped when the channel is full. + pub(crate) fn with_failure_reports(mut self, failures: mpsc::Sender) -> Self { + self.failures = Some(failures); + self + } + + fn report_failure(&self, tx_hash: Option) { + if let (Some(failures), Some(tx_hash)) = (&self.failures, tx_hash) { + let _ = failures.try_send(tx_hash); + } } /// Runs the forwarder loop until the destination queue closes. @@ -286,6 +312,9 @@ impl DestinationForwarder { error = %e, "batch item rejected", ); + if !is_already_imported_rejection(&e) { + self.report_failure(tx_hash); + } err_count += 1; self.emit_forward_event( TransactionEventType::TxpoolBuilderForwardFailure, @@ -326,6 +355,7 @@ impl DestinationForwarder { ForwarderMetrics::rpc_latency(Arc::clone(&self.url_label)) .record(overall_start.elapsed().as_secs_f64()); for (tx_hash, method) in tx_hashes.iter().zip(&methods) { + self.report_failure(*tx_hash); self.emit_forward_event( TransactionEventType::TxpoolBuilderForwardDropped, *tx_hash, @@ -912,4 +942,59 @@ mod tests { mixed_forwarder.send(&mixed).await.unwrap(); assert_eq!(methods(&received), [INSERT_METHOD]); } + + /// A destination that accepts nonce 0, reports nonce 1 as already held and rejects nonce 2, + /// with the messages the builder's insert RPC uses. + async fn rejecting_rpc_server() -> (url::Url, jsonrpsee::server::ServerHandle) { + let mut module = RpcModule::new(()); + module + .register_method(INSERT_METHOD, |params, _, _| { + let (tx,): (Value,) = params.parse()?; + let message = match tx["raw"].as_str() { + Some("0x01") => "pool rejected transaction: [0x01]: already imported", + Some("0x02") => "pool rejected transaction: [0x02]: nonce too low", + _ => return Ok(()), + }; + Err(ErrorObjectOwned::owned(-32603, message, None::<()>)) + }) + .unwrap(); + let server = Server::builder().build(SocketAddr::from(([127, 0, 0, 1], 0))).await.unwrap(); + let address = server.local_addr().unwrap(); + (url::Url::parse(&format!("http://{address}")).unwrap(), server.start(module)) + } + + fn reported(failures: &mut mpsc::Receiver) -> Vec { + std::iter::from_fn(|| failures.try_recv().ok()).collect() + } + + /// A rejection is a failed forward unless the destination already holds the transaction. + #[tokio::test] + async fn rejections_other_than_already_imported_are_reported_as_failures() { + let (url, _server) = rejecting_rpc_server().await; + let (_sender, receiver) = mpsc::channel(1); + let (report, mut failures) = mpsc::channel(8); + let forwarder = forwarder(url, receiver, config(0, 0)).with_failure_reports(report); + let batch: Vec<_> = (0..3).map(transaction::).collect(); + + forwarder.send_with_retries(batch).await; + + assert_eq!(reported(&mut failures), [transaction::(2).tx_hash]); + } + + /// Every transaction in a batch dropped after its retries is reported as a failure. + #[tokio::test] + async fn a_dropped_batch_reports_every_transaction() { + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let url = url::Url::parse(&format!("http://{}", listener.local_addr().unwrap())).unwrap(); + drop(listener); + let (_sender, receiver) = mpsc::channel(1); + let (report, mut failures) = mpsc::channel(8); + let forwarder = forwarder(url, receiver, config(0, 0)).with_failure_reports(report); + let batch: Vec<_> = (0..2).map(transaction::).collect(); + let expected: Vec<_> = batch.iter().map(|request| request.tx_hash).collect(); + + forwarder.send_with_retries(batch).await; + + assert_eq!(reported(&mut failures), expected); + } } diff --git a/crates/execution/tx-forwarding/src/reader/config.rs b/crates/execution/tx-forwarding/src/reader/config.rs index 20adbd9764..e42be54511 100644 --- a/crates/execution/tx-forwarding/src/reader/config.rs +++ b/crates/execution/tx-forwarding/src/reader/config.rs @@ -12,6 +12,15 @@ pub(crate) struct ReaderConfig { /// duplicates to the forwarder. pub(crate) resend_after: Duration, + /// How long a validity transaction forwarded without a reported failure is held before it + /// may be re-sent. + /// + /// Longer than any validity transaction lives, so in practice one is never resent on a timer: + /// staying in the pool until its predicates hold is its expected state, and the destination + /// keeps it parked until its block expiry. A failed forward puts it on the + /// [`Self::resend_after`] schedule instead. + pub(crate) validity_hold: Duration, + /// Bounded per-destination queue capacity for outgoing transactions. /// /// A transaction's position is fixed once it is queued, so a deep queue delays the point diff --git a/crates/execution/tx-forwarding/src/reader/metrics.rs b/crates/execution/tx-forwarding/src/reader/metrics.rs index 03288da137..e2141587f9 100644 --- a/crates/execution/tx-forwarding/src/reader/metrics.rs +++ b/crates/execution/tx-forwarding/src/reader/metrics.rs @@ -11,6 +11,10 @@ base_metrics::define_metrics! { #[describe("Total transactions queued after per-destination deduplication")] #[label(builder_url)] txs_sent: counter, + #[describe("Total transactions queued again after their resend window passed, by transaction kind")] + #[label(builder_url)] + #[label(kind)] + txs_resent: counter, #[describe("Total transactions skipped by the validator")] #[label(builder_url)] txs_ignored: counter, diff --git a/crates/execution/tx-forwarding/src/reader/task.rs b/crates/execution/tx-forwarding/src/reader/task.rs index ebd8442af0..17116d7e86 100644 --- a/crates/execution/tx-forwarding/src/reader/task.rs +++ b/crates/execution/tx-forwarding/src/reader/task.rs @@ -5,7 +5,7 @@ use std::{ }; use alloy_eips::Encodable2718; -use alloy_primitives::Bytes; +use alloy_primitives::{Bytes, TxHash}; use base_execution_txpool::{ BasePooledTx, BestTransactionLane, NoExtensions, UnifiedTipOrdering, UnifiedTipPriority, ValidatedTransaction, ValidatedTransactionExtensions, @@ -49,6 +49,8 @@ pub(crate) struct DestinationReader { pool: P, config: ReaderConfig, recently_sent: RecentlySent, + /// Hashes whose forward to this destination failed, reported by its forwarder. + failures: Option>, lane_mix: LaneMix, sender: mpsc::Sender>, cancel: CancellationToken, @@ -81,10 +83,42 @@ where cancel: CancellationToken, builder_url: Url, ) -> Self { - let recently_sent = RecentlySent::new(config.resend_after); + let recently_sent = RecentlySent::new(config.resend_after, config.validity_hold); let lane_mix = LaneMix::new(config.fifo_percent); let url_label = builder_url.to_string().into(); - Self { pool, config, recently_sent, lane_mix, sender, cancel, builder_url, url_label } + Self { + pool, + config, + recently_sent, + failures: None, + lane_mix, + sender, + cancel, + builder_url, + url_label, + } + } + + /// Puts transactions whose forward failed, as reported on `failures`, back on the standard + /// resend schedule. A validity transaction is otherwise never resent on a timer. + pub(crate) fn with_failure_reports(mut self, failures: mpsc::Receiver) -> Self { + self.failures = Some(failures); + self + } + + /// Applies the failures the forwarder has reported since the last call. + fn drain_failures(&mut self) { + let Some(failures) = self.failures.as_mut() else { return }; + loop { + match failures.try_recv() { + Ok(hash) => self.recently_sent.mark_failed(&hash), + Err(mpsc::error::TryRecvError::Empty) => return, + Err(mpsc::error::TryRecvError::Disconnected) => { + self.failures = None; + return; + } + } + } } /// Blocking loop — runs until the [`CancellationToken`] is cancelled. @@ -92,6 +126,7 @@ where info!( builder_url = %self.builder_url, resend_after_ms = self.config.resend_after.as_millis() as u64, + validity_hold_ms = self.config.validity_hold.as_millis() as u64, channel_capacity = self.config.channel_capacity, poll_interval_ms = self.config.poll_interval.as_millis() as u64, fifo_percent = self.config.fifo_percent, @@ -100,6 +135,7 @@ where let mut snapshot: Option<(Instant, Snapshot)> = None; while self.wait_for_capacity() { + self.drain_failures(); let stale = snapshot.as_ref().is_none_or(|(taken_at, lanes)| { lanes.is_empty() || taken_at.elapsed() >= Self::SNAPSHOT_MAX_AGE }); @@ -168,12 +204,17 @@ where /// forwarder is gone. fn fill_queue(&mut self, lanes: &mut Snapshot) -> Option { let mut txs_sent: u64 = 0; + let (mut standard_resent, mut validity_resent) = (0u64, 0u64); while self.sender.capacity() > 0 { let Some(tx) = lanes.pop(self.lane_mix.next_lane()) else { break }; let hash = *tx.hash(); match self.try_enqueue(&tx) { Ok(()) => { - self.recently_sent.mark_sent(hash); + let is_validity = Self::is_validity(&tx); + if self.recently_sent.was_sent(&hash) { + if is_validity { validity_resent += 1 } else { standard_resent += 1 } + } + self.recently_sent.mark_sent(hash, is_validity); txs_sent += 1; } // Only this reader produces into the queue, so capacity cannot shrink between the @@ -183,6 +224,11 @@ where } } Metrics::txs_sent(Arc::clone(&self.url_label)).increment(txs_sent); + for (kind, resent) in [("standard", standard_resent), ("validity", validity_resent)] { + if resent > 0 { + Metrics::txs_resent(Arc::clone(&self.url_label), kind).increment(resent); + } + } Some(txs_sent) } @@ -202,6 +248,12 @@ where !self.sender.is_closed() } + /// Returns whether `transaction` carries validity predicates, which hold it for + /// [`ReaderConfig::validity_hold`] instead of resending it after the standard window. + fn is_validity(transaction: &ValidPoolTransaction) -> bool { + !transaction.transaction.validity_conditions().is_empty() + } + /// Attempts to queue the transaction without waiting on a stale pool snapshot. fn try_enqueue( &self, @@ -247,7 +299,7 @@ mod tests { use alloy_consensus::transaction::Recovered; use alloy_primitives::{Address, B256, TxKind, U256}; use base_common_consensus::{BaseTransactionSigned, TxDeposit}; - use base_execution_txpool::BasePooledTransaction; + use base_execution_txpool::{BasePooledTransaction, ValidityOperator, ValidityPredicate}; use reth_transaction_pool::{ TransactionOrigin, identifier::TransactionId, noop::NoopTransactionPool, }; @@ -301,6 +353,7 @@ mod tests { NoopTransactionPool::new(), ReaderConfig { resend_after: Duration::from_secs(4), + validity_hold: Duration::from_secs(60), channel_capacity: 1, poll_interval: Duration::from_millis(1), fifo_percent: 20, @@ -348,4 +401,53 @@ mod tests { Err(mpsc::error::TrySendError::Closed(())) )); } + + /// A validity transaction is not resent on a timer, while a standard transaction sent with it + /// is; a reported forwarding failure puts the validity transaction back on the standard + /// schedule. + #[test] + fn a_validity_transaction_is_resent_only_after_a_reported_failure() { + let standard = transaction(0); + let plain = transaction(1); + let predicate = ValidityPredicate::BlockNumber { + op: ValidityOperator::LessThan, + value: U256::from(1_000), + }; + let validity = Arc::new(ValidPoolTransaction { + transaction_id: plain.transaction_id, + transaction: plain.transaction.clone().with_validity_predicates(vec![predicate]), + propagate: plain.propagate, + timestamp: plain.timestamp, + origin: plain.origin, + authority_ids: None, + }); + let (sender, _queue) = mpsc::channel::>(8); + let (report, failures) = mpsc::channel(4); + let mut reader = DestinationReader::new( + NoopTransactionPool::::new(), + ReaderConfig { + resend_after: Duration::ZERO, + validity_hold: Duration::from_secs(60), + channel_capacity: 8, + poll_interval: ReaderConfig::POLL_INTERVAL, + fifo_percent: 20, + }, + sender, + CancellationToken::new(), + "http://builder.test".parse().unwrap(), + ) + .with_failure_reports(failures); + let mut lanes = LaneScheduler::new(); + for tx in [&standard, &validity] { + lanes.push(Arc::clone(tx), None, tx.timestamp, Priority::None); + } + + assert_eq!(reader.fill_queue(&mut lanes), Some(2)); + assert!(!reader.recently_sent.was_recently_sent(standard.hash())); + assert!(reader.recently_sent.was_recently_sent(validity.hash()), "held"); + + report.try_send(*validity.hash()).unwrap(); + reader.drain_failures(); + assert!(!reader.recently_sent.was_recently_sent(validity.hash()), "due after a failure"); + } } diff --git a/crates/execution/tx-forwarding/src/reader/validator.rs b/crates/execution/tx-forwarding/src/reader/validator.rs index dfcb7e23bd..777045e822 100644 --- a/crates/execution/tx-forwarding/src/reader/validator.rs +++ b/crates/execution/tx-forwarding/src/reader/validator.rs @@ -8,26 +8,41 @@ use alloy_primitives::B256; /// Number of [`RecentlySent::was_recently_sent`] calls between pruning sweeps. const PRUNE_INTERVAL: u64 = 1000; -/// Per-destination deduplication tracker with a configurable time-to-live. +/// One transaction accepted by a destination's queue. +#[derive(Debug, Clone, Copy)] +struct Entry { + /// When the transaction was queued. + sent: Instant, + /// Whether the transaction waits out the validity hold instead of the resend window. + held: bool, +} + +/// Per-destination deduplication tracker. +/// +/// Tracks transaction hashes accepted by one destination's queue. A standard transaction becomes +/// due for a resend `resend_after` after it was queued. A validity transaction is held for +/// `validity_hold` instead, long enough to outlive any validity transaction, so it is never resent +/// on a timer: staying in the pool until its predicates hold is its expected state, and the +/// destination keeps it parked until its block expiry. Only a failed forward, reported through +/// [`Self::mark_failed`], puts it back on the standard schedule. /// -/// Tracks transaction hashes accepted by one destination's queue. A transaction -/// whose hash appears in the map -/// and whose entry is younger than `resend_after` will be skipped. Expired -/// entries are pruned periodically (every [`PRUNE_INTERVAL`] lookups) to -/// bound memory. +/// Entries are kept for at least `resend_after` past their due time, so a transaction due for a +/// resend can be told apart from one never sent. Older entries are pruned every +/// [`PRUNE_INTERVAL`] lookups to bound memory, so one can outlive that until the next sweep. pub(super) struct RecentlySent { - seen: HashMap, + seen: HashMap, resend_after: Duration, + validity_hold: Duration, check_count: u64, } impl RecentlySent { /// Creates a new tracker. - pub(super) fn new(resend_after: Duration) -> Self { - Self { seen: HashMap::new(), resend_after, check_count: 0 } + pub(super) fn new(resend_after: Duration, validity_hold: Duration) -> Self { + Self { seen: HashMap::new(), resend_after, validity_hold, check_count: 0 } } - /// Returns `true` if the hash was sent within the `resend_after` window. + /// Returns `true` if the hash was sent and is not yet due for a resend. /// /// Triggers a pruning sweep every [`PRUNE_INTERVAL`] calls. pub(super) fn was_recently_sent(&mut self, hash: &B256) -> bool { @@ -36,12 +51,27 @@ impl RecentlySent { self.prune_expired(); } - self.seen.get(hash).is_some_and(|instant| instant.elapsed() < self.resend_after) + self.seen.get(hash).is_some_and(|entry| Instant::now() < self.due(entry)) + } + + /// Returns `true` if the hash has an entry, which lasts at least `resend_after` past its due + /// time; see [`RecentlySent`]. + pub(super) fn was_sent(&self, hash: &B256) -> bool { + self.seen.contains_key(hash) } - /// Records a hash as sent at the current instant. - pub(super) fn mark_sent(&mut self, hash: B256) { - self.seen.insert(hash, Instant::now()); + /// Records a hash as sent now. A validity transaction is held for the validity hold rather + /// than resent after the standard window. + pub(super) fn mark_sent(&mut self, hash: B256, is_validity: bool) { + self.seen.insert(hash, Entry { sent: Instant::now(), held: is_validity }); + } + + /// Puts a transaction whose forward failed back on the standard resend schedule. Has no + /// effect on a transaction without an entry or already on that schedule. + pub(super) fn mark_failed(&mut self, hash: &B256) { + if let Some(entry) = self.seen.get_mut(hash) { + entry.held = false; + } } /// Returns the current cache size. @@ -55,9 +85,17 @@ impl RecentlySent { self.seen.is_empty() } + fn due(&self, entry: &Entry) -> Instant { + entry.sent + if entry.held { self.validity_hold } else { self.resend_after } + } + fn prune_expired(&mut self) { let now = Instant::now(); - self.seen.retain(|_, instant| now.duration_since(*instant) < self.resend_after); + let (resend_after, validity_hold) = (self.resend_after, self.validity_hold); + self.seen.retain(|_, entry| { + let window = if entry.held { validity_hold } else { resend_after }; + now < entry.sent + window + resend_after + }); } } @@ -65,6 +103,7 @@ impl std::fmt::Debug for RecentlySent { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("RecentlySent") .field("resend_after", &self.resend_after) + .field("validity_hold", &self.validity_hold) .field("entries", &self.seen.len()) .finish() } @@ -74,28 +113,31 @@ impl std::fmt::Debug for RecentlySent { mod tests { use super::*; + const WINDOW: Duration = Duration::from_secs(5); + const HOLD: Duration = Duration::from_secs(60); + #[test] fn unseen_hash_is_not_recent() { - let mut tracker = RecentlySent::new(Duration::from_secs(5)); + let mut tracker = RecentlySent::new(WINDOW, HOLD); assert!(!tracker.was_recently_sent(&B256::random())); } #[test] fn sent_hash_is_recent() { - let mut tracker = RecentlySent::new(Duration::from_secs(5)); + let mut tracker = RecentlySent::new(WINDOW, HOLD); let hash = B256::random(); - tracker.mark_sent(hash); + tracker.mark_sent(hash, false); assert!(tracker.was_recently_sent(&hash)); } #[test] fn trackers_deduplicate_independently() { - let mut first_destination = RecentlySent::new(Duration::from_secs(5)); - let mut second_destination = RecentlySent::new(Duration::from_secs(5)); + let mut first_destination = RecentlySent::new(WINDOW, HOLD); + let mut second_destination = RecentlySent::new(WINDOW, HOLD); let hash = B256::random(); - first_destination.mark_sent(hash); + first_destination.mark_sent(hash, false); assert!(first_destination.was_recently_sent(&hash)); assert!(!second_destination.was_recently_sent(&hash)); @@ -103,38 +145,89 @@ mod tests { #[test] fn expired_hash_is_not_recent() { - let mut tracker = RecentlySent::new(Duration::from_millis(10)); + let mut tracker = RecentlySent::new(Duration::from_millis(10), HOLD); + let hash = B256::random(); + + tracker.mark_sent(hash, false); + std::thread::sleep(Duration::from_millis(15)); + assert!(!tracker.was_recently_sent(&hash)); + assert!(tracker.was_sent(&hash), "a due entry is kept to mark the resend"); + } + + /// A validity transaction is held after a standard transaction sent at the same time has + /// become due. + #[test] + fn a_validity_transaction_is_held_past_the_resend_window() { + let mut tracker = RecentlySent::new(Duration::from_millis(10), HOLD); + let (standard, validity) = (B256::random(), B256::random()); + + tracker.mark_sent(standard, false); + tracker.mark_sent(validity, true); + std::thread::sleep(Duration::from_millis(15)); + + assert!(!tracker.was_recently_sent(&standard)); + assert!(tracker.was_recently_sent(&validity)); + } + + /// A failed forward releases a held validity transaction onto the standard schedule, counted + /// from when it was sent. + #[test] + fn a_failed_validity_transaction_follows_the_resend_window() { + let mut tracker = RecentlySent::new(Duration::from_millis(10), HOLD); let hash = B256::random(); - tracker.mark_sent(hash); + tracker.mark_sent(hash, true); + tracker.mark_failed(&hash); + assert!(tracker.was_recently_sent(&hash), "still inside the standard window"); + std::thread::sleep(Duration::from_millis(15)); assert!(!tracker.was_recently_sent(&hash)); } + #[test] + fn marking_an_unsent_hash_failed_adds_nothing() { + let mut tracker = RecentlySent::new(WINDOW, HOLD); + + tracker.mark_failed(&B256::random()); + + assert!(tracker.is_empty()); + } + #[test] fn len_tracks_entries() { - let mut tracker = RecentlySent::new(Duration::from_secs(5)); + let mut tracker = RecentlySent::new(WINDOW, HOLD); assert!(tracker.is_empty()); - tracker.mark_sent(B256::random()); - tracker.mark_sent(B256::random()); - tracker.mark_sent(B256::random()); + tracker.mark_sent(B256::random(), false); + tracker.mark_sent(B256::random(), false); + tracker.mark_sent(B256::random(), true); assert_eq!(tracker.len(), 3); } #[test] fn prune_removes_expired() { - let mut tracker = RecentlySent::new(Duration::from_millis(10)); + let mut tracker = RecentlySent::new(Duration::from_millis(10), HOLD); for _ in 0..5 { - tracker.mark_sent(B256::random()); + tracker.mark_sent(B256::random(), false); } assert_eq!(tracker.len(), 5); - std::thread::sleep(Duration::from_millis(15)); + std::thread::sleep(Duration::from_millis(25)); tracker.prune_expired(); assert_eq!(tracker.len(), 0); } + + #[test] + fn prune_keeps_held_entries() { + let mut tracker = RecentlySent::new(Duration::from_millis(10), HOLD); + tracker.mark_sent(B256::random(), true); + + std::thread::sleep(Duration::from_millis(25)); + tracker.prune_expired(); + + assert_eq!(tracker.len(), 1); + } } diff --git a/crates/execution/tx-forwarding/src/service.rs b/crates/execution/tx-forwarding/src/service.rs index f250f5dc62..22fc341e4a 100644 --- a/crates/execution/tx-forwarding/src/service.rs +++ b/crates/execution/tx-forwarding/src/service.rs @@ -19,6 +19,12 @@ use crate::{ reader::DestinationReader, }; +/// Capacity of each destination's failure-report channel from forwarder to reader. +/// +/// A report that finds the channel full is dropped, and that transaction is then not resent; +/// standard transactions are resent on their own schedule regardless. +const FAILURE_REPORT_CAPACITY: usize = 4096; + /// Why a forwarding destination could not be started. #[derive(Debug, thiserror::Error)] pub enum ForwardingSetupError { @@ -98,6 +104,7 @@ impl TxForwardingService { }; let (sender, receiver) = mpsc::channel(reader_config.channel_capacity); + let (report, failures) = mpsc::channel(FAILURE_REPORT_CAPACITY); // `E` is named explicitly: nothing else in this expression pins the queue's item type, // since both ends are generic over it. let mut reader = DestinationReader::::new( @@ -106,13 +113,15 @@ impl TxForwardingService { sender, reader_cancel.child_token(), builder_url.clone(), - ); + ) + .with_failure_reports(failures); let forwarder = DestinationForwarder::new( builder_url.clone(), client, receiver, Arc::clone(&forwarder_config), - ); + ) + .with_failure_reports(report); reader_tasks.push(executor.spawn_blocking_task(Box::pin(async move { reader.run(); diff --git a/crates/execution/txpool-rpc/src/rpc.rs b/crates/execution/txpool-rpc/src/rpc.rs index 879f544094..038dbafe3e 100644 --- a/crates/execution/txpool-rpc/src/rpc.rs +++ b/crates/execution/txpool-rpc/src/rpc.rs @@ -36,6 +36,9 @@ use tracing::{debug, info, warn}; pub const VALIDITY_TX_PRE_EVEREST_RPC_ERROR: &str = "EIP-8130 validity transactions are gated behind \ the Everest hard fork; they are not accepted before Everest is active"; +/// Error returned for flashblock-index predicates once blocks are built without flashblocks. +const FLASHBLOCK_INDEX_UNSUPPORTED_MESSAGE: &str = "flashblock_index predicates are not supported after Denim: blocks are built every 200ms without flashblocks"; + /// Legacy full-block cadence used before Denim activates. const LEGACY_BLOCK_INTERVAL_MILLIS: u64 = 2_000; @@ -211,16 +214,21 @@ where Ok(Some((header.number(), header.timestamp()))) } - /// Returns the full-block interval, in milliseconds, of the block being built. - fn block_interval_millis(&self, latest_timestamp: u64) -> u64 { + /// Returns whether the block being built uses Denim's native 200ms cadence. + fn builds_subsecond_blocks(&self, latest_timestamp: u64) -> bool { // The target build can be the first Denim block even though the latest committed header // is pre-Denim. Check both its parent timestamp and the next legacy block timestamp so - // the window uses Denim's 200ms full-block cadence at that transition. + // the transition block counts as a subsecond block. let next_legacy_timestamp = latest_timestamp.saturating_add(LEGACY_BLOCK_INTERVAL_MILLIS.saturating_div(1_000)); - if self.provider.chain_spec().is_denim_active_at_timestamp(latest_timestamp) - || self.provider.chain_spec().is_denim_active_at_timestamp(next_legacy_timestamp) - { + let chain_spec = self.provider.chain_spec(); + chain_spec.is_denim_active_at_timestamp(latest_timestamp) + || chain_spec.is_denim_active_at_timestamp(next_legacy_timestamp) + } + + /// Returns the full-block interval, in milliseconds, of the block being built. + fn block_interval_millis(&self, latest_timestamp: u64) -> u64 { + if self.builds_subsecond_blocks(latest_timestamp) { RollupConfig::NATIVE_SUBSECOND_BLOCK_INTERVAL_MILLIS } else { LEGACY_BLOCK_INTERVAL_MILLIS @@ -288,6 +296,28 @@ where tx: Bytes, options: SendRawTransactionValidityOptions, ) -> RpcResult { + let latest = self.latest_block_number_and_timestamp(); + + // Native 200ms blocks have no flashblocks, and the native builder skips every transaction + // carrying a flashblock-index predicate. Reject them before forwarding so the submitter + // gets an error from the first node instead of a hash for a transaction that will never + // be included. A forwarding node that cannot read its head defers to the upstream node, + // which runs the same check. + let latest_timestamp = + latest.as_ref().ok().copied().flatten().map_or(0, |(_, timestamp)| timestamp); + if self.builds_subsecond_blocks(latest_timestamp) + && options + .validity + .iter() + .any(|predicate| matches!(predicate, ValidityPredicate::FlashblockIndex { .. })) + { + return Err(ErrorObjectOwned::owned( + ErrorCode::InvalidParams.code(), + FLASHBLOCK_INDEX_UNSUPPORTED_MESSAGE, + None::<()>, + )); + } + if let Some(client) = &self.sequencer_client { return client .request("base_sendRawTransactionValidity", (tx, options)) @@ -295,7 +325,7 @@ where .map_err(ErrorObjectOwned::from); } - let latest = self.latest_block_number_and_timestamp()?; + let latest = latest?; ValidityPredicate::validate_batch(&options.validity, self.max_validity_predicates) .map_err(|error| { @@ -772,6 +802,102 @@ mod tests { ); } + /// Provider whose chain builds native 200ms blocks from genesis. + fn denim_provider() -> MockEthProvider> { + MockEthProvider::::new() + .with_chain_spec(Arc::new( + BaseChainSpecBuilder::base_mainnet() + .with_fork(BaseUpgrade::Denim, ForkCondition::Timestamp(0)) + .build(), + )) + .with_genesis_block() + } + + fn flashblock_index_request() -> (Bytes, SendRawTransactionValidityOptions) { + let signer = PrivateKeySigner::random(); + let (raw, mut options) = validity_request(signed_eip1559(&signer, 0, 1)); + options.validity.push(ValidityPredicate::FlashblockIndex { + op: base_execution_txpool::ValidityOperator::GreaterThanOrEqual, + value: U256::from(1), + }); + (raw, options) + } + + #[tokio::test] + async fn send_raw_transaction_validity_rejects_flashblock_index_after_denim() { + let rpc = validity_rpc(denim_provider()); + let (raw, options) = flashblock_index_request(); + + let error = rpc + .send_raw_transaction_validity(raw, options) + .await + .expect_err("flashblock-index predicates should be rejected after Denim"); + + assert_eq!(error.code(), ErrorCode::InvalidParams.code()); + assert_eq!(error.message(), FLASHBLOCK_INDEX_UNSUPPORTED_MESSAGE); + } + + #[tokio::test] + async fn send_raw_transaction_validity_rejects_flashblock_index_on_the_denim_transition_block() + { + // The latest header predates Denim, but the block being built is the first Denim block. + let provider = MockEthProvider::::new() + .with_chain_spec(Arc::new( + BaseChainSpecBuilder::base_mainnet() + .with_fork(BaseUpgrade::Denim, ForkCondition::Timestamp(2)) + .build(), + )) + .with_genesis_block(); + let rpc = validity_rpc(provider); + let (raw, options) = flashblock_index_request(); + + let error = rpc + .send_raw_transaction_validity(raw, options) + .await + .expect_err("flashblock-index predicates should be rejected on the transition block"); + + assert_eq!(error.message(), FLASHBLOCK_INDEX_UNSUPPORTED_MESSAGE); + } + + #[tokio::test] + async fn forwarding_node_rejects_flashblock_index_after_denim_without_forwarding() { + let sequencer = MockServer::start(); + let mock = sequencer.mock(|when, then| { + when.method(POST).path("/"); + then.status(200) + .header("content-type", "application/json") + .json_body(json!({"jsonrpc": "2.0", "id": 0, "result": TxHash::ZERO})); + }); + let client = SequencerClient::new_http_with_headers(sequencer.base_url(), vec![]).unwrap(); + let rpc = + SendRawTransactionValidityApiImpl::new(denim_provider(), test_transaction_sender()) + .with_sequencer_client(client); + let (raw, options) = flashblock_index_request(); + + let error = rpc + .send_raw_transaction_validity(raw, options) + .await + .expect_err("a forwarding node should reject flashblock-index predicates after Denim"); + + assert_eq!(error.code(), ErrorCode::InvalidParams.code()); + assert_eq!(error.message(), FLASHBLOCK_INDEX_UNSUPPORTED_MESSAGE); + mock.assert_calls(0); + } + + #[tokio::test] + async fn send_raw_transaction_validity_allows_flashblock_index_before_denim() { + // Mainnet's chain spec has not activated Denim at the genesis timestamp. + let rpc = validity_rpc(pre_everest_provider()); + let (raw, options) = flashblock_index_request(); + + // The noop pool rejects insertion, so any error must come from a later stage. + let result = rpc.send_raw_transaction_validity(raw, options).await; + + if let Err(error) = result { + assert_ne!(error.message(), FLASHBLOCK_INDEX_UNSUPPORTED_MESSAGE); + } + } + fn signed_eip1559(signer: &PrivateKeySigner, nonce: u64, priority_fee: u128) -> Bytes { let tx = TxEip1559 { chain_id: 8453, @@ -934,7 +1060,8 @@ mod tests { let capture = TransactionEventCapture::install(); let signer = PrivateKeySigner::random(); let raw = signed_eip1559(&signer, 0, 1); - let rpc = validity_rpc(everest_provider()); + // Pre-Denim, so the flashblock-index variant is still admitted. + let rpc = validity_rpc(pre_everest_provider()); let options = SendRawTransactionValidityOptions { validity: all_predicate_variants(), validity_signature: None, @@ -1215,8 +1342,11 @@ mod tests { #[tokio::test] async fn send_raw_transaction_validity_rejects_unsatisfiable_flashblock_index() { - let rpc = - SendRawTransactionValidityApiImpl::new(everest_provider(), test_transaction_sender()); + // Pre-Denim; after Denim every flashblock-index predicate is rejected before this check. + let rpc = SendRawTransactionValidityApiImpl::new( + pre_everest_provider(), + test_transaction_sender(), + ); let (raw, mut options) = validity_request(Bytes::from_static(&[0x02])); // A flashblock-index predicate that only holds at index 0, which pooled // transactions never reach, would park forever if admitted. diff --git a/crates/execution/txpool-tracing/Cargo.toml b/crates/execution/txpool-tracing/Cargo.toml index 49a6ed3d6f..123fe0d5c6 100644 --- a/crates/execution/txpool-tracing/Cargo.toml +++ b/crates/execution/txpool-tracing/Cargo.toml @@ -13,6 +13,7 @@ workspace = true [dependencies] # workspace +base-execution-txpool.workspace = true base-flashblocks.workspace = true base-node-runner.workspace = true base-observability-events.workspace = true @@ -53,6 +54,9 @@ reth-transaction-pool = { workspace = true, features = ["test-utils"] } # revm revm-database.workspace = true +# metrics +metrics-exporter-prometheus.workspace = true + # alloy alloy-eips = { workspace = true, features = ["std"] } alloy-consensus = { workspace = true, features = ["std"] } @@ -63,6 +67,7 @@ base-common-consensus = { workspace = true, features = ["reth"] } # misc eyre.workspace = true +rstest.workspace = true [features] default = [ "metrics" ] diff --git a/crates/execution/txpool-tracing/README.md b/crates/execution/txpool-tracing/README.md index ee45736ebc..ff2f2bea22 100644 --- a/crates/execution/txpool-tracing/README.md +++ b/crates/execution/txpool-tracing/README.md @@ -12,7 +12,7 @@ This crate provides: - **Transaction Tracing**: Subscribes to mempool events and chain notifications to track how long a transaction spends in each stage of the lifecycle before it is included, dropped, or replaced. - **Transaction Status RPC**: An RPC API (`txpool_transactionStatus`) to query the current status and lifecycle events of a transaction by hash. - **Tracker**: Core tracking logic for recording pending/queued transitions, replacements, drops, and block inclusion. -- **Metrics**: Histogram metrics for mempool residency by event type to help spot latency regressions. +- **Metrics**: Histogram metrics for mempool residency by event type to help spot latency regressions. Inclusion latency metrics carry a `flow` label (`standard` or `validity`), because validity transactions can wait in the pool until their predicates hold. ## Usage diff --git a/crates/execution/txpool-tracing/src/events.rs b/crates/execution/txpool-tracing/src/events.rs index 9245f5b620..1099475bb3 100644 --- a/crates/execution/txpool-tracing/src/events.rs +++ b/crates/execution/txpool-tracing/src/events.rs @@ -44,6 +44,43 @@ pub enum Pool { Queued, } +/// Transaction flow used to label inclusion metrics. +/// +/// Validity transactions carry predicates that can hold them in the pool until the predicates +/// are satisfied, so their inclusion latency is reported separately from standard transactions. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub enum TxFlow { + /// Transaction without validity predicates. + #[default] + Standard, + /// Transaction carrying validity predicates. + Validity, +} + +impl TxFlow { + /// Returns the flow for a transaction, given whether it carries validity predicates. + pub const fn from_validity_predicates(has_validity_predicates: bool) -> Self { + if has_validity_predicates { Self::Validity } else { Self::Standard } + } + + /// Returns the metric label value for this flow. + pub const fn as_str(self) -> &'static str { + match self { + Self::Standard => "standard", + Self::Validity => "validity", + } + } +} + +/// Pool metadata resolved for a transaction when it enters the pending or queued pool. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct PooledTxInfo { + /// Sender and nonce of the transaction. + pub nonce_slot: NonceSlot, + /// Flow used to label the transaction's inclusion metrics. + pub flow: TxFlow, +} + /// Key for tracking a unique nonce slot: `(sender, nonce)`. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub struct NonceSlot { @@ -89,6 +126,7 @@ pub struct EventLog { pub(crate) mempool_time: Instant, pub(crate) pending_time: Option, pub(crate) fb_included: bool, + pub(crate) flow: TxFlow, pub(crate) events: Vec<(DateTime, TxEvent)>, pub(crate) limit: usize, } @@ -102,6 +140,7 @@ impl EventLog { mempool_time: now, pending_time, fb_included: false, + flow: TxFlow::Standard, events: vec![(t, event)], limit: 10, } diff --git a/crates/execution/txpool-tracing/src/lib.rs b/crates/execution/txpool-tracing/src/lib.rs index eecc70663e..56aa17bd5a 100644 --- a/crates/execution/txpool-tracing/src/lib.rs +++ b/crates/execution/txpool-tracing/src/lib.rs @@ -8,7 +8,7 @@ #![cfg_attr(not(test), warn(unused_crate_dependencies))] mod events; -pub use events::{EventLog, NonceSlot, NonceSummary, Pool, TxEvent}; +pub use events::{EventLog, NonceSlot, NonceSummary, Pool, PooledTxInfo, TxEvent, TxFlow}; mod subscription; pub use subscription::tracex_subscription; diff --git a/crates/execution/txpool-tracing/src/metrics.rs b/crates/execution/txpool-tracing/src/metrics.rs index 11c148057c..c30f2f63ee 100644 --- a/crates/execution/txpool-tracing/src/metrics.rs +++ b/crates/execution/txpool-tracing/src/metrics.rs @@ -3,18 +3,25 @@ base_metrics::define_metrics! { reth_transaction_tracing #[describe("Time taken for a transaction to be included in a block from when it's marked as pending")] + #[label(name = "flow", default = ["standard", "validity"])] inclusion_duration: histogram, #[describe("Time taken for a transaction to be included in a flashblock from when it's marked as pending")] + #[label(name = "flow", default = ["standard", "validity"])] fb_inclusion_duration: histogram, #[describe("Number of transactions included in a flashblock within the healthy threshold")] + #[label(name = "flow", default = ["standard", "validity"])] fb_healthy_inclusions: counter, #[describe("Number of transactions that exceeded the healthy flashblock inclusion threshold")] + #[label(name = "flow", default = ["standard", "validity"])] fb_slow_inclusions: counter, #[describe("Number of transactions included in a block within the healthy threshold")] + #[label(name = "flow", default = ["standard", "validity"])] healthy_inclusions: counter, #[describe("Number of transactions that exceeded the healthy block inclusion threshold")] + #[label(name = "flow", default = ["standard", "validity"])] slow_inclusions: counter, #[describe("End-to-end time from first submission to block inclusion for a (sender, nonce) pair")] + #[label(name = "flow", default = ["standard", "validity"])] e2e_inclusion_duration: histogram, #[describe("Number of replacement transactions per (sender, nonce) pair")] replacement_count: histogram, diff --git a/crates/execution/txpool-tracing/src/subscription.rs b/crates/execution/txpool-tracing/src/subscription.rs index eafb37f07e..8842d508c1 100644 --- a/crates/execution/txpool-tracing/src/subscription.rs +++ b/crates/execution/txpool-tracing/src/subscription.rs @@ -2,6 +2,7 @@ use std::{sync::Arc, time::Instant}; +use base_execution_txpool::BasePooledTx; use base_flashblocks::{FlashblocksAPI, PendingBlocks}; use futures::StreamExt; use reth_node_api::NodePrimitives; @@ -11,7 +12,7 @@ use reth_transaction_pool::{FullTransactionEvent, TransactionPool}; use tokio::sync::broadcast::Receiver; use tokio_stream::wrappers::BroadcastStream; -use crate::{NonceSlot, tracker::Tracker}; +use crate::{NonceSlot, PooledTxInfo, TxFlow, tracker::Tracker}; /// Subscription task that tracks transaction timing from mempool to block inclusion. /// @@ -26,6 +27,7 @@ pub async fn tracex_subscription( ) where N: NodePrimitives, Pool: TransactionPool + 'static, + Pool::Transaction: BasePooledTx, FB: FlashblocksAPI + 'static, { debug!(target: "tracex", "Starting transaction tracking subscription"); @@ -42,8 +44,8 @@ pub async fn tracex_subscription( loop { tokio::select! { Some(full_event) = all_events_stream.next() => { - let nonce_slot = resolve_nonce_slot(&full_event, &pool); - tracker.handle_event(full_event, nonce_slot); + let info = resolve_pooled_tx_info(&full_event, &pool); + tracker.handle_event(full_event, info); }, // Use canonical state notifications to track time to inclusion. @@ -61,13 +63,20 @@ pub async fn tracex_subscription( } } -fn resolve_nonce_slot( +fn resolve_pooled_tx_info( event: &FullTransactionEvent, pool: &Pool, -) -> Option { +) -> Option +where + Pool: TransactionPool, + Pool::Transaction: BasePooledTx, +{ let tx_hash = match event { FullTransactionEvent::Pending(hash) | FullTransactionEvent::Queued(hash, _) => hash, _ => return None, }; - pool.get(tx_hash).map(|tx| NonceSlot::new(tx.sender(), tx.nonce())) + pool.get(tx_hash).map(|tx| PooledTxInfo { + nonce_slot: NonceSlot::new(tx.sender(), tx.nonce()), + flow: TxFlow::from_validity_predicates(!tx.transaction.validity_conditions().is_empty()), + }) } diff --git a/crates/execution/txpool-tracing/src/tracker.rs b/crates/execution/txpool-tracing/src/tracker.rs index 41ab04e07a..bdfd418452 100644 --- a/crates/execution/txpool-tracing/src/tracker.rs +++ b/crates/execution/txpool-tracing/src/tracker.rs @@ -20,7 +20,9 @@ use reth_tracing::tracing::{debug, info}; use reth_transaction_pool::{FullTransactionEvent, PoolTransaction}; use serde_json::{Map, Value, json}; -use crate::{EventLog, NonceSlot, NonceSummary, Pool, TxEvent, metrics::Metrics}; +use crate::{ + EventLog, NonceSlot, NonceSummary, Pool, PooledTxInfo, TxEvent, TxFlow, metrics::Metrics, +}; /// Tracks transactions as they move through the mempool and into blocks. #[derive(Debug, Clone)] @@ -70,26 +72,26 @@ impl Tracker { /// Parse [`FullTransactionEvent`]s and update the tracker. /// - /// `nonce_slot` is populated by the subscription layer for events that only + /// `info` is populated by the subscription layer for events that only /// carry a [`TxHash`] (Pending, Queued) by looking up the pool. pub fn handle_event( &mut self, event: FullTransactionEvent, - nonce_slot: Option, + info: Option, ) { match event { FullTransactionEvent::Pending(tx_hash) => { self.transaction_inserted(tx_hash, TxEvent::Pending); self.transaction_moved(tx_hash, Pool::Pending); - if let Some(slot) = nonce_slot { - self.track_nonce_slot(tx_hash, slot); + if let Some(info) = info { + self.track_pooled_tx_info(tx_hash, info); } } FullTransactionEvent::Queued(tx_hash, _) => { self.transaction_inserted(tx_hash, TxEvent::Queued); self.transaction_moved(tx_hash, Pool::Queued); - if let Some(slot) = nonce_slot { - self.track_nonce_slot(tx_hash, slot); + if let Some(info) = info { + self.track_pooled_tx_info(tx_hash, info); } } FullTransactionEvent::Discarded(tx_hash) => { @@ -250,6 +252,7 @@ impl Tracker { if let Some(mut event_log) = self.txs.pop(&tx_hash) { let mempool_time = event_log.mempool_time; let time_in_mempool = received_at.duration_since(mempool_time); + let flow = event_log.flow; if self.is_overflowed(&tx_hash, &event_log) { return; @@ -267,16 +270,17 @@ impl Tracker { if event == TxEvent::BlockInclusion && let Some(time_pending_to_inclusion) = time_pending_to_inclusion { - Metrics::inclusion_duration().record(time_pending_to_inclusion.as_millis() as f64); + Metrics::inclusion_duration(flow.as_str()) + .record(time_pending_to_inclusion.as_millis() as f64); if time_pending_to_inclusion > Self::SLOW_BLOCK_INCLUSION_THRESHOLD { - Metrics::slow_inclusions().increment(1); + Metrics::slow_inclusions(flow.as_str()).increment(1); } else { - Metrics::healthy_inclusions().increment(1); + Metrics::healthy_inclusions(flow.as_str()).increment(1); } } - self.nonce_completed(&tx_hash, &event, received_at); + self.nonce_completed(&tx_hash, &event, flow, received_at); self.log(&tx_hash, &event_log, &format!("Transaction {event}")); // Inclusion is journaled by the builder; tracer emits cover mempool lifecycle only. if event != TxEvent::BlockInclusion { @@ -313,13 +317,14 @@ impl Tracker { if let Some(pending_time) = event_log.pending_time { let time_pending_to_fb_inclusion = received_at.duration_since(pending_time); - Metrics::fb_inclusion_duration() + let flow = event_log.flow.as_str(); + Metrics::fb_inclusion_duration(flow) .record(time_pending_to_fb_inclusion.as_millis() as f64); if time_pending_to_fb_inclusion > Self::SLOW_FLASHBLOCK_INCLUSION_THRESHOLD { - Metrics::fb_slow_inclusions().increment(1); + Metrics::fb_slow_inclusions(flow).increment(1); } else { - Metrics::fb_healthy_inclusions().increment(1); + Metrics::fb_healthy_inclusions(flow).increment(1); } debug!( @@ -350,6 +355,11 @@ impl Tracker { // inclusion duration rather than inheriting from the original. event_log.pending_time = Some(Instant::now()); event_log.fb_included = false; + // The replacement may already be tracked if the pool reported it first; its flow + // was resolved from the replacement itself and takes precedence. + if let Some(replacement) = self.txs.peek(&replaced_by) { + event_log.flow = replacement.flow; + } self.tx_nonce_slots.pop(&tx_hash); self.txs.put(replaced_by, event_log); @@ -367,6 +377,19 @@ impl Tracker { } } + /// Records the pool metadata of a pending or queued transaction. + fn track_pooled_tx_info(&mut self, tx_hash: TxHash, info: PooledTxInfo) { + self.set_flow(tx_hash, info.flow); + self.track_nonce_slot(tx_hash, info.nonce_slot); + } + + /// Sets the flow used to label the inclusion metrics of a tracked transaction. + pub fn set_flow(&mut self, tx_hash: TxHash, flow: TxFlow) { + if let Some(event_log) = self.txs.peek_mut(&tx_hash) { + event_log.flow = flow; + } + } + fn track_nonce_slot(&mut self, tx_hash: TxHash, slot: NonceSlot) { self.tx_nonce_slots.put(tx_hash, slot); if !self.nonce_summaries.contains(&slot) { @@ -381,7 +404,13 @@ impl Tracker { } } - fn nonce_completed(&mut self, tx_hash: &TxHash, event: &TxEvent, received_at: Instant) { + fn nonce_completed( + &mut self, + tx_hash: &TxHash, + event: &TxEvent, + flow: TxFlow, + received_at: Instant, + ) { let Some(slot) = self.tx_nonce_slots.pop(tx_hash) else { return; }; @@ -390,7 +419,7 @@ impl Tracker { }; if *event == TxEvent::BlockInclusion { let e2e_duration = received_at.duration_since(summary.first_seen); - Metrics::e2e_inclusion_duration().record(e2e_duration.as_millis() as f64); + Metrics::e2e_inclusion_duration(flow.as_str()).record(e2e_duration.as_millis() as f64); Metrics::replacement_count().record(summary.replacement_count as f64); } } @@ -523,6 +552,8 @@ mod tests { use base_flashblocks::FlashblocksAPI; use base_flashblocks_node::test_harness::{FlashblockBuilder, FlashblocksBuilderTestHarness}; use base_test_utils::Account; + use reth_transaction_pool::test_utils::MockTransaction; + use rstest::rstest; use tokio::time; use super::*; @@ -893,7 +924,12 @@ mod tests { assert!(tracker.nonce_summaries.contains(&slot)); assert!(tracker.tx_nonce_slots.contains(&tx_hash)); - tracker.nonce_completed(&tx_hash, &TxEvent::BlockInclusion, Instant::now()); + tracker.nonce_completed( + &tx_hash, + &TxEvent::BlockInclusion, + TxFlow::Standard, + Instant::now(), + ); assert!(!tracker.nonce_summaries.contains(&slot)); assert!(!tracker.tx_nonce_slots.contains(&tx_hash)); @@ -921,7 +957,12 @@ mod tests { // Original hash slot mapping should be gone (overwritten by replacement) assert_eq!(*tracker.tx_nonce_slots.get(&replacement_hash).unwrap(), slot); - tracker.nonce_completed(&replacement_hash, &TxEvent::BlockInclusion, Instant::now()); + tracker.nonce_completed( + &replacement_hash, + &TxEvent::BlockInclusion, + TxFlow::Standard, + Instant::now(), + ); assert!(!tracker.nonce_summaries.contains(&slot)); } @@ -950,6 +991,87 @@ mod tests { assert!(event_log.fb_included); } + fn render_metrics(f: impl FnOnce()) -> String { + let recorder = metrics_exporter_prometheus::PrometheusBuilder::new().build_recorder(); + let handle = recorder.handle(); + metrics::with_local_recorder(&recorder, f); + handle.render() + } + + fn pending_event(tx_hash: TxHash) -> FullTransactionEvent { + FullTransactionEvent::Pending(tx_hash) + } + + #[rstest] + #[case::standard(TxFlow::Standard, "standard")] + #[case::validity(TxFlow::Validity, "validity")] + fn inclusion_metrics_are_labeled_by_flow(#[case] flow: TxFlow, #[case] label: &str) { + let rendered = render_metrics(|| { + let mut tracker = Tracker::new(false); + let tx_hash = TxHash::random(); + let nonce_slot = NonceSlot::new(Address::random(), 0); + tracker.handle_event(pending_event(tx_hash), Some(PooledTxInfo { nonce_slot, flow })); + + tracker.transaction_fb_included(tx_hash, Instant::now() + Duration::from_secs(2)); + tracker.transaction_completed( + tx_hash, + TxEvent::BlockInclusion, + Instant::now() + Duration::from_secs(4), + ); + }); + + for metric in [ + "reth_transaction_tracing_fb_slow_inclusions", + "reth_transaction_tracing_slow_inclusions", + "reth_transaction_tracing_fb_inclusion_duration_count", + "reth_transaction_tracing_inclusion_duration_count", + "reth_transaction_tracing_e2e_inclusion_duration_count", + ] { + assert!( + rendered.contains(&format!("{metric}{{flow=\"{label}\"}} 1")), + "expected {metric} with flow={label}, got: {rendered}" + ); + } + } + + #[test] + fn transactions_default_to_standard_flow() { + let mut tracker = Tracker::new(false); + let tx_hash = TxHash::random(); + tracker.handle_event(pending_event(tx_hash), None); + + assert_eq!( + tracker.txs.peek(&tx_hash).expect("tx should be tracked").flow, + TxFlow::Standard + ); + } + + #[rstest] + #[case::replacement_reported_first(true)] + #[case::replacement_reported_later(false)] + fn replacement_uses_its_own_flow(#[case] replacement_reported_first: bool) { + let mut tracker = Tracker::new(false); + let original = TxHash::random(); + let replacement = TxHash::random(); + let nonce_slot = NonceSlot::new(Address::random(), 0); + let validity = Some(PooledTxInfo { nonce_slot, flow: TxFlow::Validity }); + tracker.handle_event(pending_event(original), validity); + + let standard = Some(PooledTxInfo { nonce_slot, flow: TxFlow::Standard }); + if replacement_reported_first { + tracker.handle_event(pending_event(replacement), standard); + tracker.transaction_replaced(original, replacement); + } else { + tracker.transaction_replaced(original, replacement); + tracker.handle_event(pending_event(replacement), standard); + } + + assert_eq!( + tracker.txs.peek(&replacement).expect("replacement should be tracked").flow, + TxFlow::Standard + ); + } + #[tokio::test] async fn test_fb_inclusion() -> eyre::Result<()> { // Setup diff --git a/crates/execution/txpool/README.md b/crates/execution/txpool/README.md index 42c12c0876..5f0b3c6daa 100644 --- a/crates/execution/txpool/README.md +++ b/crates/execution/txpool/README.md @@ -134,6 +134,51 @@ at `ingress` and `builder`, independently of generic extension errors. sidecars to help operators assess migration readiness. RPC signature errors carry the same stable rejection reason in their `data` field. +### Validity pool + +`BaseTransactionPool` keeps three stores behind one `TransactionPool` surface: +the reth `Pool` for standard transactions, the EIP-8130 sidecar for finite +2D-nonce channels and nonce-free transactions, and a validity pool for every +transaction that carries validity predicates (conditional transactions), whatever +its envelope type. Validity transactions rest in the pool until a predicate on hot +state flips or their block or flashblock expiry passes, so the validity pool has +its own count, size and per-sender caps and evicts only among validity +transactions. A burst of conditional transactions therefore cannot push standard +user transactions out of reth's pending subpool, and reth's own eviction cannot +drop resting validity transactions. + +EIP-7702 delegation and validity predicates do not mix. The validator rejects a +transaction with validity predicates that is an EIP-7702 transaction or comes +from a delegated sender, and a state diff that changes an account's code drops +that account's validity transactions. The validity pool therefore never needs +reth's in-flight limits for delegated accounts and pending authorizations, which +apply only inside reth. + +`PoolStore::of` picks the store. A nonce slot `(sender, nonce key, nonce)`, or a +nonce-free replay id, is held by at most one store at a time: a newcomer bound for +another store replaces the incumbent under the usual price-bump rule and the +incumbent's listeners see `Replaced`. Account semantics span the stores: a +validity transaction right behind reth's or the sidecar's pending run is pending +(`LaneCursor::next_nonce` follows that run), while a standard transaction behind a +resting validity transaction stays queued in reth until the nonce gap closes. +`best_transactions` merges the three stores by priority but never yields a +sender's nonce `n + 1` before `n`: a validity head continuing a standard run waits +until the merge has yielded the transaction before it, and standard transactions +queued right behind a validity run are released by the merge once that run has +been yielded, so a predicate-delayed transaction does not keep its descendants out +of the same block. The lanes that straddle stores are indexed incrementally +(`TwoDNoncePool::lanes_ahead_of_state`, `shared_lanes_pending_end`), so building +the iterator costs those lanes plus each store's own snapshot, not the pool sizes. + +Capacity defaults to reth's pending subpool limit (`--txpool.pending-max-count`, +`--txpool.pending-max-size`, `--txpool.max-account-slots`) and is overridable with +`--rollup.validity-pool-max-count`, `--rollup.validity-pool-max-size` (megabytes) +and `--rollup.validity-pool-max-account-slots`; see `ValidityPoolLimits`. +Occupancy is published as `txpool.validity.pool_transactions` and +`txpool.validity.pool_bytes`, evictions as `txpool.validity.evicted`, and +replacements that crossed a store boundary as +`txpool.validity.cross_store_replaced{direction}`. + ## Usage Add the dependency to your `Cargo.toml`: diff --git a/crates/execution/txpool/src/best.rs b/crates/execution/txpool/src/best.rs index 69a72cc87f..5ecbb8b22f 100644 --- a/crates/execution/txpool/src/best.rs +++ b/crates/execution/txpool/src/best.rs @@ -1,63 +1,362 @@ -//! Merged best-transaction iteration across the protocol pool and 2D nonce sidecar. +//! Merged best-transaction iteration across the protocol pool, the EIP-8130 +//! sidecar and the validity pool. -use std::sync::Arc; +use std::{ + collections::{BinaryHeap, VecDeque}, + sync::Arc, +}; +use alloy_primitives::{ + U256, + map::{HashMap, HashSet}, +}; +use reth_primitives_traits::transaction::error::InvalidTransactionError; use reth_transaction_pool::{ BestTransactions, TransactionOrdering, ValidPoolTransaction, error::InvalidPoolTransactionError, }; -use crate::{BasePooledTx, BestTransactionPriority}; +use crate::{BasePooledTx, BestTransactionPriority, PoolStore, two_d_nonce_pool::LaneId}; + +/// A store's best-transactions iterator as the merge drives it. +/// +/// When the merge sets a lane's head aside, it asks the store to hold the rest +/// of that lane rather than pulling the whole lane into the merge one member +/// at a time. Iterators that cannot hold lanes keep the no-op defaults; the +/// merge then holds every member it pulls from such a lane itself. +pub(crate) trait MergeSource: + BestTransactions>> +{ + /// Stops offering members of `lane_id` until [`Self::release_lane`]. + fn hold_lane(&mut self, _lane_id: &LaneId) {} + + /// Offers the members of a held lane again, from where it stopped. + fn release_lane(&mut self, _lane_id: &LaneId) {} +} + +impl MergeSource + for Box>>> +{ +} + +type BoxedBest = Box>; + +/// Standard transactions a standard store holds queued directly behind a +/// validity lane's pending run, per lane and in nonce order. Their store never +/// yields them (it sees a nonce gap), so the merge releases them itself once +/// the validity run in front has been yielded. +pub(crate) type BridgedDescendants = HashMap>>>; + +/// A buffered transaction with its priority, computed once when it was buffered. +struct Head { + transaction: Arc>, + priority: BestTransactionPriority

, +} + +/// One store's iterator together with its buffered head. +struct Source { + best: BoxedBest, + head: Option>, +} + +/// A ready lane's front in [`HeldLanes::ready`]; the priority carries the +/// front's hash, which tells a live entry from one left behind by a pop or drop. +#[derive(PartialEq, Eq, PartialOrd, Ord)] +struct ReadyHead { + priority: BestTransactionPriority

, + lane_id: LaneId, +} + +/// Transactions held back per lane, in nonce order, until the merge has +/// yielded the slot right before each lane's front. +/// +/// Ready fronts sit in a heap by priority, so a [`MergeBestTransactions::pop_best`] +/// round looks at the best one in constant time and a pop costs a logarithm of +/// the ready lanes, however many lanes are held. +struct HeldLanes { + lanes: HashMap>>>, + ready: BinaryHeap>, +} + +impl Default for HeldLanes { + fn default() -> Self { + Self { lanes: HashMap::default(), ready: BinaryHeap::new() } + } +} + +impl HeldLanes { + const fn from_lanes(lanes: HashMap>>>) -> Self { + Self { lanes, ready: BinaryHeap::new() } + } + + fn holds(&self, lane_id: &LaneId) -> bool { + self.lanes.contains_key(lane_id) + } + + /// Appends `transaction` to its lane; callers push a lane in nonce order. + fn push(&mut self, lane_id: LaneId, transaction: Arc>) { + self.lanes.entry(lane_id).or_default().push_back(transaction); + } + + /// Marks the lane ready when its front directly follows the yielded `nonce`. + fn release_after( + &mut self, + lane_id: LaneId, + nonce: u64, + priority_of: impl Fn(&Arc>) -> BestTransactionPriority

, + ) { + let front = self + .lanes + .get(&lane_id) + .and_then(VecDeque::front) + .filter(|front| Some(front.nonce()) == nonce.checked_add(1)); + if let Some(front) = front { + self.ready.push(ReadyHead { priority: priority_of(front), lane_id }); + } + } + + /// The best ready front, discarding entries whose lane was popped or dropped since. + fn peek_ready(&mut self) -> Option<&BestTransactionPriority

> { + while let Some(top) = self.ready.peek() { + let live = self + .lanes + .get(&top.lane_id) + .and_then(VecDeque::front) + .is_some_and(|front| front.hash() == top.priority.hash()); + if live { + break; + } + self.ready.pop(); + } + self.ready.peek().map(|top| &top.priority) + } + + /// Takes the best ready front; the next member of its lane waits for it to + /// be yielded. Lanes left empty are removed. + fn pop_ready(&mut self) -> Option<(LaneId, Arc>)> { + self.peek_ready()?; + let lane_id = self.ready.pop()?.lane_id; + let lane = self.lanes.get_mut(&lane_id)?; + let front = lane.pop_front()?; + if lane.is_empty() { + self.lanes.remove(&lane_id); + } + Some((lane_id, front)) + } + + /// Removes every held member of `lane_id`; nothing behind an invalid + /// transaction executes. + fn drop_lane(&mut self, lane_id: &LaneId) -> Option>>> { + self.lanes.remove(lane_id) + } +} + +/// Where the best candidate of a [`MergeBestTransactions::pop_best`] round came from. +#[derive(Clone, Copy)] +enum Candidate { + Source(usize), + Deferred, + Bridged, +} -/// Merges best-transaction iterators from the protocol pool and the 2D nonce sidecar. +/// Merges the best-transaction iterators of every [`PoolStore`] into one +/// priority order under a single ordering. +/// +/// Each store yields a sender's transactions in nonce order, but a lane can +/// straddle stores: a validity lane may continue a run the protocol pool or +/// the sidecar holds the start of, and a standard store may hold queued +/// transactions right behind a validity run. The merge therefore releases a +/// cross-store successor only once it has yielded the slot right before it, +/// so the builder never sees nonce `n + 1` ahead of `n` however the two are +/// priced. A validity head that is not yet released is set aside in +/// `deferred`, its lane is held in the validity iterator, and that iterator +/// keeps advancing, so one sender's waiting lane never holds back the others. +/// +/// Invalidation is lane-wide across stores: the store owning the transaction +/// parks the rest of its lane, any other store's buffered or held member of +/// the lane is dropped and parked there too, and nothing pulled from the lane +/// afterwards is offered. pub(crate) struct MergeBestTransactions where O: TransactionOrdering, { - protocol: Box>>>, - sidecar: Box>>>, + sources: [Source; PoolStore::ALL.len()], ordering: O, base_fee: u64, - next_protocol: Option>>, - next_sidecar: Option>>, + /// Validity lanes riding another store's pending run, with the canonical + /// nonce each one is anchored at. Empty unless such lanes exist. + external_anchors: HashMap, + /// Validity lane members pulled from their iterator before their + /// cross-store predecessor was yielded. + deferred: HeldLanes, + /// Queued standard transactions behind validity runs; see [`BridgedDescendants`]. + bridged: HeldLanes, + /// The latest nonce yielded in each lane that straddles stores. A lane + /// yields in nonce order, so this one slot says which successor is next; + /// the entry goes when a member of the lane is marked invalid. + yielded: HashMap, + /// Lanes with a member marked invalid; nothing else pulled from them is offered. + invalid_lanes: HashSet, } impl MergeBestTransactions where O: TransactionOrdering, { - /// Creates a merged iterator from the protocol pool and 2D nonce sidecar. + /// Creates a merged iterator over the protocol pool, sidecar and validity + /// pool iterators, in [`PoolStore::ALL`] order. pub(crate) fn new( - protocol: Box>>>, - sidecar: Box>>>, + protocol: BoxedBest, + sidecar: BoxedBest, + validity: BoxedBest, ordering: O, base_fee: u64, ) -> Self { - Self { protocol, sidecar, ordering, base_fee, next_protocol: None, next_sidecar: None } + let sources = [protocol, sidecar, validity].map(|best| Source { best, head: None }); + Self { + sources, + ordering, + base_fee, + external_anchors: HashMap::default(), + deferred: HeldLanes::default(), + bridged: HeldLanes::default(), + yielded: HashMap::default(), + invalid_lanes: HashSet::default(), + } + } + + /// Registers the validity lanes whose executable cursor sits past their + /// canonical anchor (see `TwoDNoncePool::lanes_ahead_of_state`), so their + /// heads wait for the run in front of them to be yielded first. + pub(crate) fn with_external_anchors(mut self, anchors: HashMap) -> Self { + self.external_anchors = anchors; + self + } + + /// Registers queued standard transactions sitting right behind validity + /// runs, to be yielded once the run in front of them has been. + pub(crate) fn with_bridged_descendants(mut self, bridged: BridgedDescendants) -> Self { + self.bridged = HeldLanes::from_lanes(bridged); + self + } + + /// The nonce slot `transaction` occupies, with the protocol nonce as key + /// zero. Nonce-free transactions have no slot. + fn slot(transaction: &ValidPoolTransaction) -> Option<(LaneId, u64)> { + if transaction.transaction.is_eip8130_nonce_free() { + return None; + } + let nonce_key = transaction.transaction.eip8130_nonce_channel_key().unwrap_or(U256::ZERO); + Some(((transaction.sender(), nonce_key), transaction.nonce())) + } + + fn in_lane(transaction: &ValidPoolTransaction, lane_id: &LaneId) -> bool { + Self::slot(transaction).is_some_and(|(lane, _)| lane == *lane_id) + } + + /// The error a store is handed for a member whose lane predecessor was + /// marked invalid: the account's nonce stays behind it. Advisory only: + /// the stores park the lane whatever the kind, so `state` is approximate. + const fn predecessor_invalid(nonce: u64) -> InvalidPoolTransactionError { + InvalidPoolTransactionError::Consensus(InvalidTransactionError::NonceNotConsistent { + tx: nonce, + state: nonce.saturating_sub(1), + }) } - fn protocol_is_better( + fn priority( &self, - protocol: &Arc>, - sidecar: &Arc>, - ) -> bool { - let protocol_priority = - BestTransactionPriority::new(&self.ordering, protocol, self.base_fee); - let sidecar_priority = BestTransactionPriority::new(&self.ordering, sidecar, self.base_fee); - protocol_priority >= sidecar_priority + transaction: &Arc>, + ) -> BestTransactionPriority { + BestTransactionPriority::new(&self.ordering, transaction, self.base_fee) + } + + /// Whether a lane's bookkeeping crosses stores, i.e. yields in it must be remembered. + fn straddles(&self, lane_id: &LaneId) -> bool { + self.external_anchors.contains_key(lane_id) || self.bridged.holds(lane_id) + } + + /// Whether the validity transaction may be offered as soon as its iterator + /// yields it: either its lane is anchored at canonical state, so the lane's + /// own order suffices, or the slot right before it has already been yielded. + fn released(&self, lane_id: LaneId, nonce: u64) -> bool { + let Some(anchor) = self.external_anchors.get(&lane_id) else { return true }; + nonce <= *anchor + || nonce + .checked_sub(1) + .is_some_and(|predecessor| self.yielded.get(&lane_id) == Some(&predecessor)) + } + + /// Buffers the next transaction of every store, setting aside validity + /// transactions that are not released yet and dropping members of + /// invalidated lanes. + fn fill_heads(&mut self) { + for index in 0..self.sources.len() { + while self.sources[index].head.is_none() { + let Some(transaction) = self.sources[index].best.next() else { break }; + if let Some((lane_id, nonce)) = Self::slot(&transaction) { + if self.invalid_lanes.contains(&lane_id) { + let kind = Self::predecessor_invalid(nonce); + self.sources[index].best.mark_invalid(&transaction, kind); + continue; + } + if index == PoolStore::Validity.index() && !self.released(lane_id, nonce) { + self.deferred.push(lane_id, transaction); + self.sources[index].best.hold_lane(&lane_id); + continue; + } + } + let priority = self.priority(&transaction); + self.sources[index].head = Some(Head { transaction, priority }); + } + } + } + + fn record_yielded(&mut self, transaction: &ValidPoolTransaction) { + let Some((lane_id, nonce)) = Self::slot(transaction) else { return }; + if self.straddles(&lane_id) { + self.yielded.insert(lane_id, nonce); + } + let priority_of = |transaction: &Arc>| { + BestTransactionPriority::new(&self.ordering, transaction, self.base_fee) + }; + self.deferred.release_after(lane_id, nonce, priority_of); + self.bridged.release_after(lane_id, nonce, priority_of); } fn pop_best(&mut self) -> Option>> { - match (&self.next_protocol, &self.next_sidecar) { - (Some(protocol), Some(sidecar)) => { - if self.protocol_is_better(protocol, sidecar) { - self.next_protocol.take() - } else { - self.next_sidecar.take() + let mut best: Option<(Candidate, &BestTransactionPriority)> = None; + // Ties go to the earlier candidate, so a protocol transaction wins over + // an equally ranked sidecar, validity, deferred or bridged one. + let mut offer = |candidate, priority| { + if best.is_none_or(|(_, current)| priority > current) { + best = Some((candidate, priority)); + } + }; + for (index, source) in self.sources.iter().enumerate() { + if let Some(head) = &source.head { + offer(Candidate::Source(index), &head.priority); + } + } + if let Some(priority) = self.deferred.peek_ready() { + offer(Candidate::Deferred, priority); + } + if let Some(priority) = self.bridged.peek_ready() { + offer(Candidate::Bridged, priority); + } + match best?.0 { + Candidate::Source(index) => { + self.sources[index].head.take().map(|head| head.transaction) + } + Candidate::Deferred => { + let (lane_id, transaction) = self.deferred.pop_ready()?; + // The validity iterator held the rest of the lane while its + // head waited; with the head out, let it offer them again. + if !self.deferred.holds(&lane_id) { + self.sources[PoolStore::Validity.index()].best.release_lane(&lane_id); } + Some(transaction) } - (Some(_), None) => self.next_protocol.take(), - (None, Some(_)) => self.next_sidecar.take(), - (None, None) => None, + Candidate::Bridged => self.bridged.pop_ready().map(|(_, transaction)| transaction), } } } @@ -79,15 +378,11 @@ where fn next(&mut self) -> Option { loop { - if self.next_protocol.is_none() { - self.next_protocol = self.protocol.next(); - } - if self.next_sidecar.is_none() { - self.next_sidecar = self.sidecar.next(); - } + self.fill_heads(); let transaction = self.pop_best()?; if transaction.effective_tip_per_gas(self.base_fee).is_some() { + self.record_yielded(&transaction); return Some(transaction); } @@ -101,23 +396,44 @@ where O: TransactionOrdering, { fn mark_invalid(&mut self, transaction: &Self::Item, kind: InvalidPoolTransactionError) { - if transaction.transaction.is_eip8130_sidecar_transaction() { - self.next_sidecar = None; - self.sidecar.mark_invalid(transaction, kind); - } else { - self.next_protocol = None; - self.protocol.mark_invalid(transaction, kind); + let owner = PoolStore::of(&transaction.transaction).index(); + let Some((lane_id, _)) = Self::slot(transaction) else { + // A nonce-free transaction sequences nothing; only its own store hears of it. + self.sources[owner].best.mark_invalid(transaction, kind); + return; + }; + // Nothing behind a dropped transaction can execute, whichever store + // holds it: park the lane in its own store, drop and park the lane's + // members buffered from or held for other stores, and refuse anything + // the lane yields from now on. + self.invalid_lanes.insert(lane_id); + self.yielded.remove(&lane_id); + self.sources[owner].best.mark_invalid(transaction, kind); + for source in &mut self.sources { + if source.head.as_ref().is_some_and(|head| Self::in_lane(&head.transaction, &lane_id)) + && let Some(head) = source.head.take() + { + let kind = Self::predecessor_invalid(head.transaction.nonce()); + source.best.mark_invalid(&head.transaction, kind); + } + } + if let Some(front) = self.deferred.drop_lane(&lane_id).as_ref().and_then(VecDeque::front) { + let kind = Self::predecessor_invalid(front.nonce()); + self.sources[PoolStore::Validity.index()].best.mark_invalid(front, kind); } + self.bridged.drop_lane(&lane_id); } fn no_updates(&mut self) { - self.protocol.no_updates(); - self.sidecar.no_updates(); + for source in &mut self.sources { + source.best.no_updates(); + } } fn set_skip_blobs(&mut self, skip_blobs: bool) { - self.protocol.set_skip_blobs(skip_blobs); - self.sidecar.set_skip_blobs(skip_blobs); + for source in &mut self.sources { + source.best.set_skip_blobs(skip_blobs); + } } } @@ -132,12 +448,16 @@ mod tests { use alloy_signer_local::PrivateKeySigner; use base_common_chains::ChainConfig; use base_common_consensus::{ - BasePooledTransaction as ConsensusPooledTransaction, Eip8130Signed, TxEip8130, + BasePooledTransaction as ConsensusPooledTransaction, Eip8130Constants, Eip8130Signed, + TxEip8130, }; use reth_transaction_pool::{TransactionOrigin, identifier::TransactionId}; use super::*; - use crate::{BaseOrdering, BasePooledTransaction, TimestampedTransaction}; + use crate::{ + BaseOrdering, BasePooledTransaction, TimestampedTransaction, ValidityOperator, + ValidityPredicate, + }; #[derive(Debug)] struct StaticBestTransactions { @@ -158,6 +478,8 @@ mod tests { } } + impl MergeSource for StaticBestTransactions {} + impl BestTransactions for StaticBestTransactions { fn mark_invalid(&mut self, transaction: &Self::Item, _kind: InvalidPoolTransactionError) { let nonce_key = transaction.transaction.eip8130_nonce_channel_key(); @@ -265,6 +587,7 @@ mod tests { let mut merged = MergeBestTransactions::new( Box::new(StaticBestTransactions::new(vec![protocol])), Box::new(StaticBestTransactions::new(vec![sidecar])), + Box::new(StaticBestTransactions::new(Vec::new())), BaseOrdering::coinbase_tip(), 10, ); @@ -279,6 +602,7 @@ mod tests { let mut merged = MergeBestTransactions::new( Box::new(StaticBestTransactions::new(Vec::new())), Box::new(StaticBestTransactions::new(vec![sidecar])), + Box::new(StaticBestTransactions::new(Vec::new())), BaseOrdering::coinbase_tip(), 10, ); @@ -302,6 +626,7 @@ mod tests { let mut merged = MergeBestTransactions::new( Box::new(StaticBestTransactions::new(vec![Arc::clone(&older_protocol)])), Box::new(StaticBestTransactions::new(vec![newer_sidecar])), + Box::new(StaticBestTransactions::new(Vec::new())), BaseOrdering::timestamp(), 10, ); @@ -324,6 +649,7 @@ mod tests { let mut merged = MergeBestTransactions::new( Box::new(StaticBestTransactions::new(vec![older_protocol])), Box::new(StaticBestTransactions::new(vec![newer_sidecar])), + Box::new(StaticBestTransactions::new(Vec::new())), BaseOrdering::coinbase_tip(), 10, ); @@ -344,6 +670,7 @@ mod tests { let mut merged = MergeBestTransactions::new( Box::new(StaticBestTransactions::new(protocol)), Box::new(StaticBestTransactions::new(sidecar)), + Box::new(StaticBestTransactions::new(Vec::new())), BaseOrdering::coinbase_tip(), 0, ); @@ -359,4 +686,371 @@ mod tests { assert!(merged.next().is_none()); } + + fn resting_predicates() -> Vec { + vec![ValidityPredicate::BlockNumber { + op: ValidityOperator::LessThanOrEqual, + value: U256::from(1_000), + }] + } + + #[test] + fn merges_every_store_by_priority() { + let protocol = valid_pool_transaction(signed_tx(&signer(), U256::ZERO, 0, 100, 100)); + let sidecar = valid_pool_transaction(signed_tx(&signer(), U256::from(1), 0, 300, 300)); + let validity = valid_pool_transaction( + signed_tx(&signer(), U256::ZERO, 0, 200, 200) + .with_validity_predicates(resting_predicates()), + ); + let expected = [*sidecar.hash(), *validity.hash(), *protocol.hash()]; + + let merged = MergeBestTransactions::new( + Box::new(StaticBestTransactions::new(vec![protocol])), + Box::new(StaticBestTransactions::new(vec![sidecar])), + Box::new(StaticBestTransactions::new(vec![validity])), + BaseOrdering::coinbase_tip(), + 0, + ); + + let order: Vec<_> = merged.map(|transaction| *transaction.hash()).collect(); + assert_eq!(order, expected); + } + + #[test] + fn equal_priority_prefers_the_earlier_store() { + let protocol = valid_pool_transaction(signed_tx(&signer(), U256::ZERO, 0, 100, 100)); + let sidecar = valid_pool_transaction(signed_tx(&signer(), U256::from(1), 0, 100, 100)); + let validity = valid_pool_transaction( + signed_tx(&signer(), U256::ZERO, 0, 100, 100) + .with_validity_predicates(resting_predicates()), + ); + let expected = [*protocol.hash(), *sidecar.hash(), *validity.hash()]; + + let merged = MergeBestTransactions::new( + Box::new(StaticBestTransactions::new(vec![protocol])), + Box::new(StaticBestTransactions::new(vec![sidecar])), + Box::new(StaticBestTransactions::new(vec![validity])), + BaseOrdering::coinbase_tip(), + 0, + ); + + let order: Vec<_> = merged.map(|transaction| *transaction.hash()).collect(); + assert_eq!(order, expected); + } + + #[test] + fn validity_successor_waits_for_the_standard_predecessor_it_outbids() { + let signer = signer(); + let protocol = valid_pool_transaction(signed_tx(&signer, U256::ZERO, 0, 1, 1_000)); + let validity: Vec<_> = [(1, 900), (2, 800)] + .map(|(nonce, tip)| { + valid_pool_transaction( + signed_tx(&signer, U256::ZERO, nonce, tip, 1_000) + .with_validity_predicates(resting_predicates()), + ) + }) + .into(); + let expected = [*protocol.hash(), *validity[0].hash(), *validity[1].hash()]; + let lane = (signer.address(), U256::ZERO); + let build = |protocol, validity| { + MergeBestTransactions::new( + Box::new(StaticBestTransactions::new(protocol)), + Box::new(StaticBestTransactions::new(Vec::new())), + Box::new(StaticBestTransactions::new(validity)), + BaseOrdering::coinbase_tip(), + 0, + ) + .with_external_anchors(HashMap::from_iter([(lane, 0)])) + }; + + let merged = build(vec![Arc::clone(&protocol)], validity.clone()); + let order: Vec<_> = merged.map(|transaction| *transaction.hash()).collect(); + assert_eq!(order, expected); + + // The predecessor never shows up: the successors are never released. + let merged = build(Vec::new(), validity.clone()); + assert_eq!(merged.count(), 0); + + // The predecessor is dropped by the builder: likewise. + let mut merged = build(vec![protocol], validity); + let first = merged.next().expect("protocol predecessor"); + assert_eq!(first.nonce(), 0); + merged.mark_invalid(&first, InvalidPoolTransactionError::Underpriced); + assert_eq!(merged.count(), 0); + } + + #[test] + fn a_waiting_validity_head_does_not_hold_back_other_validity_lanes() { + let waiting = signer(); + let other = signer(); + let protocol = valid_pool_transaction(signed_tx(&waiting, U256::ZERO, 0, 1, 1_000)); + let waiting_validity = valid_pool_transaction( + signed_tx(&waiting, U256::ZERO, 1, 900, 1_000) + .with_validity_predicates(resting_predicates()), + ); + let other_validity = valid_pool_transaction( + signed_tx(&other, U256::ZERO, 0, 500, 1_000) + .with_validity_predicates(resting_predicates()), + ); + let build = |protocol| { + MergeBestTransactions::new( + Box::new(StaticBestTransactions::new(protocol)), + Box::new(StaticBestTransactions::new(Vec::new())), + Box::new(StaticBestTransactions::new(vec![ + Arc::clone(&waiting_validity), + Arc::clone(&other_validity), + ])), + BaseOrdering::coinbase_tip(), + 0, + ) + .with_external_anchors(HashMap::from_iter([((waiting.address(), U256::ZERO), 0)])) + }; + + // The waiting head is set aside; the other lane and the predecessor + // come out by priority, then the released successor. + let order: Vec<_> = + build(vec![Arc::clone(&protocol)]).map(|transaction| *transaction.hash()).collect(); + assert_eq!(order, [*other_validity.hash(), *protocol.hash(), *waiting_validity.hash()]); + + // Without its predecessor the waiting head alone is left out. + let order: Vec<_> = build(Vec::new()).map(|transaction| *transaction.hash()).collect(); + assert_eq!(order, [*other_validity.hash()]); + + // Dropping the predecessor after the fact drops the held successor too. + let mut merged = build(vec![protocol]); + let first = merged.next().expect("other lane"); + let second = merged.next().expect("predecessor"); + assert_eq!(second.nonce(), 0); + merged.mark_invalid(&second, InvalidPoolTransactionError::Underpriced); + assert_eq!(first.sender(), other.address()); + assert_eq!(merged.count(), 0); + } + + #[test] + fn channel_validity_successor_waits_for_the_sidecar_predecessor() { + let signer = signer(); + let channel = U256::from(3); + let sidecar = valid_pool_transaction(signed_tx(&signer, channel, 0, 1, 1_000)); + let validity = valid_pool_transaction( + signed_tx(&signer, channel, 1, 900, 1_000) + .with_validity_predicates(resting_predicates()), + ); + let expected = [*sidecar.hash(), *validity.hash()]; + + let merged = MergeBestTransactions::new( + Box::new(StaticBestTransactions::new(Vec::new())), + Box::new(StaticBestTransactions::new(vec![sidecar])), + Box::new(StaticBestTransactions::new(vec![validity])), + BaseOrdering::coinbase_tip(), + 0, + ) + .with_external_anchors(HashMap::from_iter([((signer.address(), channel), 0)])); + + let order: Vec<_> = merged.map(|transaction| *transaction.hash()).collect(); + assert_eq!(order, expected); + } + + #[test] + fn bridged_standard_descendants_follow_the_validity_run_they_wait_for() { + let signer = signer(); + let lane = (signer.address(), U256::ZERO); + let validity = valid_pool_transaction( + signed_tx(&signer, U256::ZERO, 0, 1, 1_000) + .with_validity_predicates(resting_predicates()), + ); + let descendants: Vec<_> = [(1, 900), (2, 950)] + .map(|(nonce, tip)| { + valid_pool_transaction(signed_tx(&signer, U256::ZERO, nonce, tip, 1_000)) + }) + .into(); + let expected = [*validity.hash(), *descendants[0].hash(), *descendants[1].hash()]; + let build = |validity: Vec<_>| { + MergeBestTransactions::new( + Box::new(StaticBestTransactions::new(Vec::new())), + Box::new(StaticBestTransactions::new(Vec::new())), + Box::new(StaticBestTransactions::new(validity)), + BaseOrdering::coinbase_tip(), + 0, + ) + .with_bridged_descendants(HashMap::from_iter([( + lane, + descendants.iter().cloned().collect::>(), + )])) + }; + + let order: Vec<_> = + build(vec![Arc::clone(&validity)]).map(|transaction| *transaction.hash()).collect(); + assert_eq!(order, expected); + + // Without the validity head nothing bridged is released. + assert_eq!(build(Vec::new()).count(), 0); + + // Dropping the validity head drops the bridged run with it. + let mut merged = build(vec![validity]); + let head = merged.next().expect("validity head"); + merged.mark_invalid(&head, InvalidPoolTransactionError::Underpriced); + assert_eq!(merged.count(), 0); + } + + #[test] + fn marking_a_nonce_free_transaction_invalid_keeps_another_nonce_free_head() { + let nonce_free = |tip| { + valid_pool_transaction(signed_tx( + &signer(), + Eip8130Constants::NONCE_KEY_MAX, + 0, + tip, + 1_000, + )) + }; + let (first, second) = (nonce_free(900), nonce_free(800)); + assert!(first.transaction.is_eip8130_nonce_free()); + let second_hash = *second.hash(); + let mut merged = MergeBestTransactions::new( + Box::new(StaticBestTransactions::new(Vec::new())), + Box::new(StaticBestTransactions::new(vec![first, second])), + Box::new(StaticBestTransactions::new(Vec::new())), + BaseOrdering::coinbase_tip(), + 0, + ); + let head = merged.next().expect("first nonce-free transaction"); + merged.mark_invalid(&head, InvalidPoolTransactionError::Underpriced); + assert_eq!(merged.next().map(|transaction| *transaction.hash()), Some(second_hash)); + } + + #[test] + fn invalidating_a_validity_transaction_parks_its_lane_in_every_store() { + // Validity lane (A, 0) holds nonces 0 and 1; a static protocol source + // offers nonce 2 of the same lane and a transaction in another lane. + let signer_a = signer(); + let other = PrivateKeySigner::random(); + let validity = vec![ + valid_pool_transaction( + signed_tx(&signer_a, U256::ZERO, 0, 500, 500) + .with_validity_predicates(resting_predicates()), + ), + valid_pool_transaction( + signed_tx(&signer_a, U256::ZERO, 1, 400, 400) + .with_validity_predicates(resting_predicates()), + ), + ]; + let protocol = vec![ + valid_pool_transaction(signed_tx(&signer_a, U256::ZERO, 2, 100, 100)), + valid_pool_transaction(signed_tx(&other, U256::ZERO, 0, 50, 50)), + ]; + let other_hash = *protocol[1].hash(); + + let mut merged = MergeBestTransactions::new( + Box::new(StaticBestTransactions::new(protocol)), + Box::new(StaticBestTransactions::new(Vec::new())), + Box::new(StaticBestTransactions::new(validity)), + BaseOrdering::coinbase_tip(), + 0, + ); + + let head = merged.next().expect("validity head"); + assert_eq!(head.nonce(), 0); + merged.mark_invalid(&head, InvalidPoolTransactionError::Underpriced); + + // Nothing behind nonce 0 executes, whichever store holds it; other lanes go on. + let rest: Vec<_> = merged.map(|transaction| *transaction.hash()).collect(); + assert_eq!(rest, vec![other_hash]); + } + + #[test] + fn invalidating_a_standard_predecessor_drops_the_buffered_validity_successor() { + // Protocol holds (A, 0) plus two other senders; validity holds (A, 1) + // and (A, 2) riding (A, 0), plus an unrelated lane. The priorities are + // arranged so (A, 1) is pulled into the validity head right after + // (A, 0) was yielded, then loses a round to a standard transaction. + let (a, b, c, d) = ( + signer(), + PrivateKeySigner::random(), + PrivateKeySigner::random(), + PrivateKeySigner::random(), + ); + let protocol = vec![ + valid_pool_transaction(signed_tx(&a, U256::ZERO, 0, 300, 300)), + valid_pool_transaction(signed_tx(&b, U256::ZERO, 0, 95, 95)), + valid_pool_transaction(signed_tx(&c, U256::ZERO, 0, 50, 50)), + ]; + let validity = vec![ + valid_pool_transaction( + signed_tx(&d, U256::ZERO, 0, 200, 200) + .with_validity_predicates(resting_predicates()), + ), + valid_pool_transaction( + signed_tx(&a, U256::ZERO, 1, 90, 90).with_validity_predicates(resting_predicates()), + ), + valid_pool_transaction( + signed_tx(&a, U256::ZERO, 2, 80, 80).with_validity_predicates(resting_predicates()), + ), + ]; + let (b0, c0, d0) = (*protocol[1].hash(), *protocol[2].hash(), *validity[0].hash()); + let anchors = HashMap::from_iter([((a.address(), U256::ZERO), 0)]); + + let mut merged = MergeBestTransactions::new( + Box::new(StaticBestTransactions::new(protocol)), + Box::new(StaticBestTransactions::new(Vec::new())), + Box::new(StaticBestTransactions::new(validity)), + BaseOrdering::coinbase_tip(), + 0, + ) + .with_external_anchors(anchors); + + let a0 = merged.next().expect("standard (A, 0)"); + assert_eq!((a0.sender(), a0.nonce()), (a.address(), 0)); + assert_eq!(merged.next().map(|tx| *tx.hash()), Some(d0)); + // This round pulls the released (A, 1) into the validity head, where + // it loses to (B, 0) and stays buffered. + assert_eq!(merged.next().map(|tx| *tx.hash()), Some(b0)); + + merged.mark_invalid(&a0, InvalidPoolTransactionError::Underpriced); + + let rest: Vec<_> = merged.map(|transaction| *transaction.hash()).collect(); + assert_eq!(rest, vec![c0]); + } + + #[test] + fn ready_held_lanes_are_offered_by_priority() { + // Two validity heads release two bridged standard children; the + // children are offered against each other and the remaining sources + // by priority, not by lane. + let (a, b, c) = (signer(), PrivateKeySigner::random(), PrivateKeySigner::random()); + let validity = vec![ + valid_pool_transaction( + signed_tx(&a, U256::ZERO, 0, 900, 900) + .with_validity_predicates(resting_predicates()), + ), + valid_pool_transaction( + signed_tx(&b, U256::ZERO, 0, 800, 800) + .with_validity_predicates(resting_predicates()), + ), + ]; + let child_a = valid_pool_transaction(signed_tx(&a, U256::ZERO, 1, 10, 10)); + let child_b = valid_pool_transaction(signed_tx(&b, U256::ZERO, 1, 30, 30)); + let standard_c = valid_pool_transaction(signed_tx(&c, U256::ZERO, 0, 20, 20)); + let expected = vec![ + *validity[0].hash(), + *validity[1].hash(), + *child_b.hash(), + *standard_c.hash(), + *child_a.hash(), + ]; + let bridged = BridgedDescendants::from_iter([ + ((a.address(), U256::ZERO), VecDeque::from([child_a])), + ((b.address(), U256::ZERO), VecDeque::from([child_b])), + ]); + + let merged = MergeBestTransactions::new( + Box::new(StaticBestTransactions::new(vec![standard_c])), + Box::new(StaticBestTransactions::new(Vec::new())), + Box::new(StaticBestTransactions::new(validity)), + BaseOrdering::coinbase_tip(), + 0, + ) + .with_bridged_descendants(bridged); + + assert_eq!(merged.map(|tx| *tx.hash()).collect::>(), expected); + } } diff --git a/crates/execution/txpool/src/block_expiry.rs b/crates/execution/txpool/src/block_expiry.rs index 5fe0557f83..28da509c3b 100644 --- a/crates/execution/txpool/src/block_expiry.rs +++ b/crates/execution/txpool/src/block_expiry.rs @@ -4,7 +4,12 @@ use std::collections::{BTreeMap, HashMap, HashSet}; use alloy_primitives::TxHash; -/// Coordinates successful flashblock publication with deadline eviction and admission. +/// Coordinates flashblock publication with deadline eviction and admission. +/// +/// The builder publishes and expires in one critical section. A node that follows the +/// builder's flashblock stream (a mempool or RPC node) only observes publication, so it +/// expires from the received position instead of waiting for the block to commit; until then +/// an expired transaction would still hold its same-nonce slot against an unbumped replacement. pub trait FlashblockExpiry: Clone + Send + Sync + 'static { /// Publishes while holding the admission lock, then evicts expired transactions. /// A failed publication must leave the pool unchanged. @@ -14,6 +19,11 @@ pub trait FlashblockExpiry: Clone + Send + Sync + 'static { flashblock_index: u64, publish: impl FnOnce() -> Result, ) -> Result<(R, usize), E>; + + /// Records a flashblock published elsewhere and evicts transactions whose deadline it + /// passed, returning how many were evicted. Positions at or behind the last recorded one + /// are ignored. + fn expire_published(&self, block_number: u64, flashblock_index: u64) -> usize; } /// An inclusive validity deadline, ordered by block number before flashblock index. diff --git a/crates/execution/txpool/src/builder/mod.rs b/crates/execution/txpool/src/builder/mod.rs index 9674290d2b..6de229f8c0 100644 --- a/crates/execution/txpool/src/builder/mod.rs +++ b/crates/execution/txpool/src/builder/mod.rs @@ -1,7 +1,10 @@ //! Builder RPC API server and associated metrics. mod rpc; -pub use rpc::{BuilderApiImpl, BuilderApiServer, InsertMetering, MAX_VALIDATED_INSERT_BATCH}; +pub use rpc::{ + BuilderApiImpl, BuilderApiServer, InsertMetering, MAX_VALIDATED_INSERT_BATCH, + is_already_imported_rejection, +}; mod metrics; pub use metrics::Metrics as BuilderApiMetrics; diff --git a/crates/execution/txpool/src/builder/rpc.rs b/crates/execution/txpool/src/builder/rpc.rs index bb6d475a3d..8d80d47b9b 100644 --- a/crates/execution/txpool/src/builder/rpc.rs +++ b/crates/execution/txpool/src/builder/rpc.rs @@ -264,8 +264,9 @@ where ErrorObjectOwned::owned(ErrorCode::InvalidParams.code(), error.to_string(), None::<()>) })?; // Every extension implementation must obey this builder's local rollout policy. + let has_validity_predicates = !pool_tx.validity_conditions().is_empty(); self.validity_signature_mode - .check(pool_tx.validity_predicates(), pool_tx.validity_signature().as_ref()) + .check(has_validity_predicates, pool_tx.validity_signature().as_ref()) .map_err(|error| { ValiditySignatureMetrics::rejected("builder", error.as_label()).increment(1); ErrorObjectOwned::owned( @@ -274,7 +275,7 @@ where Some(error.as_label()), ) })?; - if !pool_tx.validity_predicates().is_empty() { + if has_validity_predicates { ValiditySignatureMetrics::accepted( "builder", if pool_tx.validity_signature().is_some() { "signed" } else { "unsigned" }, @@ -335,6 +336,20 @@ where } } +/// End of the message a builder's insert RPC returns for a transaction its pool already holds: +/// the `Display` of [`PoolErrorKind::AlreadyImported`], which the rejection message ends with. +const ALREADY_IMPORTED: &str = "already imported"; + +/// Returns `true` if `error` is a builder insert rejection for a transaction the builder's pool +/// already holds, which delivered it as surely as an accepted insert. +/// +/// The insert RPC reports every pool rejection with the same code, so this matches the end of the +/// message. Tests pin [`ALREADY_IMPORTED`] to the pool error's `Display` and run a real rejection +/// through this check, so a change on either side fails them. +pub fn is_already_imported_rejection(error: &ErrorObjectOwned) -> bool { + error.message().ends_with(ALREADY_IMPORTED) +} + impl BuilderApiImpl { fn emit_validated_insert_event( &self, @@ -369,7 +384,10 @@ mod tests { use alloy_signer_local::PrivateKeySigner; use base_bundles::MeterBundleResponse; use base_common_consensus::{BaseTransactionSigned, BaseTypedTransaction, TxDeposit}; - use reth_transaction_pool::noop::NoopTransactionPool; + use reth_transaction_pool::{ + error::{PoolError, PoolErrorKind}, + noop::NoopTransactionPool, + }; use super::*; use crate::{ @@ -863,4 +881,22 @@ mod tests { // This error code should be InternalError assert_eq!(err.code(), ErrorCode::InternalError.code()); } + + #[test] + fn already_imported_matches_the_pool_error_display() { + assert_eq!(ALREADY_IMPORTED, PoolErrorKind::AlreadyImported.to_string()); + } + + /// A duplicate's rejection is recognised as already imported; any other rejection is not. + #[test] + fn insert_rejections_are_classified_by_is_already_imported_rejection() { + let handler = BuilderApiImpl::new(NoopTransactionPool::::new()); + let hash = TxHash::repeat_byte(0x11); + let reject = |kind| handler.finish_insert(hash, 0, None, Err(PoolError::new(hash, kind))); + + let duplicate = reject(PoolErrorKind::AlreadyImported).unwrap_err(); + assert!(is_already_imported_rejection(&duplicate), "{duplicate:?}"); + let underpriced = reject(PoolErrorKind::ReplacementUnderpriced).unwrap_err(); + assert!(!is_already_imported_rejection(&underpriced), "{underpriced:?}"); + } } diff --git a/crates/execution/txpool/src/conditions.rs b/crates/execution/txpool/src/conditions.rs index 8782b42c35..1bafcd28b1 100644 --- a/crates/execution/txpool/src/conditions.rs +++ b/crates/execution/txpool/src/conditions.rs @@ -127,25 +127,31 @@ impl ValidityBounds { /// Conditions compiled at ingress for repeated pool and builder checks. /// -/// Timing comparisons are reduced to bounds. State predicates are partitioned -/// by type; timing always gates state reads. The submitted count is retained -/// because transaction ordering uses it, even when redundant bounds collapse. +/// Timing comparisons are reduced to bounds. Every submitted predicate is moved +/// into exactly one partition, so the pool holds each predicate once; timing +/// always gates state reads. The submitted count is retained because transaction +/// ordering uses it, even when redundant bounds collapse. #[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct ValidityConditions { block: ValidityBounds, flashblock: ValidityBounds, + /// Block and flashblock predicates as submitted, in wire order. + /// + /// Bounds normalization is lossy for the wire form the sender signed, so these + /// are kept verbatim for forwarding and never used for evaluation. + timing: Vec, balance: Vec, nonce: Vec, storage: Vec, - /// Original wire batch, used only when forwarding to a builder. - submitted: Vec, } impl ValidityConditions { - /// Compiles submitted predicates without modifying the wire representation. + /// Compiles submitted predicates, moving each into its partition without cloning. pub fn new(predicates: Vec) -> Self { - // Count partitions before cloning large predicates so larger batches do not - // repeatedly grow buffers, and small partitions do not retain minimum Vec capacity. + // Count partitions first so each backing vector is allocated exactly once at its + // final size: larger batches do not repeatedly grow buffers, and small or empty + // partitions do not retain minimum Vec capacity. + let mut timing_count = 0; let mut balance_count = 0; let mut nonce_count = 0; let mut storage_count = 0; @@ -156,32 +162,40 @@ impl ValidityConditions { ValidityPredicate::Balance { .. } => balance_count += 1, ValidityPredicate::Nonce { .. } => nonce_count += 1, ValidityPredicate::Storage { .. } => storage_count += 1, - ValidityPredicate::BlockNumber { op: ValidityOperator::NotEqual, .. } => { - block_exclusions += 1; + ValidityPredicate::BlockNumber { op, .. } => { + timing_count += 1; + if *op == ValidityOperator::NotEqual { + block_exclusions += 1; + } } - ValidityPredicate::FlashblockIndex { op: ValidityOperator::NotEqual, .. } => { - flashblock_exclusions += 1; + ValidityPredicate::FlashblockIndex { op, .. } => { + timing_count += 1; + if *op == ValidityOperator::NotEqual { + flashblock_exclusions += 1; + } } - _ => {} } } - let mut conditions = Self { submitted: predicates, ..Self::default() }; + let mut conditions = Self::default(); + conditions.timing.reserve_exact(timing_count); conditions.balance.reserve_exact(balance_count); conditions.nonce.reserve_exact(nonce_count); conditions.storage.reserve_exact(storage_count); conditions.block.excluded.reserve_exact(block_exclusions); conditions.flashblock.excluded.reserve_exact(flashblock_exclusions); - for predicate in &conditions.submitted { - match predicate { + for predicate in predicates { + match &predicate { ValidityPredicate::BlockNumber { op, value } => { - conditions.block.insert(*op, *value) + conditions.block.insert(*op, *value); + conditions.timing.push(predicate); } ValidityPredicate::FlashblockIndex { op, value } => { - conditions.flashblock.insert(*op, *value) + conditions.flashblock.insert(*op, *value); + conditions.timing.push(predicate); } - ValidityPredicate::Balance { .. } => conditions.balance.push(predicate.clone()), - ValidityPredicate::Nonce { .. } => conditions.nonce.push(predicate.clone()), - ValidityPredicate::Storage { .. } => conditions.storage.push(predicate.clone()), + ValidityPredicate::Balance { .. } => conditions.balance.push(predicate), + ValidityPredicate::Nonce { .. } => conditions.nonce.push(predicate), + ValidityPredicate::Storage { .. } => conditions.storage.push(predicate), } } conditions @@ -193,9 +207,15 @@ impl ValidityConditions { EMPTY.get_or_init(Self::default) } - /// Original predicates for lossless builder forwarding; never used for evaluation. - pub const fn submitted(&self) -> &[ValidityPredicate] { - self.submitted.as_slice() + /// Every submitted predicate, re-combined for builder forwarding; never used for + /// evaluation. + /// + /// Yields timing predicates in wire order, then balance, nonce, and storage + /// partitions. State predicates therefore leave in a different order than they + /// arrived, which the EIP-712 authorization tolerates because it sorts predicates + /// by struct hash before hashing. + pub fn submitted(&self) -> impl Iterator { + self.timing.iter().chain(self.state_predicates()) } /// Block-number bounds. @@ -210,12 +230,12 @@ impl ValidityConditions { /// Number of submitted predicates, before normalization. pub const fn len(&self) -> usize { - self.submitted.len() + self.timing.len() + self.balance.len() + self.nonce.len() + self.storage.len() } /// Whether the transaction carries no validity conditions. pub const fn is_empty(&self) -> bool { - self.submitted.is_empty() + self.len() == 0 } /// State predicates in balance, nonce, then storage order. @@ -272,10 +292,10 @@ impl ValidityConditions { pub const fn heap_size(&self) -> usize { self.block.heap_size() + self.flashblock.heap_size() - + (self.balance.capacity() + + (self.timing.capacity() + + self.balance.capacity() + self.nonce.capacity() - + self.storage.capacity() - + self.submitted.capacity()) + + self.storage.capacity()) * core::mem::size_of::() } } @@ -409,10 +429,10 @@ mod tests { #[test] fn heap_size_counts_spare_capacity_in_every_backing_vector() { let mut conditions = ValidityConditions { + timing: Vec::with_capacity(11), balance: Vec::with_capacity(3), nonce: Vec::with_capacity(5), storage: Vec::with_capacity(7), - submitted: Vec::with_capacity(11), ..Default::default() }; conditions.block.excluded.reserve_exact(4); @@ -423,15 +443,26 @@ mod tests { conditions.heap_size(), 26 * core::mem::size_of::() + 10 * core::mem::size_of::() ); + } - let mut submitted = Vec::with_capacity(13); - submitted.push(ValidityPredicate::BlockNumber { - op: ValidityOperator::GreaterThan, + #[test] + fn compiled_conditions_hold_each_predicate_once_without_submitted_spare_capacity() { + let block = + ValidityPredicate::BlockNumber { op: ValidityOperator::GreaterThan, value: U256::ZERO }; + let balance = ValidityPredicate::Balance { + address: Address::ZERO, + op: ValidityOperator::Equal, value: U256::ZERO, - }); + }; + let mut submitted = Vec::with_capacity(13); + submitted.extend([balance.clone(), block.clone(), balance.clone()]); let conditions = ValidityConditions::new(submitted); - assert_eq!(conditions.heap_size(), 13 * core::mem::size_of::()); - assert_eq!(conditions.len(), 1); + assert_eq!(conditions.len(), 3); + assert_eq!(conditions.heap_size(), 3 * core::mem::size_of::()); + assert_eq!( + conditions.submitted().cloned().collect::>(), + [block, balance.clone(), balance] + ); } #[test] @@ -494,7 +525,11 @@ mod tests { op: ValidityOperator::Equal, value: U256::from(1), })); - assert_eq!(conditions.submitted(), raw); + // Forwarding regroups state predicates after the timing predicates. + assert_eq!( + conditions.submitted().cloned().collect::>(), + [block, flashblock, balance, nonce, storage] + ); assert_eq!(conditions.len(), raw.len()); } @@ -597,7 +632,7 @@ mod tests { }, ]; let conditions = ValidityConditions::new(raw.clone()); - assert_eq!(conditions.submitted(), raw); + assert_eq!(conditions.submitted().cloned().collect::>(), raw); assert_eq!(conditions.block().min(), Some(U256::from(10))); assert_eq!(conditions.block().max(), Some(U256::from(20))); assert_eq!(conditions.len(), raw.len()); diff --git a/crates/execution/txpool/src/lib.rs b/crates/execution/txpool/src/lib.rs index 0585840272..1436a659be 100644 --- a/crates/execution/txpool/src/lib.rs +++ b/crates/execution/txpool/src/lib.rs @@ -65,8 +65,8 @@ pub use ordering::{ mod parking; pub use parking::{ - BestTransactionLane, BestTransactionLaneState, ParkableBestTransactions, - ParkableTransactionPool, ParkedBestTransactions, + BestTransactionLane, BestTransactionLaneState, BestTransactionsHold, HOLD_INTERRUPT_INTERVAL, + ParkableBestTransactions, ParkableTransactionPool, ParkedBestTransactions, }; mod pool; @@ -82,7 +82,8 @@ pub use pool_error_label::PoolRejectionLabel; mod builder; pub use builder::{ - BuilderApiImpl, BuilderApiMetrics, BuilderApiServer, InsertMetering, MAX_VALIDATED_INSERT_BATCH, + BuilderApiImpl, BuilderApiMetrics, BuilderApiServer, InsertMetering, + MAX_VALIDATED_INSERT_BATCH, is_already_imported_rejection, }; mod wire; @@ -91,6 +92,13 @@ pub use wire::{ }; mod two_d_nonce_pool; +pub use two_d_nonce_pool::SidecarSenderIds; + +mod store; +pub use store::PoolStore; + +mod validity_limits; +pub use validity_limits::ValidityPoolLimits; mod metrics; pub use metrics::{GuardMetrics, ValidatorMetrics, ValidityPoolMetrics, ValiditySignatureMetrics}; diff --git a/crates/execution/txpool/src/metrics.rs b/crates/execution/txpool/src/metrics.rs index adb727c150..e779064a53 100644 --- a/crates/execution/txpool/src/metrics.rs +++ b/crates/execution/txpool/src/metrics.rs @@ -100,6 +100,17 @@ base_metrics::define_metrics! { #[describe("Validity transactions admitted to the pool, labeled by whether the admission replaced an existing same-sender/nonce transaction or added a new pool entry")] #[label(name = "outcome", default = ["added", "replaced"])] admitted: counter, + #[describe("Validity predicates carried by each validity transaction admitted to the pool")] + predicates_per_tx: histogram, + #[describe("Validity transactions evicted by the validity pool's own count or size cap")] + evicted: counter, + #[describe("Same-slot replacements that crossed between the validity pool and a standard store, labeled by the direction a validity transaction moved")] + #[label(name = "direction", default = ["into_validity", "out_of_validity"])] + cross_store_replaced: counter, + #[describe("Transactions currently held by the validity pool")] + pool_transactions: gauge, + #[describe("Summed size of transactions currently held by the validity pool, in bytes")] + pool_bytes: gauge, } impl ValidityPoolMetrics { @@ -108,8 +119,32 @@ impl ValidityPoolMetrics { /// a fee bump or cancellation) from a net-new pool entry. Together the two /// series decompose lane churn: their sum is total validity admissions and /// `replaced` over that sum is the replacement rate. - pub fn record_admission(replaced: bool) { + /// + /// Also records how many validity predicates the admitted transaction carries. + pub fn record_admission(replaced: bool, predicates: usize) { Self::admitted(if replaced { "replaced" } else { "added" }).increment(1); + Self::predicates_per_tx().record(predicates as f64); + } + + /// Records validity transactions evicted by the validity pool's capacity cap. + pub fn record_evictions(count: usize) { + if count > 0 { + Self::evicted().increment(count as u64); + } + } + + /// Records a same-slot replacement whose old and new transaction live in + /// different stores. `into_validity` means a validity transaction replaced + /// a standard one; `out_of_validity` the reverse. + pub fn record_cross_store_replacement(into_validity: bool) { + Self::cross_store_replaced(if into_validity { "into_validity" } else { "out_of_validity" }) + .increment(1); + } + + /// Publishes the validity pool's current occupancy. + pub fn record_occupancy(transactions: usize, bytes: usize) { + Self::pool_transactions().set(transactions as f64); + Self::pool_bytes().set(bytes as f64); } } @@ -172,18 +207,32 @@ mod tests { } #[test] - fn record_admission_splits_added_and_replaced() { + fn record_admission_splits_added_and_replaced_and_counts_predicates() { let recorder = DebuggingRecorder::new(); let snapshotter = recorder.snapshotter(); metrics::with_local_recorder(&recorder, || { - ValidityPoolMetrics::record_admission(false); - ValidityPoolMetrics::record_admission(false); - ValidityPoolMetrics::record_admission(true); + ValidityPoolMetrics::record_admission(false, 1); + ValidityPoolMetrics::record_admission(false, 2); + ValidityPoolMetrics::record_admission(true, 6); }); // `snapshot()` drains, so materialize once and query the vec. let snapshot = snapshotter.snapshot().into_vec(); assert_eq!(admitted_count(&snapshot, "added"), Some(2)); assert_eq!(admitted_count(&snapshot, "replaced"), Some(1)); + + let predicates = snapshot + .iter() + .find_map(|(ck, _, _, value)| match value { + DebugValue::Histogram(values) + if ck.kind() == MetricKind::Histogram + && ck.key().name() == "txpool.validity.predicates_per_tx" => + { + Some(values.iter().map(|value| value.into_inner()).collect::>()) + } + _ => None, + }) + .expect("predicates_per_tx histogram recorded"); + assert_eq!(predicates, vec![1.0, 2.0, 6.0]); } } diff --git a/crates/execution/txpool/src/ordering.rs b/crates/execution/txpool/src/ordering.rs index 9534bd9a11..fb3770d918 100644 --- a/crates/execution/txpool/src/ordering.rs +++ b/crates/execution/txpool/src/ordering.rs @@ -45,6 +45,11 @@ impl BestTransactionPriority

{ hash: *transaction.hash(), } } + + /// The hash of the transaction this key was computed for. + pub const fn hash(&self) -> &TxHash { + &self.hash + } } /// Transaction ordering strategy for the pool. diff --git a/crates/execution/txpool/src/parking.rs b/crates/execution/txpool/src/parking.rs index dc90f1f656..57b686fe95 100644 --- a/crates/execution/txpool/src/parking.rs +++ b/crates/execution/txpool/src/parking.rs @@ -3,6 +3,7 @@ use std::{ collections::{BinaryHeap, VecDeque}, sync::Arc, + time::{Duration, Instant}, }; use alloy_primitives::{ @@ -76,6 +77,37 @@ where fn mark_committed(&mut self, transaction: &Arc>); } +/// Decides which validity candidates a parking iterator parks before yielding them. +/// +/// The build loop parks a validity transaction once it finds one of its predicates unsatisfied, +/// and the pool re-yields that transaction in every later iterator until a state change wakes it. +/// A hold recognises such transactions at the source, so they are parked exactly as if the build +/// loop had yielded and parked them, without the build loop seeing them: their lane stays +/// occupied and [`ParkableBestTransactions::promote`] returns them to priority competition. +pub trait BestTransactionsHold: Send + Sync { + /// Returns whether `transaction` is parked instead of yielded. Called once per source candidate + /// that carries validity predicates and is not blocked by its lane. + fn hold(&mut self, transaction: &ValidPoolTransaction) -> bool; + + /// Records the time one `next` call spent from its first hold until it found a candidate, + /// which includes advancing past further held transactions. Calls that hold nothing record + /// nothing. + fn record_duration(&mut self, duration: Duration); + + /// Returns whether the build driving the iterator has stopped selecting transactions. + /// + /// One `next` call can advance past many held transactions without returning to the build + /// loop, so the iterator checks this every [`HOLD_INTERRUPT_INTERVAL`] holds and returns + /// early once it is true. The default never interrupts. + fn is_interrupted(&self) -> bool { + false + } +} + +/// Number of held transactions a parking iterator advances past between checks of +/// [`BestTransactionsHold::is_interrupted`], so the check stays off the per-transaction path. +pub const HOLD_INTERRUPT_INTERVAL: u32 = 256; + /// A transaction pool that can create lane-aware parkable best iterators. pub trait ParkableTransactionPool: TransactionPool where @@ -85,14 +117,16 @@ where fn best_transactions_with_attributes_and_parking( &self, attributes: BestTransactionsAttributes, + hold: Option>>, ) -> Box>; } /// Lane-aware parking over an existing best-transactions iterator. /// /// The inner iterator remains responsible for nonce contiguity and source ordering. This adapter -/// only buffers descendants that the inner iterator unlocks while an earlier member of their lane -/// is parked or waiting for an execution outcome. +/// buffers descendants that the inner iterator unlocks while an earlier member of their lane is +/// parked or waiting for an execution outcome, and parks the candidates a +/// [`BestTransactionsHold`] holds before they are yielded. pub struct ParkedBestTransactions where T: BasePooledTx, @@ -107,6 +141,7 @@ where parked: HashMap>, Option)>, ready: HashMap>, Option)>, ready_heap: BinaryHeap<(BestTransactionPriority, TxHash)>, + hold: Option>>, } impl std::fmt::Debug for ParkedBestTransactions @@ -120,6 +155,7 @@ where .field("lanes", &self.lanes.len()) .field("parked", &self.parked.len()) .field("ready", &self.ready.len()) + .field("hold", &self.hold.is_some()) .finish_non_exhaustive() } } @@ -141,9 +177,16 @@ where parked: HashMap::default(), ready: HashMap::default(), ready_heap: BinaryHeap::new(), + hold: None, } } + /// Parks the candidates `hold` holds before they are yielded. + pub fn with_hold(mut self, hold: Option>>) -> Self { + self.hold = hold; + self + } + /// Returns a complete priority key for a transaction. pub fn priority( &self, @@ -211,30 +254,62 @@ where }) } - /// Pulls through blocked descendants until the next source candidate is lane-eligible. + /// Pulls through blocked descendants and held candidates until the next source candidate is + /// lane-eligible. /// /// Consuming occupied-lane descendants is the intentionally naive BASE-253 implementation: /// the source only exposes other lanes after these transactions are pulled and buffered. The /// `Invalid` arm is defensive because terminal invalidation is already forwarded to `inner`, /// whose [`BestTransactions`] contract excludes the invalid transaction's descendants. + /// + /// Every [`HOLD_INTERRUPT_INTERVAL`] holds it stops early, without a source head, once the + /// hold reports its build interrupted. `next` then yields a promoted transaction or `None`, + /// and the build loop observes the same interruption and stops. pub fn fill_source_head(&mut self) { + let mut hold_start = None; + let mut held = 0_u32; while self.source_head.is_none() { let Some(transaction) = self.inner.next() else { - return; + break; }; - let Some(lane) = BestTransactionLane::for_transaction(&transaction) else { - self.source_head = Some((transaction, None)); - return; + let lane = BestTransactionLane::for_transaction(&transaction); + // One lookup serves both the lane check and occupying the lane of a held head. + let vacant_lane = match lane.map(|lane| self.lanes.entry(lane)) { + Some(Entry::Occupied(mut entry)) => match entry.get_mut() { + BestTransactionLaneState::Invalid => continue, + BestTransactionLaneState::Occupied(buffered) => { + buffered.push_back(transaction); + continue; + } + }, + Some(Entry::Vacant(vacant)) => Some(vacant), + None => None, }; - match self.lanes.get_mut(&lane) { - Some(BestTransactionLaneState::Invalid) => continue, - Some(BestTransactionLaneState::Occupied(buffered)) => { - buffered.push_back(transaction); - continue; + if let Some(hold) = self.hold.as_mut() + && !transaction.transaction.validity_conditions().is_empty() + && hold.hold(&transaction) + { + // The clock starts at the first hold, so candidates that hold nothing pay no + // timer read. + hold_start.get_or_insert_with(Instant::now); + // Parked as if yielded and parked: the lane is occupied so descendants are + // buffered until the held head is promoted and committed. + if let Some(vacant) = vacant_lane { + vacant.insert(BestTransactionLaneState::Occupied(VecDeque::new())); } - None => {} + self.parked.insert(*transaction.hash(), (transaction, lane)); + held += 1; + if held.is_multiple_of(HOLD_INTERRUPT_INTERVAL) && hold.is_interrupted() { + break; + } + continue; } - self.source_head = Some((transaction, Some(lane))); + self.source_head = Some((transaction, lane)); + } + if let Some(start) = hold_start + && let Some(hold) = self.hold.as_mut() + { + hold.record_duration(start.elapsed()); } } @@ -389,6 +464,7 @@ where fn best_transactions_with_attributes_and_parking( &self, attributes: BestTransactionsAttributes, + _hold: Option>>, ) -> Box> { Box::new(ParkedBestTransactions::new( std::iter::empty(), @@ -411,11 +487,12 @@ mod tests { BasePooledTransaction as ConsensusPooledTransaction, Eip8130Constants, Eip8130Signed, TxEip8130, }; + use parking_lot::Mutex; use reth_primitives_traits::Recovered; use reth_transaction_pool::{TransactionOrigin, identifier::TransactionId}; use super::*; - use crate::{BaseOrdering, BasePooledTransaction}; + use crate::{BaseOrdering, BasePooledTransaction, ValidityOperator, ValidityPredicate}; #[derive(Debug)] struct StaticBestTransactions { @@ -454,6 +531,16 @@ mod tests { nonce_key: U256, nonce: u64, priority_fee: u128, + ) -> Arc> { + validity_transaction(signer, nonce_key, nonce, priority_fee, Vec::new()) + } + + fn validity_transaction( + signer: &PrivateKeySigner, + nonce_key: U256, + nonce: u64, + priority_fee: u128, + predicates: Vec, ) -> Arc> { let tx = TxEip8130 { chain_id: ChainConfig::mainnet().chain_id, @@ -477,7 +564,8 @@ mod tests { let transaction = BasePooledTransaction::new( Recovered::new_unchecked(pooled.into(), signer.address()), encoded_length, - ); + ) + .with_validity_predicates(predicates); Arc::new(ValidPoolTransaction { transaction_id: TransactionId::new(0u64.into(), nonce), transaction, @@ -647,4 +735,135 @@ mod tests { assert_eq!(best.next().unwrap().hash(), other_parked.hash()); assert!(best.next().is_none()); } + + /// Holds every validity transaction whose hash is in `held`, recording each check. + struct HashSetHold { + held: Vec, + checked: Vec, + durations: usize, + } + + impl BestTransactionsHold for Arc> { + fn hold(&mut self, transaction: &ValidPoolTransaction) -> bool { + let mut hold = self.lock(); + hold.checked.push(*transaction.hash()); + hold.held.contains(transaction.hash()) + } + + fn record_duration(&mut self, _duration: Duration) { + self.lock().durations += 1; + } + } + + fn balance_predicate() -> ValidityPredicate { + ValidityPredicate::Balance { + address: Address::repeat_byte(0xaa), + op: ValidityOperator::GreaterThanOrEqual, + value: U256::ONE, + } + } + + #[test] + fn held_transaction_is_parked_with_its_lane_until_promoted() { + let held_signer = PrivateKeySigner::random(); + let other_signer = PrivateKeySigner::random(); + let held = + validity_transaction(&held_signer, U256::ZERO, 0, 100, vec![balance_predicate()]); + let child = transaction(&held_signer, U256::ZERO, 1, 90); + let other = transaction(&other_signer, U256::ZERO, 0, 50); + let hold = Arc::new(Mutex::new(HashSetHold { + held: vec![*held.hash()], + checked: Vec::new(), + durations: 0, + })); + let inner = StaticBestTransactions::new(vec![ + Arc::clone(&held), + Arc::clone(&child), + Arc::clone(&other), + ]); + let mut best = ParkedBestTransactions::new(inner, BaseOrdering::coinbase_tip(), 0) + .with_hold(Some(Box::new(Arc::clone(&hold)))); + + // The held head and its descendant are not yielded; the other lane is. + assert_eq!(best.next().unwrap().hash(), other.hash()); + best.mark_committed(&other); + assert!(best.next().is_none()); + assert_eq!(hold.lock().checked, vec![*held.hash()], "plain transactions are not checked"); + assert_eq!(hold.lock().durations, 1); + + // Promotion returns the held head, and committing it releases its descendant. + assert!(best.promote(*held.hash())); + assert_eq!(best.next().unwrap().hash(), held.hash()); + best.mark_committed(&held); + assert_eq!(best.next().unwrap().hash(), child.hash()); + assert!(best.next().is_none()); + } + + #[test] + fn unheld_validity_transaction_is_yielded_without_recording_hold_work() { + let signer = PrivateKeySigner::random(); + let candidate = + validity_transaction(&signer, U256::ZERO, 0, 100, vec![balance_predicate()]); + let hold = Arc::new(Mutex::new(HashSetHold { + held: Vec::new(), + checked: Vec::new(), + durations: 0, + })); + let inner = StaticBestTransactions::new(vec![Arc::clone(&candidate)]); + let mut best = ParkedBestTransactions::new(inner, BaseOrdering::coinbase_tip(), 0) + .with_hold(Some(Box::new(Arc::clone(&hold)))); + + assert_eq!(best.next().unwrap().hash(), candidate.hash()); + assert_eq!(hold.lock().checked, vec![*candidate.hash()]); + assert_eq!(hold.lock().durations, 0); + } + + /// Holds every validity transaction until `interrupted` is cleared, counting holds. + struct InterruptibleHold { + interrupted: bool, + held: usize, + } + + impl BestTransactionsHold for Arc> { + fn hold(&mut self, _transaction: &ValidPoolTransaction) -> bool { + self.lock().held += 1; + true + } + + fn record_duration(&mut self, _duration: Duration) {} + + fn is_interrupted(&self) -> bool { + self.lock().interrupted + } + } + + #[test] + fn interrupted_hold_stops_advancing_past_held_transactions() { + let held_count = 2 * HOLD_INTERRUPT_INTERVAL as usize + 1; + let mut transactions = (0..held_count) + .map(|_| { + let signer = PrivateKeySigner::random(); + validity_transaction(&signer, U256::ZERO, 0, 100, vec![balance_predicate()]) + }) + .collect::>(); + let plain = transaction(&PrivateKeySigner::random(), U256::ZERO, 0, 50); + transactions.push(Arc::clone(&plain)); + let hold = Arc::new(Mutex::new(InterruptibleHold { interrupted: true, held: 0 })); + let mut best = ParkedBestTransactions::new( + StaticBestTransactions::new(transactions), + BaseOrdering::coinbase_tip(), + 0, + ) + .with_hold(Some(Box::new(Arc::clone(&hold)))); + + // An interrupted build gets control back after one check interval of holds. + assert!(best.next().is_none()); + assert_eq!(hold.lock().held, HOLD_INTERRUPT_INTERVAL as usize); + + // Nothing is lost: an uninterrupted call resumes the walk where it stopped. + hold.lock().interrupted = false; + assert_eq!(best.next().unwrap().hash(), plain.hash()); + assert_eq!(hold.lock().held, held_count); + assert_eq!(best.parked.len(), held_count); + } } diff --git a/crates/execution/txpool/src/pool.rs b/crates/execution/txpool/src/pool.rs index a983a4a105..1f195e7eef 100644 --- a/crates/execution/txpool/src/pool.rs +++ b/crates/execution/txpool/src/pool.rs @@ -1,6 +1,11 @@ -//! Base transaction-pool wrapper that combines the protocol pool with a 2D nonce sidecar. +//! Base transaction-pool wrapper that combines the reth protocol pool with an +//! EIP-8130 2D nonce sidecar and a separate validity pool. -use std::{collections::HashMap, fmt, sync::Arc}; +use std::{ + collections::{HashMap, VecDeque}, + fmt, + sync::Arc, +}; use alloy_eips::{ eip4844::{BlobAndProofV1, BlobAndProofV2, BlobCellsAndProofsV1}, @@ -15,23 +20,27 @@ use reth_primitives_traits::Recovered; use reth_transaction_pool::{ AddedTransactionOutcome, AllPoolTransactions, AllTransactionsEvents, BestTransactions, BestTransactionsAttributes, BlobStore, BlobStoreError, BlockInfo, FullTransactionEvent, - GetPooledTransactionLimit, NewBlobSidecar, NewTransactionEvent, Pool, PoolResult, PoolSize, - PoolTransaction, PoolUpdateKind, PropagatedTransactions, SubPool, TransactionEvents, + GetPooledTransactionLimit, NewBlobSidecar, NewTransactionEvent, Pool, PoolConfig, PoolResult, + PoolSize, PoolTransaction, PoolUpdateKind, PropagatedTransactions, SubPool, TransactionEvents, TransactionListenerKind, TransactionOrigin, TransactionPool, TransactionPoolExt, TransactionValidationOutcome, TransactionValidationTaskExecutor, TransactionValidator, ValidPoolTransaction, - pool::{AddedTransactionState, TransactionEvent}, + error::{PoolError, PoolErrorKind}, + pool::{AddedTransactionState, QueuedReason, TransactionEvent}, +}; +use tokio::{ + spawn, + sync::mpsc::{self, error::TrySendError}, }; -use tokio::{spawn, sync::mpsc}; use tracing::debug; use crate::{ - Admission, BasePooledTx, BaseTransactionValidator, FlashblockExpiry, GuardLimits, GuardMetrics, - InvalidationCause, InvalidationKey, LimitRejection, MempoolGuard, ParkableBestTransactions, - ParkableTransactionPool, ParkedBestTransactions, PredicateContext, StateDiffInvalidation, - ValidityPoolMetrics, - best::MergeBestTransactions, - two_d_nonce_pool::{InsertOutcome, TwoDNoncePool}, + Admission, BasePooledTx, BaseTransactionValidator, BestTransactionsHold, FlashblockExpiry, + GuardLimits, GuardMetrics, InvalidationCause, InvalidationKey, LimitRejection, MempoolGuard, + ParkableBestTransactions, ParkableTransactionPool, ParkedBestTransactions, PoolStore, + PredicateContext, StateDiffInvalidation, ValidityPoolLimits, ValidityPoolMetrics, + best::{BridgedDescendants, MergeBestTransactions}, + two_d_nonce_pool::{InsertOutcome, LaneCursor, LaneId, LaneUpdate, TwoDNoncePool}, }; const SIDE_CAR_EVENT_CHANNEL_SIZE: usize = 1024; @@ -73,7 +82,49 @@ impl AccountStateDiff { } } -/// Wrapper around reth's transaction pool that adds a 2D nonce sidecar for EIP-8130 channels. +/// A lane store ([`TwoDNoncePool`]) together with the listeners following its +/// transactions. Lock order within a store is pool, then listeners, then the +/// guard; across stores the sidecar is locked before the validity pool. +struct LaneStore { + pool: Arc>>, + listeners: Arc>>, +} + +impl LaneStore { + fn new(pool: TwoDNoncePool) -> Self { + Self { + pool: Arc::new(RwLock::new(pool)), + listeners: Arc::new(RwLock::new(SidecarListeners::default())), + } + } +} + +impl Clone for LaneStore { + fn clone(&self) -> Self { + Self { pool: Arc::clone(&self.pool), listeners: Arc::clone(&self.listeners) } + } +} + +/// A transaction occupying the same `(sender, nonce key, nonce)` slot as a +/// newcomer, but held by a different store than the one the newcomer targets. +struct CrossStoreReplacement { + store: PoolStore, + transaction: Arc>, +} + +/// How [`BaseTransactionPool::remove_from_stores`] treats what it removes. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum RemoveMode { + /// Remove exactly the named transactions; the protocol pool parks their descendants. + Exact, + /// Remove the named transactions and every later one from the same sender. + WithDescendants, + /// The named transactions were mined: lane cursors advance past them. + Mined, +} + +/// Wrapper around reth's transaction pool that adds an EIP-8130 2D nonce +/// sidecar and a separate pool for transactions carrying validity predicates. pub struct BaseTransactionPool< Client, S, @@ -89,8 +140,11 @@ pub struct BaseTransactionPool< protocol_pool: Pool>, O, S>, ordering: O, - nonce_pool: Arc>>, - listeners: Arc>>, + /// Standard EIP-8130 transactions on finite nonce channels or nonce-free. + sidecar: LaneStore, + /// Every transaction carrying validity predicates, in lanes keyed by + /// `(sender, nonce key)` with the protocol nonce as key zero. + validity: LaneStore, /// Shared admission and invalidation ledger for EIP-8130 transactions. guard: Arc>, /// Block and flashblock deadline index for validity transactions, including @@ -131,8 +185,8 @@ where Self { protocol_pool: self.protocol_pool.clone(), ordering: self.ordering.clone(), - nonce_pool: Arc::clone(&self.nonce_pool), - listeners: Arc::clone(&self.listeners), + sidecar: self.sidecar.clone(), + validity: self.validity.clone(), guard: Arc::clone(&self.guard), block_expiry: Arc::clone(&self.block_expiry), published_flashblock: Arc::clone(&self.published_flashblock), @@ -161,7 +215,9 @@ where O: reth_transaction_pool::TransactionOrdering + Clone, S: BlobStore + Clone, { - /// Creates a new wrapper around the reth protocol pool. + /// Creates a new wrapper around the reth protocol pool. The validity pool + /// starts with [`ValidityPoolLimits::inherit`]; see + /// [`Self::with_validity_pool_limits`]. pub fn new( protocol_pool: Pool< TransactionValidationTaskExecutor>, @@ -170,12 +226,16 @@ where >, ordering: O, ) -> Self { - let price_bump_config = protocol_pool.config().price_bumps; + let config = protocol_pool.config(); + // The sidecar keeps this release's uncapped behaviour; only the + // validity pool is bounded. + let sidecar = TwoDNoncePool::new(config.price_bumps); + let validity = Self::validity_pool(config, ValidityPoolLimits::inherit()); Self { protocol_pool, ordering, - nonce_pool: Arc::new(RwLock::new(TwoDNoncePool::new(price_bump_config))), - listeners: Arc::new(RwLock::new(SidecarListeners::default())), + sidecar: LaneStore::new(sidecar), + validity: LaneStore::new(validity), guard: Arc::new(RwLock::new(MempoolGuard::unlimited())), block_expiry: Arc::new(RwLock::new(crate::BlockExpiryIndex::new())), published_flashblock: Arc::new(RwLock::new(None)), @@ -183,6 +243,12 @@ where } } + fn validity_pool(config: &PoolConfig, limits: ValidityPoolLimits) -> TwoDNoncePool { + TwoDNoncePool::new(config.price_bumps) + .with_limits(limits.resolve(config)) + .with_protocol_nonce_lanes() + } + /// Configures EIP-8130 admission limits. Call before sharing the pool. #[must_use] pub fn with_guard_limits(self, limits: GuardLimits) -> Self { @@ -190,6 +256,13 @@ where self } + /// Configures the validity pool's capacity. Call before sharing the pool. + #[must_use] + pub fn with_validity_pool_limits(self, limits: ValidityPoolLimits) -> Self { + *self.validity.pool.write() = Self::validity_pool(self.protocol_pool.config(), limits); + self + } + /// Builds guard admission metadata carried by a validated EIP-8130 transaction. pub fn admission_for(transaction: &T) -> Option { transaction.as_eip8130()?; @@ -224,14 +297,37 @@ where self.protocol_pool.validator() } - fn is_sidecar_transaction(&self, transaction: &T) -> bool { - transaction.is_eip8130_sidecar_transaction() + /// Returns the number of transactions and their summed size held by the + /// validity pool. + pub fn validity_pool_occupancy(&self) -> (usize, usize) { + let validity = self.validity.pool.read(); + (validity.len(), validity.size()) } - fn limit_rejection_error( - hash: TxHash, - rejection: LimitRejection, - ) -> reth_transaction_pool::error::PoolError { + /// The lane store behind `store`, or `None` for the protocol pool. + const fn lane_store(&self, store: PoolStore) -> Option<&LaneStore> { + match store { + PoolStore::Protocol => None, + PoolStore::Sidecar => Some(&self.sidecar), + PoolStore::Validity => Some(&self.validity), + } + } + + /// Both lane stores, in lock order. + const fn lane_stores(&self) -> [&LaneStore; 2] { + [&self.sidecar, &self.validity] + } + + /// The lane a non-nonce-free transaction is sequenced in: its EIP-8130 + /// channel, or nonce key zero for the account's protocol nonce. + fn lane_id_of(transaction: &T) -> Option { + if transaction.is_eip8130_nonce_free() { + return None; + } + Some((transaction.sender(), transaction.eip8130_nonce_channel_key().unwrap_or(U256::ZERO))) + } + + fn limit_rejection_error(hash: TxHash, rejection: LimitRejection) -> PoolError { GuardMetrics::admission_rejected(GuardMetrics::rejection_reason(rejection)).increment(1); let reason = match rejection { LimitRejection::SenderLimit => "sender EIP-8130 signature limit reached", @@ -240,7 +336,7 @@ where LimitRejection::PayerBalance => "payer cannot fund another EIP-8130 transaction", }; debug!(reason = GuardMetrics::rejection_reason(rejection), "EIP-8130 admission rejected"); - reth_transaction_pool::error::PoolError::other(hash, reason) + PoolError::other(hash, reason) } /// Applies canonical state changes and evicts affected EIP-8130 transactions. @@ -256,6 +352,18 @@ where for diff in diffs { diff.push_exact_keys(&mut exact); } + // An account's code changes when it gains or loses an EIP-7702 + // delegation. Delegated senders may not hold validity transactions, so a + // sender delegated after admission (for example through an authorization + // that was still pending then) loses them here. + let delegated = { + let validity = self.validity.pool.read(); + diffs + .iter() + .filter(|diff| diff.code_changed) + .flat_map(|diff| validity.sender_transaction_hashes(&diff.address)) + .collect::>() + }; let dropped = { let mut guard = self.guard.write(); let mut dropped = guard.invalidate_exact(exact); @@ -264,9 +372,10 @@ where dropped.extend(guard.on_balance_changed(diff.address, balance)); } } + dropped.extend(delegated); dropped }; - let removed = self.remove_dropped_across_pools(dropped); + let removed = self.remove_from_stores(dropped, RemoveMode::Exact); GuardMetrics::record_state_diff_invalidations(removed.len()); if !removed.is_empty() { debug!(count = removed.len(), "EIP-8130 transactions invalidated by state diff"); @@ -284,34 +393,232 @@ where let _admission_guard = self.protocol_admission_lock.lock(); self.validator().validator().clear_limit_class_cache(); let dropped = self.guard.write().invalidate_all(); - let removed = self.remove_dropped_across_pools(dropped); + let removed = self.remove_from_stores(dropped, RemoveMode::Exact); GuardMetrics::record_bulk_invalidations(removed.len(), cause); removed } - fn remove_dropped_across_pools( + /// Publishes a lane update of `store` and frees the guard capacity its + /// discarded transactions held. Callers hold the store's pool lock so the + /// events stay ordered with the lane change. + fn publish_lane_update(&self, store: &LaneStore, update: &LaneUpdate) { + if update.discarded.is_empty() && update.promoted.is_empty() && update.queued.is_empty() { + return; + } + store.listeners.write().on_lane_update(update); + self.release_from_guard(&update.discarded); + } + + /// Re-anchors validity lanes after a standard store or canonical state + /// moved. Each entry names a lane and, when known, the canonical nonce it + /// now sits at; the executable cursor then follows the standard store's + /// pending run from that anchor. Returns the transactions discarded for + /// falling below a cursor. + /// + /// Anchors only move forward unless `authoritative_account_nonces` is set: + /// then the reported account nonces come from canonical state that may + /// have moved back (a reorg, or reth's resync of drifted accounts), and a + /// protocol-nonce lane anchors at the highest nonce reported for it, as + /// reth does for the sender's standard transactions. Channel lanes and + /// pre-canonical (flashblock) nonces always move forward only. + /// + /// Runs under `protocol_admission_lock`: the target is derived from the + /// standard stores' state without holding the validity lock, so a + /// concurrent admission anchoring the same lane could otherwise be + /// overwritten by a stale cursor and lose its transaction. + fn sync_validity_lanes( + &self, + lanes: impl IntoIterator)>, + authoritative_account_nonces: bool, + ) -> Vec>> { + debug_assert!( + self.protocol_admission_lock.is_locked(), + "validity lanes are re-anchored under the admission lock" + ); + let mut targets: HashMap> = HashMap::new(); + for (lane_id, nonce) in lanes { + let entry = targets.entry(lane_id).or_default(); + *entry = (*entry).max(nonce); + } + let mut discarded = Vec::new(); + for (lane_id, nonce) in targets { + let (cursor, shared) = { + let validity = self.validity.pool.read(); + let Some(cursor) = validity.lane_cursor(&lane_id) else { continue }; + (cursor, validity.is_shared_lane(&lane_id)) + }; + // Every change to a lane's standard run passes through here, so + // this is where a shared flag is dropped once the standard store + // holds nothing in the lane; the next admission in it re-flags. + if shared && !self.standard_store_holds_lane(lane_id) { + self.validity.pool.write().unmark_shared_lane(&lane_id); + } + let state_nonce = match nonce { + Some(nonce) if authoritative_account_nonces && lane_id.1.is_zero() => nonce, + Some(nonce) => nonce.max(cursor.state_nonce), + None => cursor.state_nonce, + }; + let target = LaneCursor { + state_nonce, + next_nonce: self.standard_pending_end(lane_id, state_nonce), + }; + if target == cursor { + continue; + } + let mut validity = self.validity.pool.write(); + let update = validity.set_lane_cursor(lane_id, target); + self.publish_lane_update(&self.validity, &update); + drop(validity); + discarded.extend(update.discarded); + } + discarded + } + + /// The nonce right after the standard store's consecutive pending run that + /// starts at `from` in `lane_id`, or `from` when no such run starts there. + /// Validity transactions behind that run are executable; those behind a gap + /// or a parked standard transaction are not. + fn standard_pending_end(&self, lane_id: LaneId, from: u64) -> u64 { + let (sender, nonce_key) = lane_id; + if nonce_key.is_zero() { + if self.protocol_pool.get_transaction_by_sender_and_nonce(sender, from).is_none() { + return from; + } + let Some(head) = + self.protocol_pool.get_highest_consecutive_transaction_by_sender(sender, from) + else { + return from; + }; + // The run is executable only if its head is pending: a parked head + // means a parked ancestor, a fee or balance shortfall, or a gap. + let head_nonce = head.nonce(); + if self + .protocol_pool + .get_pending_transaction_by_sender_and_nonce(sender, head_nonce) + .is_some() + { + head_nonce.saturating_add(1).max(from) + } else { + from + } + } else { + let sidecar = self.sidecar.pool.read(); + if sidecar.lane_transaction(&lane_id, from).is_none() { + return from; + } + sidecar.lane_pending_end(&lane_id).map_or(from, |end| end.max(from)) + } + } + + /// Splits `hashes` by the store currently holding each, in [`PoolStore::ALL`] + /// order. Unknown hashes are attributed to the protocol pool. + fn partition_hashes_by_store(&self, hashes: Vec) -> [Vec; 3] { + let sidecar = self.sidecar.pool.read(); + let validity = self.validity.pool.read(); + let mut partitioned: [Vec; 3] = Default::default(); + for hash in hashes { + let store = if sidecar.contains(&hash) { + PoolStore::Sidecar + } else if validity.contains(&hash) { + PoolStore::Validity + } else { + PoolStore::Protocol + }; + partitioned[store.index()].push(hash); + } + partitioned + } + + /// Removes `hashes` from whichever store holds each, publishes the matching + /// listener events, frees guard capacity and re-anchors the validity lanes + /// whose standard run changed. Returns everything removed, including + /// validity transactions discarded by a lane re-anchor. + fn remove_from_stores( &self, - dropped: Vec, + hashes: Vec, + mode: RemoveMode, ) -> Vec>> { - if dropped.is_empty() { + if hashes.is_empty() { return Vec::new(); } - let (protocol_hashes, sidecar_hashes) = self.partition_hashes_by_pool(dropped); + let [protocol_hashes, sidecar_hashes, validity_hashes] = + self.partition_hashes_by_store(hashes); let mut removed = if protocol_hashes.is_empty() { Vec::new() } else { - self.protocol_pool.remove_transactions(protocol_hashes) - }; - if !sidecar_hashes.is_empty() { - let sidecar_removed = self.nonce_pool.write().remove_transactions(&sidecar_hashes); - if !sidecar_removed.is_empty() { - self.listeners.write().on_discarded(&sidecar_removed); + match mode { + RemoveMode::Exact => self.protocol_pool.remove_transactions(protocol_hashes), + RemoveMode::WithDescendants => { + self.protocol_pool.remove_transactions_and_descendants(protocol_hashes) + } + RemoveMode::Mined => self.protocol_pool.prune_transactions(protocol_hashes), } - removed.extend(sidecar_removed); + }; + // A mined transaction proves its lane's canonical nonce moved past it; + // any other removal leaves the anchor alone and only re-checks the run. + let anchor = |transaction: &Arc>| { + (mode == RemoveMode::Mined).then(|| transaction.nonce().saturating_add(1)) + }; + let mut touched_lanes: Vec<(LaneId, Option)> = removed + .iter() + .map(|transaction| ((transaction.sender(), U256::ZERO), anchor(transaction))) + .collect(); + let sidecar_removed = self.remove_from_lane_store(&self.sidecar, &sidecar_hashes, mode); + touched_lanes.extend(sidecar_removed.iter().filter_map(|transaction| { + Self::lane_id_of(&transaction.transaction).map(|lane_id| (lane_id, anchor(transaction))) + })); + removed.extend(sidecar_removed); + let validity_removed = self.remove_from_lane_store(&self.validity, &validity_hashes, mode); + if mode == RemoveMode::Mined { + // A mined validity transaction advanced its own lane's anchor; the + // cursor must now also follow whatever standard run begins there. + touched_lanes.extend(validity_removed.iter().filter_map(|transaction| { + Self::lane_id_of(&transaction.transaction) + .map(|lane_id| (lane_id, anchor(transaction))) + })); } + removed.extend(validity_removed); + self.release_from_guard(&removed); + removed.extend(self.sync_validity_lanes(touched_lanes, false)); removed } + /// Removes `hashes` from one lane store and publishes the listener events + /// for what changed. Lane-cursor discards are included in the result. + fn remove_from_lane_store( + &self, + store: &LaneStore, + hashes: &[TxHash], + mode: RemoveMode, + ) -> Vec>> { + if hashes.is_empty() { + return Vec::new(); + } + let mut pool = store.pool.write(); + match mode { + RemoveMode::Exact => { + let removed = pool.remove_transactions(hashes); + if !removed.is_empty() { + store.listeners.write().on_discarded(&removed); + } + removed + } + RemoveMode::WithDescendants => { + let removed = pool.remove_transactions_and_descendants(hashes); + if !removed.is_empty() { + store.listeners.write().on_discarded(&removed); + } + removed + } + RemoveMode::Mined => { + let mut pruned = pool.prune_mined(hashes); + store.listeners.write().on_lane_update(&pruned.lanes); + pruned.removed.append(&mut pruned.lanes.discarded); + pruned.removed + } + } + } + fn release_from_guard(&self, removed: &[Arc>]) { if removed.is_empty() { return; @@ -339,7 +646,7 @@ where let horizon = now / InvalidationKey::EXPIRY_BUCKET_SECS; let (dropped, bucket_count) = self.guard.write().invalidate_expiry_buckets_through(horizon); GuardMetrics::expiry_buckets_fired().increment(bucket_count as u64); - let removed = self.remove_dropped_across_pools(dropped); + let removed = self.remove_from_stores(dropped, RemoveMode::Exact); GuardMetrics::record_expiry_invalidations(removed.len()); if !removed.is_empty() { debug!(count = removed.len(), "EIP-8130 transactions invalidated by expiry"); @@ -362,7 +669,7 @@ where } /// Rejects a late arrival whose deadline is behind the published position. - /// Called while holding `protocol_admission_lock`, including on the sidecar path. + /// Called while holding `protocol_admission_lock`, on every admission path. fn ensure_not_published_expired( &self, hash: TxHash, @@ -375,7 +682,7 @@ where flashblock_index: position.flashblock_index.saturating_add(1), }; if transaction.transaction().validity_conditions().is_expired(&next) { - return Err(reth_transaction_pool::error::PoolError::other( + return Err(PoolError::other( hash, "validity deadline expired at published flashblock", )); @@ -387,6 +694,20 @@ where /// Records a transaction's last valid block in the block-expiry index, /// dropping any replaced transaction's stale entry in the same pass so a /// fee-bump replacement does not orphan the old hash until its expiry block. + /// Drops the block-expiry entries of transactions that left the pool by + /// another path (eviction), so the index stays bounded by the live pool. + /// Takes the index lock only when there is something to drop; call it with + /// no store lock held. + fn remove_block_expiry(&self, removed: &[Arc>]) { + if removed.is_empty() { + return; + } + let mut block_expiry = self.block_expiry.write(); + for transaction in removed { + block_expiry.remove(transaction.hash()); + } + } + fn register_block_expiry( &self, hash: TxHash, @@ -416,12 +737,11 @@ where fn expire_by_block(&self, committed_block: u64) { let _admission_guard = self.protocol_admission_lock.lock(); let expired = self.block_expiry.write().drain_expired(committed_block); - let removed = self.remove_dropped_across_pools(expired); - // Release any guard slots directly rather than deferring to the - // reconciliation sweep: an EIP-8130 transaction may carry block_number - // validity predicates, so an evicted tx can also hold guard capacity. - // `release` is a no-op for the common EIP-1559 case that is not tracked. - self.release_from_guard(&removed); + // `remove_from_stores` also frees any guard slot directly rather than + // deferring to the reconciliation sweep: an EIP-8130 transaction may + // carry block_number validity predicates, so an evicted tx can also + // hold guard capacity. + let removed = self.remove_from_stores(expired, RemoveMode::Exact); if !removed.is_empty() { GuardMetrics::record_block_expiry_invalidations(removed.len()); debug!( @@ -444,8 +764,7 @@ where *published = Some(position); drop(published); let expired = self.block_expiry.write().drain_published(block_number, flashblock_index); - let removed = self.remove_dropped_across_pools(expired); - self.release_from_guard(&removed); + let removed = self.remove_from_stores(expired, RemoveMode::Exact); GuardMetrics::record_block_expiry_invalidations(removed.len()); if !removed.is_empty() { debug!( @@ -458,26 +777,12 @@ where removed.len() } - /// Returns whether a validated transaction carries validity predicates, for - /// lane-churn accounting. Invalid outcomes carry no pooled transaction and - /// are never counted. - fn has_validity_predicates(validated: &TransactionValidationOutcome) -> bool { - validated - .as_valid_transaction() - .is_some_and(|transaction| !transaction.transaction().validity_conditions().is_empty()) - } - fn reconcile_guard(&self) { let tracked = self.guard.read().tracked_hashes(); if tracked.is_empty() { return; } - let nonce_pool = self.nonce_pool.read(); - let stale: Vec<_> = tracked - .into_iter() - .filter(|hash| self.protocol_pool.get(hash).is_none() && !nonce_pool.contains(hash)) - .collect(); - drop(nonce_pool); + let stale = self.untracked_in_every_store(tracked); if stale.is_empty() { return; } @@ -486,12 +791,7 @@ where // initial full scan stays outside this critical section, while a // reserved-before-insert transaction cannot be mistaken for an orphan. let _admission_guard = self.protocol_admission_lock.lock(); - let nonce_pool = self.nonce_pool.read(); - let stale: Vec<_> = stale - .into_iter() - .filter(|hash| self.protocol_pool.get(hash).is_none() && !nonce_pool.contains(hash)) - .collect(); - drop(nonce_pool); + let stale = self.untracked_in_every_store(stale); if stale.is_empty() { return; } @@ -503,19 +803,104 @@ where GuardMetrics::record_reconcile_releases(stale.len()); } + /// Filters `hashes` down to those no store holds. + fn untracked_in_every_store(&self, hashes: Vec) -> Vec { + let sidecar = self.sidecar.pool.read(); + let validity = self.validity.pool.read(); + hashes + .into_iter() + .filter(|hash| { + self.protocol_pool.get(hash).is_none() + && !sidecar.contains(hash) + && !validity.contains(hash) + }) + .collect() + } + fn protocol_replacement_hash(&self, sender: Address, nonce: u64) -> Option { self.protocol_pool - .get_transactions_by_sender(sender) - .into_iter() - .find(|existing| existing.nonce() == nonce) + .get_transaction_by_sender_and_nonce(sender, nonce) .map(|existing| *existing.hash()) } - fn stale_classification_error(hash: TxHash) -> reth_transaction_pool::error::PoolError { - reth_transaction_pool::error::PoolError::other( - hash, - "EIP-8130 admission classification changed during validation", - ) + /// Finds the transaction another store holds in `transaction`'s slot, so a + /// newcomer bound for `target` replaces it under the usual fee-bump rule + /// instead of coexisting with it. The protocol pool and validity pool share + /// protocol-nonce slots; the sidecar and validity pool share channel slots + /// and nonce-free replay ids. + fn cross_store_replacement( + &self, + target: PoolStore, + transaction: &T, + ) -> PoolResult>> { + let sender = transaction.sender(); + let nonce = transaction.nonce(); + let found = match target { + PoolStore::Protocol => self + .validity + .pool + .read() + .lane_transaction(&(sender, U256::ZERO), nonce) + .map(|existing| (PoolStore::Validity, existing)), + PoolStore::Sidecar => Self::lane_slot(&self.validity.pool.read(), transaction) + .map(|existing| (PoolStore::Validity, existing)), + PoolStore::Validity => { + if transaction.is_eip8130_sidecar_transaction() { + Self::lane_slot(&self.sidecar.pool.read(), transaction) + .map(|existing| (PoolStore::Sidecar, existing)) + } else { + self.protocol_pool + .get_transaction_by_sender_and_nonce(sender, nonce) + .map(|existing| (PoolStore::Protocol, existing)) + } + } + }; + let Some((store, existing)) = found else { + return Ok(None); + }; + let hash = *transaction.hash(); + if *existing.hash() == hash { + return Err(PoolError::new(hash, PoolErrorKind::AlreadyImported)); + } + let price_bumps = self.protocol_pool.config().price_bumps; + if existing.transaction.is_replacement_underpriced(transaction, &price_bumps) { + return Err(PoolError::new(hash, PoolErrorKind::ReplacementUnderpriced)); + } + Ok(Some(CrossStoreReplacement { store, transaction: existing })) + } + + /// The transaction `pool` holds in `transaction`'s slot: the same replay id + /// for a nonce-free transaction, otherwise the same channel and nonce. + fn lane_slot(pool: &TwoDNoncePool, transaction: &T) -> Option>> { + if let Some(replay_id) = transaction.eip8130_replay_id() { + return pool.nonce_free_transaction(&replay_id); + } + let nonce_key = transaction.eip8130_nonce_channel_key()?; + pool.lane_transaction(&(transaction.sender(), nonce_key), transaction.nonce()) + } + + /// Removes the transaction `replaced_by` superseded from the store that + /// held it. Lane stores announce the replacement; the protocol pool + /// announces a discard and parks the replaced transaction's descendants + /// behind the newcomer. + fn remove_cross_store_replaced(&self, cross: CrossStoreReplacement, replaced_by: TxHash) { + let hash = *cross.transaction.hash(); + match self.lane_store(cross.store) { + None => { + self.protocol_pool.remove_transactions(vec![hash]); + } + Some(store) => { + let mut pool = store.pool.write(); + let removed = pool.remove_transactions(&[hash]); + store.listeners.write().on_replaced(&removed, replaced_by); + } + } + self.guard.write().release(&hash); + ValidityPoolMetrics::record_cross_store_replacement(cross.store != PoolStore::Validity); + } + + fn stale_classification_error(hash: TxHash) -> PoolError { + PoolError::other(hash, "EIP-8130 admission classification changed during validation") } fn ensure_protocol_classification_current( @@ -554,6 +939,30 @@ where Ok(true) } + /// Validates protocol-bound transactions together and admits them in order. + /// + /// Protocol validation reads canonical state, never pool contents. Every + /// transaction is validated against one snapshot taken when the batch job + /// starts and admitted in order only after the whole batch is validated, so + /// outcomes match per-transaction adds unless the canonical head or the pool + /// changes concurrently in that window. + pub async fn add_protocol_batch( + &self, + transactions: Vec<(TransactionOrigin, T)>, + ) -> Vec> { + let keys: Vec<_> = transactions + .iter() + .map(|(origin, tx)| (*origin, *tx.hash(), tx.sender(), tx.nonce())) + .collect(); + let validated = self.validator().validate_transactions(transactions).await; + keys.into_iter() + .zip(validated) + .map(|((origin, hash, sender, nonce), validated)| { + self.add_validated_protocol_transaction(origin, hash, sender, nonce, validated) + }) + .collect() + } + fn add_validated_protocol_transaction( &self, origin: TransactionOrigin, @@ -562,46 +971,65 @@ where nonce: u64, validated: TransactionValidationOutcome, ) -> PoolResult { + self.admit_protocol_transaction(hash, sender, nonce, validated, |validated| { + let mut outcomes = + self.protocol_pool.inner().add_transactions(origin, std::iter::once(validated)); + outcomes + .pop() + .unwrap_or_else(|| Err(PoolError::other(hash, "inner pool returned no outcome"))) + }) + } + + /// Admits a validated protocol-nonce transaction into the reth pool through + /// `insert`, under the admission lock. Reserves guard capacity first, + /// replaces a same-nonce transaction the validity pool holds, and + /// re-anchors the sender's validity lane to the pending run the newcomer + /// extends or, as a replacement, cuts short. + fn admit_protocol_transaction( + &self, + hash: TxHash, + sender: Address, + nonce: u64, + validated: TransactionValidationOutcome, + insert: impl FnOnce(TransactionValidationOutcome) -> PoolResult, + ) -> PoolResult { let _admission_guard = self.protocol_admission_lock.lock(); // Reject stale validation before reth can replace the currently pooled transaction. self.ensure_protocol_classification_current(hash, &validated)?; self.ensure_not_published_expired(hash, &validated)?; - let replaced = self.protocol_replacement_hash(sender, nonce); + let cross = match validated.as_valid_transaction() { + Some(transaction) => { + self.cross_store_replacement(PoolStore::Protocol, transaction.transaction())? + } + None => None, + }; + let replaced = self + .protocol_replacement_hash(sender, nonce) + .or_else(|| cross.as_ref().map(|cross| *cross.transaction.hash())); // Reserve guard capacity before reth publishes insertion events. A // tracked replacement bypasses this check and is reaccounted after the // protocol pool atomically accepts the fee bump. let pre_admitted = self.pre_admit_protocol_transaction(hash, replaced, &validated)?; - // Capture the block-expiry bound and validity-lane flag before - // `validated` is consumed below. + // Capture the block-expiry bounds before `validated` is consumed below. let block_expiry_bound = Self::validity_block_expiry_bound(&validated); let flashblock_expiry_bound = Self::validity_flashblock_expiry_bound(&validated); - let is_validity = Self::has_validity_predicates(&validated); - let mut outcomes = - self.protocol_pool.inner().add_transactions(origin, std::iter::once(validated)); - let outcome = match outcomes.pop() { - Some(Ok(outcome)) => outcome, - Some(Err(error)) => { + let result = match insert(validated) { + Ok(result) => result, + Err(error) => { if pre_admitted { self.guard.write().release(&hash); } return Err(error); } - None => { - if pre_admitted { - self.guard.write().release(&hash); - } - return Err(reth_transaction_pool::error::PoolError::other( - hash, - "inner pool returned no outcome", - )); - } }; self.gate_protocol_admission(hash, replaced, pre_admitted)?; - self.register_block_expiry(hash, block_expiry_bound, flashblock_expiry_bound, replaced); - if is_validity { - ValidityPoolMetrics::record_admission(replaced.is_some()); + if let Some(cross) = cross { + self.remove_cross_store_replaced(cross, hash); } - Ok(outcome) + self.register_block_expiry(hash, block_expiry_bound, flashblock_expiry_bound, replaced); + self.share_validity_lane((sender, U256::ZERO)); + self.sync_validity_lanes([((sender, U256::ZERO), None)], false); + Ok(result) } fn gate_protocol_admission( @@ -665,183 +1093,302 @@ where Ok(()) } - fn partition_hashes_by_pool(&self, hashes: Vec) -> (Vec, Vec) { - let nonce_pool = self.nonce_pool.read(); - let mut protocol_hashes = Vec::with_capacity(hashes.len()); - let mut sidecar_hashes = Vec::new(); - - for hash in hashes { - if nonce_pool.contains(&hash) { - sidecar_hashes.push(hash); - } else { - protocol_hashes.push(hash); - } - } - - (protocol_hashes, sidecar_hashes) - } - - async fn add_sidecar_transaction( + async fn add_lane_transaction( &self, + store: PoolStore, origin: TransactionOrigin, transaction: T, ) -> PoolResult { + if self.lane_store(store).is_none() { + return Err(PoolError::other( + *transaction.hash(), + "protocol-nonce transaction routed to a lane store", + )); + } let validated = self.validator().validate_transaction(origin, transaction).await; - self.add_validated_sidecar_transaction(validated, origin) + self.add_validated_lane_transaction(store, validated, origin) + } + + /// Where a lane transaction entering `store` anchors its lane. Sidecar + /// lanes sit at the validated channel nonce. Validity lanes additionally + /// place their executable cursor behind the standard store's pending run, + /// cut short at a same-slot transaction this insertion replaces. + fn insert_cursor( + &self, + store: PoolStore, + transaction: &T, + state_nonce: u64, + cross: Option<&CrossStoreReplacement>, + ) -> LaneCursor { + if store != PoolStore::Validity { + return LaneCursor::at(state_nonce); + } + let Some(lane_id) = Self::lane_id_of(transaction) else { + return LaneCursor::at(state_nonce); + }; + let mut next_nonce = self.standard_pending_end(lane_id, state_nonce); + if let Some(cross) = cross { + next_nonce = next_nonce.min(cross.transaction.nonce()); + } + LaneCursor { state_nonce, next_nonce: next_nonce.max(state_nonce) } + } + + /// Undoes a lane insertion that a post-insert check rejected: removes the + /// newcomer, re-announces the lane, and restores the transaction it replaced + /// within the same store. + fn roll_back_lane_insert( + pool: &mut TwoDNoncePool, + listeners: &mut SidecarListeners, + guard: &mut MempoolGuard, + outcome: &InsertOutcome, + cursor: LaneCursor, + ) { + let removed = pool.remove_transactions(&[outcome.outcome.hash]); + listeners.on_discarded(&removed); + listeners.on_lane_moved(&outcome.anchor); + let Some(replaced) = &outcome.replaced else { + return; + }; + let restored = ValidPoolTransaction { + transaction_id: replaced.transaction_id, + transaction: replaced.transaction.clone(), + propagate: replaced.propagate, + timestamp: replaced.timestamp, + origin: replaced.origin, + authority_ids: replaced.authority_ids.clone(), + }; + if pool.insert_validated_at(restored, cursor).is_err() { + guard.release(replaced.hash()); + listeners.on_discarded(std::slice::from_ref(replaced)); + } } - fn add_validated_sidecar_transaction( + /// Inserts a validated transaction into the lane store `store`. + fn add_validated_lane_transaction( &self, + store: PoolStore, validated: TransactionValidationOutcome, origin: TransactionOrigin, ) -> PoolResult { // Capture the block-expiry bound before `validated` is consumed by the - // match below, mirroring the protocol admission paths. A sidecar (EIP-8130) - // transaction carrying block-number validity predicates must be recorded - // in the block-expiry index so it is evicted once its last valid block - // passes; a finite 2D-nonce-channel sidecar has no time-based expiry, so + // match below, mirroring the protocol admission path. A transaction + // carrying block-number validity predicates must be recorded in the + // block-expiry index so it is evicted once its last valid block passes; + // a finite 2D-nonce-channel transaction has no time-based expiry, so // without this it would linger indefinitely. let block_expiry_bound = Self::validity_block_expiry_bound(&validated); let flashblock_expiry_bound = Self::validity_flashblock_expiry_bound(&validated); let _admission_guard = self.protocol_admission_lock.lock(); if let Some(transaction) = validated.as_valid_transaction() { - self.ensure_not_published_expired(*transaction.transaction().hash(), &validated)?; + let hash = *transaction.transaction().hash(); + self.ensure_not_published_expired(hash, &validated)?; } match validated { - TransactionValidationOutcome::Valid { - transaction, - propagate, - authorities, - state_nonce, - .. - } => { - // Keep the sidecar lock order consistent everywhere: nonce_pool before listeners. - let mut nonce_pool = self.nonce_pool.write(); - let mut listeners = self.listeners.write(); - let validated = self.validated_pool_transaction( - transaction, - origin, - propagate, - authorities, - &mut nonce_pool, - ); + TransactionValidationOutcome::Valid { transaction, propagate, state_nonce, .. } => { + let validated = Self::validated_pool_transaction(transaction, origin, propagate); + let hash = *validated.hash(); + let Some(lane_store) = self.lane_store(store) else { + return Err(PoolError::other( + hash, + "protocol-nonce transaction routed to a lane store", + )); + }; + // Look across stores before taking this store's lock: the + // sidecar is locked ahead of the validity pool everywhere. + let cross = self.cross_store_replacement(store, &validated.transaction)?; + let cursor = + self.insert_cursor(store, &validated.transaction, state_nonce, cross.as_ref()); let generation = validated .transaction .limit_class() .map(|class| class.classification_generation); let admission = Self::admission_for(&validated.transaction); - let is_validity = !validated.transaction.validity_conditions().is_empty(); - let outcome = nonce_pool.insert_validated(validated, state_nonce)?; - // nonce_pool serializes sidecar replacement. Never acquire it while holding guard. + let lane_id = Self::lane_id_of(&validated.transaction); + let validity_predicates = validated.transaction.validity_conditions().len(); + // Keep the lock order consistent everywhere: pool before listeners. + let mut pool = lane_store.pool.write(); + let mut listeners = lane_store.listeners.write(); + let mut outcome = pool.insert_validated_at(validated, cursor)?; + // The pool serializes lane replacement. Never acquire it while holding guard. let mut guard = self.guard.write(); + // Transactions below the anchored lane cursor are gone whether or + // not this insertion is kept. + for transaction in &outcome.anchor.discarded { + guard.release(transaction.hash()); + } + listeners.on_discarded(&outcome.anchor.discarded); let current = generation.is_none_or(|generation| { generation == self.validator().validator().limit_class_cache_generation() }); + let replaced_hash = outcome + .replaced + .as_ref() + .map(|replaced| *replaced.hash()) + .or_else(|| cross.as_ref().map(|cross| *cross.transaction.hash())); + // A tracked replacement is reaccounted by force once the lane + // accepted the fee bump; an untracked one admits the newcomer + // under the regular limits. + let tracked_replaced = replaced_hash.filter(|replaced| guard.contains(replaced)); if !current { - let hash = outcome.outcome.hash; - let removed = nonce_pool.remove_transactions(&[hash]); - listeners.on_discarded(&removed); - if let Some(replaced) = &outcome.replaced { - let restored = ValidPoolTransaction { - transaction_id: replaced.transaction_id, - transaction: replaced.transaction.clone(), - propagate: replaced.propagate, - timestamp: replaced.timestamp, - origin: replaced.origin, - authority_ids: replaced.authority_ids.clone(), - }; - if nonce_pool.insert_validated(restored, state_nonce).is_err() { - guard.release(replaced.hash()); - listeners.on_discarded(std::slice::from_ref(replaced)); - } - } + Self::roll_back_lane_insert( + &mut pool, + &mut listeners, + &mut guard, + &outcome, + cursor, + ); return Err(Self::stale_classification_error(hash)); } - match (&outcome.replaced, admission) { + // An incumbent from another store stays pooled there until the + // newcomer survives eviction below, so its reservation is only + // released then; until that point both are reserved, which + // `insert_forced` allows. + let cross_store_incumbent = tracked_replaced.filter(|_| outcome.replaced.is_none()); + match (tracked_replaced, admission) { (Some(replaced), Some(admission)) => { - guard.release(replaced.hash()); + if cross_store_incumbent.is_none() { + guard.release(&replaced); + } guard.insert_forced(admission); } (Some(replaced), None) => { - guard.release(replaced.hash()); + if cross_store_incumbent.is_none() { + guard.release(&replaced); + } } (None, Some(admission)) => { if let Err(rejection) = guard.try_admit(admission) { - let hash = outcome.outcome.hash; - let removed = nonce_pool.remove_transactions(&[hash]); - listeners.on_discarded(&removed); + Self::roll_back_lane_insert( + &mut pool, + &mut listeners, + &mut guard, + &outcome, + cursor, + ); return Err(Self::limit_rejection_error(hash, rejection)); } } (None, None) => {} } - drop(guard); - listeners.on_inserted(&nonce_pool, &outcome); - if is_validity { - ValidityPoolMetrics::record_admission(outcome.replaced.is_some()); + let mut evicted = pool.evict_over_capacity(); + for transaction in &evicted { + guard.release(transaction.hash()); } - // Record the block-expiry bound after a successful insertion, - // dropping the replaced hash's stale entry in the same pass. - // Release the sidecar locks first so the block-expiry index is - // not acquired while holding the nonce pool. let inserted_hash = outcome.outcome.hash; - let replaced_hash = outcome.replaced.as_ref().map(|replaced| *replaced.hash()); + let discarded_on_insert = + evicted.iter().any(|transaction| *transaction.hash() == inserted_hash); + if !discarded_on_insert && let Some(incumbent) = cross_store_incumbent { + guard.release(&incumbent); + } + drop(guard); + let survived = |moved: &Arc>| { + !evicted.iter().any(|transaction| transaction.hash() == moved.hash()) + }; + outcome.anchor.promoted.retain(survived); + outcome.anchor.queued.retain(survived); + outcome.promoted.retain(survived); + if store == PoolStore::Validity { + ValidityPoolMetrics::record_evictions(evicted.len()); + } + if discarded_on_insert { + evicted.retain(|transaction| *transaction.hash() != inserted_hash); + evicted.extend(outcome.replaced); + listeners.on_discarded(&evicted); + listeners.on_lane_moved(&outcome.anchor); + drop(listeners); + drop(pool); + self.remove_block_expiry(&evicted); + return Err(PoolError::new(inserted_hash, PoolErrorKind::DiscardedOnInsert)); + } + listeners.on_discarded(&evicted); + listeners.on_inserted(&pool, &outcome); + if validity_predicates > 0 { + ValidityPoolMetrics::record_admission( + replaced_hash.is_some(), + validity_predicates, + ); + } + // Release this store's locks before touching any other store or + // the block-expiry index. drop(listeners); - drop(nonce_pool); + drop(pool); + self.remove_block_expiry(&evicted); + if let Some(cross) = cross { + self.remove_cross_store_replaced(cross, inserted_hash); + } self.register_block_expiry( inserted_hash, block_expiry_bound, flashblock_expiry_bound, replaced_hash, ); + match store { + PoolStore::Sidecar => { + if let Some(lane_id) = lane_id { + self.share_validity_lane(lane_id); + } + } + PoolStore::Validity => { + if let Some(lane_id) = + lane_id.filter(|lane_id| self.standard_store_holds_lane(*lane_id)) + { + self.validity.pool.write().mark_shared_lane(lane_id); + } + } + PoolStore::Protocol => {} + } + if store == PoolStore::Sidecar { + // Validity lanes on the channels this insertion extended or + // the eviction shortened follow the sidecar's pending run. + self.sync_validity_lanes( + lane_id + .into_iter() + .chain( + evicted + .iter() + .filter_map(|evicted| Self::lane_id_of(&evicted.transaction)), + ) + .map(|lane_id| (lane_id, None)), + false, + ); + } Ok(outcome.outcome) } TransactionValidationOutcome::Invalid(transaction, error) => { - Err(reth_transaction_pool::error::PoolError::new( - *transaction.hash(), - reth_transaction_pool::error::PoolErrorKind::InvalidTransaction(error), - )) + Err(PoolError::new(*transaction.hash(), PoolErrorKind::InvalidTransaction(error))) } TransactionValidationOutcome::Error(hash, error) => { - Err(reth_transaction_pool::error::PoolError::other(hash, error.to_string())) + Err(PoolError::other(hash, error.to_string())) } } } + /// Wraps a validated lane transaction. Its sender and authority ids are + /// placeholders until [`TwoDNoncePool::insert_validated`] assigns them. fn validated_pool_transaction( - &self, transaction: reth_transaction_pool::validate::ValidTransaction, origin: TransactionOrigin, propagate: bool, - authorities: Option>, - nonce_pool: &mut TwoDNoncePool, ) -> ValidPoolTransaction { let transaction = transaction.into_transaction(); - let sender_id = nonce_pool.sender_id_or_create(transaction.sender()); - let authority_ids = authorities.map(|authorities| { - authorities - .into_iter() - .map(|authority| nonce_pool.sender_id_or_create(authority)) - .collect() - }); - ValidPoolTransaction { transaction_id: reth_transaction_pool::identifier::TransactionId::new( - sender_id, + 0u64.into(), transaction.nonce(), ), transaction, propagate, timestamp: std::time::Instant::now(), origin, - authority_ids, + authority_ids: None, } } fn merged_pending_listener(&self, kind: TransactionListenerKind) -> mpsc::Receiver { let protocol = self.protocol_pool.pending_transactions_listener_for(kind); - let sidecar = self.listeners.write().subscribe_pending(kind); - merge_receivers(protocol, sidecar) + let sidecar = self.sidecar.listeners.write().subscribe_pending(kind); + let validity = self.validity.listeners.write().subscribe_pending(kind); + merge_receivers(protocol, merge_receivers(sidecar, validity)) } fn merged_new_transactions_listener( @@ -849,40 +1396,123 @@ where kind: TransactionListenerKind, ) -> mpsc::Receiver> { let protocol = self.protocol_pool.new_transactions_listener_for(kind); - let sidecar = self.listeners.write().subscribe_new_transactions(kind); - merge_receivers(protocol, sidecar) + let sidecar = self.sidecar.listeners.write().subscribe_new_transactions(kind); + let validity = self.validity.listeners.write().subscribe_new_transactions(kind); + merge_receivers(protocol, merge_receivers(sidecar, validity)) } fn merged_all_transactions_listener(&self) -> AllTransactionsEvents { - let mut protocol = self.protocol_pool.all_transactions_event_listener(); - let mut sidecar = self.listeners.write().subscribe_all(); + let protocol = self.protocol_pool.all_transactions_event_listener(); + let sidecar = self.sidecar.listeners.write().subscribe_all(); + let validity = self.validity.listeners.write().subscribe_all(); + let mut merged = futures::stream::select_all([protocol, sidecar, validity]); let (tx, rx) = mpsc::channel(SIDE_CAR_EVENT_CHANNEL_SIZE); spawn(async move { - let mut protocol_open = true; - let mut sidecar_open = true; - while protocol_open || sidecar_open { - tokio::select! { - event = protocol.next(), if protocol_open => match event { - Some(event) => { - if tx.send(event).await.is_err() { - break; - } - } - None => protocol_open = false, - }, - event = sidecar.next(), if sidecar_open => match event { - Some(event) => { - if tx.send(event).await.is_err() { - break; - } - } - None => sidecar_open = false, - } + while let Some(event) = merged.next().await { + if tx.send(event).await.is_err() { + break; } } }); AllTransactionsEvents::new(rx) } + + fn merged_best_transactions( + &self, + attributes: BestTransactionsAttributes, + ) -> MergeBestTransactions { + let base_fee = attributes.basefee; + // The lane bookkeeping the merge relies on (anchors, shared lanes) and + // the lane stores' iterators are snapshotted under one pair of read + // guards, taken in lock order (sidecar before validity), so an admission + // cannot land between them and hand the merge a head whose lane it does + // not know is ahead of state. Only reth's snapshot is taken separately, + // and a stale one leaves a validity head waiting, never released early. + // The bookkeeping comes from the validity pool's incremental indexes, + // so this costs the lanes that straddle stores, not the pool sizes. + let (sidecar, validity, anchors, bridged) = { + let sidecar = self.sidecar.pool.read(); + let validity = self.validity.pool.read(); + ( + sidecar.best_transactions(self.ordering.clone(), base_fee), + validity.best_transactions(self.ordering.clone(), base_fee), + validity.lanes_ahead_of_state().clone(), + self.bridged_descendants(&sidecar, &validity), + ) + }; + // Reth hands back a boxed iterator of a private type, so it is boxed + // once more as a `MergeSource`: one allocation per build. + let protocol = self.protocol_pool.best_transactions_with_attributes(attributes); + MergeBestTransactions::new( + Box::new(protocol), + Box::new(sidecar), + Box::new(validity), + self.ordering.clone(), + base_fee, + ) + .with_external_anchors(anchors) + .with_bridged_descendants(bridged) + } + + /// For each validity lane that shares its nonce space with a standard + /// store, the consecutive run that store holds from where the validity + /// lane's pending run ends. The store queues those transactions behind the + /// gap the validity run fills, so only the merge can release them, after + /// that run. + fn bridged_descendants( + &self, + sidecar: &TwoDNoncePool, + validity: &TwoDNoncePool, + ) -> BridgedDescendants { + let mut bridged = BridgedDescendants::default(); + for (lane_id, end) in validity.shared_lanes_pending_end() { + let run: VecDeque<_> = (end..) + .map_while(|nonce| self.standard_transaction(sidecar, lane_id, nonce)) + .collect(); + if !run.is_empty() { + bridged.insert(lane_id, run); + } + } + bridged + } + + /// The transaction the standard store owning `lane_id` holds at `nonce`, + /// pending or queued. + fn standard_transaction( + &self, + sidecar: &TwoDNoncePool, + lane_id: LaneId, + nonce: u64, + ) -> Option>> { + let (sender, nonce_key) = lane_id; + if nonce_key.is_zero() { + self.protocol_pool.get_transaction_by_sender_and_nonce(sender, nonce) + } else { + sidecar.lane_transaction(&lane_id, nonce) + } + } + + /// Records that `lane_id` now has transactions in both the validity pool + /// and its standard store, so builds look for standard descendants behind + /// the validity run. Called from both stores' admission paths, so the two + /// meet whichever arrives second; `sync_validity_lanes` clears the flag + /// once the standard store has emptied the lane again. + fn share_validity_lane(&self, lane_id: LaneId) { + if self.validity.pool.read().lane_cursor(&lane_id).is_none() { + return; + } + self.validity.pool.write().mark_shared_lane(lane_id); + } + + /// Whether the standard store owning `lane_id` holds any transaction in it. + fn standard_store_holds_lane(&self, lane_id: LaneId) -> bool { + let (sender, nonce_key) = lane_id; + if nonce_key.is_zero() { + self.protocol_pool.get_highest_transaction_by_sender(sender).is_some() + } else { + self.sidecar.pool.read().lane_cursor(&lane_id).is_some() + } + } } impl StateDiffInvalidation for BaseTransactionPool @@ -923,7 +1553,12 @@ where let expired = self.expire_published_locked(block_number, flashblock_index); Ok((result, expired)) } -} + + fn expire_published(&self, block_number: u64, flashblock_index: u64) -> usize { + let _admission_guard = self.protocol_admission_lock.lock(); + self.expire_published_locked(block_number, flashblock_index) + } +} impl TransactionPool for BaseTransactionPool where @@ -938,17 +1573,14 @@ where fn pool_size(&self) -> PoolSize { let mut size = self.protocol_pool.pool_size(); - let nonce_pool = self.nonce_pool.read(); - let (pending, queued) = nonce_pool.pending_and_queued_txn_count(); - let pending_size: usize = - nonce_pool.pending_transactions().iter().map(|tx| tx.encoded_length()).sum(); - let queued_size: usize = - nonce_pool.queued_transactions().iter().map(|tx| tx.encoded_length()).sum(); - size.pending += pending; - size.pending_size += pending_size; - size.queued += queued; - size.queued_size += queued_size; - size.total += pending + queued; + for store in self.lane_stores() { + let store_size = store.pool.read().pool_size(); + size.pending += store_size.pending; + size.pending_size += store_size.pending_size; + size.queued += store_size.queued; + size.queued_size += store_size.queued_size; + size.total += store_size.total; + } size } @@ -961,43 +1593,20 @@ where origin: TransactionOrigin, transaction: Self::Transaction, ) -> PoolResult { - if !self.is_sidecar_transaction(&transaction) { - let hash = *transaction.hash(); + let store = PoolStore::of(&transaction); + let hash = *transaction.hash(); + let Some(lane_store) = self.lane_store(store) else { let sender = transaction.sender(); let nonce = transaction.nonce(); let validated = self.validator().validate_transaction(origin, transaction).await; - let _admission_guard = self.protocol_admission_lock.lock(); - self.ensure_protocol_classification_current(hash, &validated)?; - self.ensure_not_published_expired(hash, &validated)?; - let replaced = self.protocol_replacement_hash(sender, nonce); - let pre_admitted = self.pre_admit_protocol_transaction(hash, replaced, &validated)?; - // Capture the block-expiry bound and validity-lane flag before - // `validated` is consumed below. - let block_expiry_bound = Self::validity_block_expiry_bound(&validated); - let flashblock_expiry_bound = Self::validity_flashblock_expiry_bound(&validated); - let is_validity = Self::has_validity_predicates(&validated); - let events = - match self.protocol_pool.inner().add_transaction_and_subscribe(origin, validated) { - Ok(events) => events, - Err(error) => { - if pre_admitted { - self.guard.write().release(&hash); - } - return Err(error); - } - }; - self.gate_protocol_admission(hash, replaced, pre_admitted)?; - self.register_block_expiry(hash, block_expiry_bound, flashblock_expiry_bound, replaced); - if is_validity { - ValidityPoolMetrics::record_admission(replaced.is_some()); - } - return Ok(events); - } + return self.admit_protocol_transaction(hash, sender, nonce, validated, |validated| { + self.protocol_pool.inner().add_transaction_and_subscribe(origin, validated) + }); + }; - let hash = *transaction.hash(); - let (events, listener) = self.listeners.write().subscribe_hash(hash); - if let Err(error) = self.add_sidecar_transaction(origin, transaction).await { - self.listeners.write().unsubscribe_hash_listener(&hash, &listener); + let (events, listener) = lane_store.listeners.write().subscribe_hash(hash); + if let Err(error) = self.add_lane_transaction(store, origin, transaction).await { + lane_store.listeners.write().unsubscribe_hash_listener(&hash, &listener); return Err(error); } Ok(events) @@ -1008,14 +1617,15 @@ where origin: TransactionOrigin, transaction: Self::Transaction, ) -> PoolResult { - if self.is_sidecar_transaction(&transaction) { - self.add_sidecar_transaction(origin, transaction).await - } else { - let hash = *transaction.hash(); - let sender = transaction.sender(); - let nonce = transaction.nonce(); - let validated = self.validator().validate_transaction(origin, transaction).await; - self.add_validated_protocol_transaction(origin, hash, sender, nonce, validated) + match PoolStore::of(&transaction) { + PoolStore::Protocol => { + let hash = *transaction.hash(); + let sender = transaction.sender(); + let nonce = transaction.nonce(); + let validated = self.validator().validate_transaction(origin, transaction).await; + self.add_validated_protocol_transaction(origin, hash, sender, nonce, validated) + } + store => self.add_lane_transaction(store, origin, transaction).await, } } @@ -1034,37 +1644,55 @@ where &self, transactions: Vec<(TransactionOrigin, Self::Transaction)>, ) -> Vec> { - if transactions.iter().any(|(_, transaction)| self.is_sidecar_transaction(transaction)) { - let mut results = Vec::with_capacity(transactions.len()); - for (origin, transaction) in transactions { - results.push(self.add_transaction(origin, transaction).await); - } - return results; - } - // Protocol validation reads canonical state, never pool contents. Every - // transaction is validated against one snapshot taken when the batch - // job starts and admitted in order only after the whole batch is - // validated, so outcomes match per-transaction adds unless the - // canonical head or the pool changes concurrently in that window. - let keys: Vec<_> = transactions + // Senders with a transaction bound for a lane store keep their whole + // batch on the per-transaction path, in submission order, because their + // lane and cross-store replacement outcomes depend on that order. Every + // other transaction is protocol-bound and validated as one batch, which + // is admitted first: a batch promises no order across senders. + let lane_senders: AddressSet = transactions .iter() - .map(|(origin, tx)| (*origin, *tx.hash(), tx.sender(), tx.nonce())) + .filter(|(_, transaction)| PoolStore::of(transaction).is_lane_store()) + .map(|(_, transaction)| transaction.sender()) .collect(); - let validated = self.validator().validate_transactions(transactions).await; - keys.into_iter() - .zip(validated) - .map(|((origin, hash, sender, nonce), validated)| { - self.add_validated_protocol_transaction(origin, hash, sender, nonce, validated) - }) - .collect() + if lane_senders.is_empty() { + return self.add_protocol_batch(transactions).await; + } + + let mut results: Vec>> = + (0..transactions.len()).map(|_| None).collect(); + let mut batched = Vec::new(); + let mut batched_indexes = Vec::new(); + let mut one_by_one = Vec::new(); + for (index, (origin, transaction)) in transactions.into_iter().enumerate() { + if lane_senders.contains(&transaction.sender()) { + one_by_one.push((index, origin, transaction)); + } else { + batched_indexes.push(index); + batched.push((origin, transaction)); + } + } + if !batched.is_empty() { + for (index, result) in + batched_indexes.into_iter().zip(self.add_protocol_batch(batched).await) + { + results[index] = Some(result); + } + } + for (index, origin, transaction) in one_by_one { + results[index] = Some(self.add_transaction(origin, transaction).await); + } + results.into_iter().map(|result| result.expect("every transaction is admitted")).collect() } fn transaction_event_listener(&self, tx_hash: TxHash) -> Option { self.protocol_pool.transaction_event_listener(tx_hash).or_else(|| { - self.nonce_pool - .read() - .contains(&tx_hash) - .then(|| self.listeners.write().subscribe_hash(tx_hash).0) + self.lane_stores().into_iter().find_map(|store| { + store + .pool + .read() + .contains(&tx_hash) + .then(|| store.listeners.write().subscribe_hash(tx_hash).0) + }) }) } @@ -1092,44 +1720,52 @@ where fn pooled_transaction_hashes(&self) -> Vec { let mut hashes = self.protocol_pool.pooled_transaction_hashes(); - hashes.extend( - self.nonce_pool - .read() - .all_transactions() - .into_iter() - .filter(|transaction| transaction.propagate) - .map(|transaction| *transaction.hash()), - ); + for store in self.lane_stores() { + hashes.extend( + store + .pool + .read() + .all_transactions() + .into_iter() + .filter(|transaction| transaction.propagate) + .map(|transaction| *transaction.hash()), + ); + } hashes } fn pooled_transaction_hashes_max(&self, max: usize) -> Vec { let mut hashes = self.protocol_pool.pooled_transaction_hashes_max(max); - if hashes.len() >= max { - return hashes; - } - - let nonce_pool = self.nonce_pool.read(); - for transaction in nonce_pool.all_transactions() { - if transaction.propagate { - hashes.push(*transaction.hash()); - if hashes.len() >= max { - break; - } + for store in self.lane_stores() { + if hashes.len() >= max { + break; } + let remaining = max - hashes.len(); + hashes.extend( + store + .pool + .read() + .all_transactions_iter() + .filter(|transaction| transaction.propagate) + .map(|transaction| *transaction.hash()) + .take(remaining), + ); } hashes } fn pooled_transactions(&self) -> Vec>> { let mut transactions = self.protocol_pool.pooled_transactions(); - transactions.extend( - self.nonce_pool - .read() - .all_transactions() - .into_iter() - .filter(|transaction| transaction.propagate), - ); + for store in self.lane_stores() { + transactions.extend( + store + .pool + .read() + .all_transactions() + .into_iter() + .filter(|transaction| transaction.propagate), + ); + } transactions } @@ -1138,18 +1774,20 @@ where max: usize, ) -> Vec>> { let mut transactions = self.protocol_pool.pooled_transactions_max(max); - if transactions.len() >= max { - return transactions; - } - - let nonce_pool = self.nonce_pool.read(); - for transaction in nonce_pool.all_transactions() { - if transaction.propagate { - transactions.push(transaction); - if transactions.len() >= max { - break; - } + for store in self.lane_stores() { + if transactions.len() >= max { + break; } + let remaining = max - transactions.len(); + transactions.extend( + store + .pool + .read() + .all_transactions_iter() + .filter(|transaction| transaction.propagate) + .take(remaining) + .cloned(), + ); } transactions } @@ -1191,10 +1829,13 @@ where tx_hash: TxHash, ) -> Option::Pooled>> { self.protocol_pool.get_pooled_transaction_element(tx_hash).or_else(|| { - self.nonce_pool - .read() - .get(&tx_hash) - .and_then(|transaction| transaction.transaction.clone().try_into_pooled().ok()) + self.lane_stores().into_iter().find_map(|store| { + store + .pool + .read() + .get(&tx_hash) + .and_then(|transaction| transaction.transaction.clone().try_into_pooled().ok()) + }) }) } @@ -1206,31 +1847,21 @@ where block_info.pending_basefee, block_info.pending_blob_fee.map(|fee| u64::try_from(fee).unwrap_or(u64::MAX)), ); - let base_fee = best_transactions_attributes.basefee; - Box::new(MergeBestTransactions::new( - self.protocol_pool.best_transactions_with_attributes(best_transactions_attributes), - Box::new(self.nonce_pool.read().best_transactions(self.ordering.clone(), base_fee)), - self.ordering.clone(), - base_fee, - )) + Box::new(self.merged_best_transactions(best_transactions_attributes)) } fn best_transactions_with_attributes( &self, best_transactions_attributes: BestTransactionsAttributes, ) -> Box>>> { - let base_fee = best_transactions_attributes.basefee; - Box::new(MergeBestTransactions::new( - self.protocol_pool.best_transactions_with_attributes(best_transactions_attributes), - Box::new(self.nonce_pool.read().best_transactions(self.ordering.clone(), base_fee)), - self.ordering.clone(), - base_fee, - )) + Box::new(self.merged_best_transactions(best_transactions_attributes)) } fn pending_transactions(&self) -> Vec>> { let mut transactions = self.protocol_pool.pending_transactions(); - transactions.extend(self.nonce_pool.read().pending_transactions()); + for store in self.lane_stores() { + transactions.extend(store.pool.read().pending_transactions()); + } transactions } @@ -1240,9 +1871,11 @@ where nonce: u64, ) -> Option>> { // Channelized nonce sequences and nonce-free replay ids live in separate - // namespaces from account nonces, so this sender+nonce lookup - // intentionally remains protocol-only. - self.protocol_pool.get_pending_transaction_by_sender_and_nonce(sender, nonce) + // namespaces from account nonces, so this sender+nonce lookup covers the + // protocol pool and the validity pool's protocol-nonce lanes only. + self.protocol_pool.get_pending_transaction_by_sender_and_nonce(sender, nonce).or_else( + || self.validity.pool.read().lane_pending_transaction(&(sender, U256::ZERO), nonce), + ) } fn pending_transactions_max( @@ -1250,40 +1883,50 @@ where max: usize, ) -> Vec>> { let mut transactions = self.protocol_pool.pending_transactions_max(max); - if transactions.len() >= max { - return transactions; + for store in self.lane_stores() { + if transactions.len() >= max { + break; + } + let remaining = max - transactions.len(); + transactions + .extend(store.pool.read().pending_transactions_iter().take(remaining).cloned()); } - - let remaining = max - transactions.len(); - transactions - .extend(self.nonce_pool.read().pending_transactions().into_iter().take(remaining)); transactions } fn queued_transactions(&self) -> Vec>> { let mut transactions = self.protocol_pool.queued_transactions(); - transactions.extend(self.nonce_pool.read().queued_transactions()); + for store in self.lane_stores() { + transactions.extend(store.pool.read().queued_transactions()); + } transactions } fn pending_and_queued_txn_count(&self) -> (usize, usize) { - let (pending, queued) = self.protocol_pool.pending_and_queued_txn_count(); - let (sidecar_pending, sidecar_queued) = - self.nonce_pool.read().pending_and_queued_txn_count(); - (pending + sidecar_pending, queued + sidecar_queued) + let (mut pending, mut queued) = self.protocol_pool.pending_and_queued_txn_count(); + for store in self.lane_stores() { + let (store_pending, store_queued) = store.pool.read().pending_and_queued_txn_count(); + pending += store_pending; + queued += store_queued; + } + (pending, queued) } fn all_transactions(&self) -> AllPoolTransactions { let mut transactions = self.protocol_pool.all_transactions(); - let nonce_pool = self.nonce_pool.read(); - transactions.pending.extend(nonce_pool.pending_transactions()); - transactions.queued.extend(nonce_pool.queued_transactions()); + for store in self.lane_stores() { + let pool = store.pool.read(); + transactions.pending.extend(pool.pending_transactions()); + transactions.queued.extend(pool.queued_transactions()); + } transactions } fn all_transaction_hashes(&self) -> Vec { let mut hashes = self.protocol_pool.all_transaction_hashes(); - hashes.extend(self.nonce_pool.read().all_hashes()); + for store in self.lane_stores() { + hashes.extend(store.pool.read().all_hashes()); + } hashes } @@ -1291,47 +1934,36 @@ where &self, hashes: Vec, ) -> Vec>> { - let (protocol_hashes, sidecar_hashes) = self.partition_hashes_by_pool(hashes); - let mut removed = self.protocol_pool.remove_transactions(protocol_hashes); - self.release_from_guard(&removed); - let sidecar_removed = self.nonce_pool.write().remove_transactions(&sidecar_hashes); - if !sidecar_removed.is_empty() { - self.listeners.write().on_discarded(&sidecar_removed); - } - self.release_from_guard(&sidecar_removed); - removed.extend(sidecar_removed); - removed + // Removals re-anchor validity lanes from the standard stores' state, so + // they are serialized with admission like every other re-anchoring path. + let _admission_guard = self.protocol_admission_lock.lock(); + self.remove_from_stores(hashes, RemoveMode::Exact) } fn remove_transactions_and_descendants( &self, hashes: Vec, ) -> Vec>> { - let (protocol_hashes, sidecar_hashes) = self.partition_hashes_by_pool(hashes); - let mut removed = self.protocol_pool.remove_transactions_and_descendants(protocol_hashes); - self.release_from_guard(&removed); - let sidecar_removed = - self.nonce_pool.write().remove_transactions_and_descendants(&sidecar_hashes); - if !sidecar_removed.is_empty() { - self.listeners.write().on_discarded(&sidecar_removed); - } - self.release_from_guard(&sidecar_removed); - removed.extend(sidecar_removed); - removed + let _admission_guard = self.protocol_admission_lock.lock(); + self.remove_from_stores(hashes, RemoveMode::WithDescendants) } fn remove_transactions_by_sender( &self, sender: Address, ) -> Vec>> { + let _admission_guard = self.protocol_admission_lock.lock(); let mut removed = self.protocol_pool.remove_transactions_by_sender(sender); - self.release_from_guard(&removed); - let sidecar_removed = self.nonce_pool.write().remove_transactions_by_sender(sender); - if !sidecar_removed.is_empty() { - self.listeners.write().on_discarded(&sidecar_removed); + for store in self.lane_stores() { + let mut pool = store.pool.write(); + let store_removed = pool.remove_transactions_by_sender(sender); + if !store_removed.is_empty() { + store.listeners.write().on_discarded(&store_removed); + } + removed.extend(store_removed); } - self.release_from_guard(&sidecar_removed); - removed.extend(sidecar_removed); + // The sender's validity lanes went with the rest, so nothing is left to re-anchor. + self.release_from_guard(&removed); removed } @@ -1339,13 +1971,8 @@ where &self, hashes: Vec, ) -> Vec>> { - let (protocol_hashes, sidecar_hashes) = self.partition_hashes_by_pool(hashes); - let mut removed = self.protocol_pool.prune_transactions(protocol_hashes); - self.release_from_guard(&removed); - let pruned = self.nonce_pool.write().prune_mined(&sidecar_hashes); - self.release_from_guard(&pruned.removed); - removed.extend(pruned.removed); - removed + let _admission_guard = self.protocol_admission_lock.lock(); + self.remove_from_stores(hashes, RemoveMode::Mined) } fn retain_unknown(&self, announcement: &mut A) @@ -1357,37 +1984,54 @@ where return; } - let nonce_pool = self.nonce_pool.read(); - announcement.retain_by_hash(|hash| !nonce_pool.contains(hash)); + let sidecar = self.sidecar.pool.read(); + let validity = self.validity.pool.read(); + announcement.retain_by_hash(|hash| !sidecar.contains(hash) && !validity.contains(hash)); } fn retain_contains(&self, announcement: &mut A) where A: HandleMempoolData, { - let nonce_pool = self.nonce_pool.read(); + let sidecar = self.sidecar.pool.read(); + let validity = self.validity.pool.read(); announcement.retain_by_hash(|hash| { - self.protocol_pool.get(hash).is_some() || nonce_pool.contains(hash) + self.protocol_pool.get(hash).is_some() + || sidecar.contains(hash) + || validity.contains(hash) }); } fn get(&self, tx_hash: &TxHash) -> Option>> { - self.protocol_pool.get(tx_hash).or_else(|| self.nonce_pool.read().get(tx_hash)) + self.protocol_pool.get(tx_hash).or_else(|| { + self.lane_stores().into_iter().find_map(|store| store.pool.read().get(tx_hash)) + }) } fn get_all(&self, txs: Vec) -> Vec>> { - let nonce_pool = self.nonce_pool.read(); + let sidecar = self.sidecar.pool.read(); + let validity = self.validity.pool.read(); txs.into_iter() - .filter_map(|tx| self.protocol_pool.get(&tx).or_else(|| nonce_pool.get(&tx))) + .filter_map(|tx| { + self.protocol_pool + .get(&tx) + .or_else(|| sidecar.get(&tx)) + .or_else(|| validity.get(&tx)) + }) .collect() } fn on_propagated(&self, txs: PropagatedTransactions) { - let nonce_pool = self.nonce_pool.read(); + let sidecar = self.sidecar.pool.read(); + let validity = self.validity.pool.read(); let protocol_txs = PropagatedTransactions( - txs.0.into_iter().filter(|(hash, _)| !nonce_pool.contains(hash)).collect(), + txs.0 + .into_iter() + .filter(|(hash, _)| !sidecar.contains(hash) && !validity.contains(hash)) + .collect(), ); - drop(nonce_pool); + drop(validity); + drop(sidecar); self.protocol_pool.on_propagated(protocol_txs) } @@ -1396,7 +2040,9 @@ where sender: Address, ) -> Vec>> { let mut transactions = self.protocol_pool.get_transactions_by_sender(sender); - transactions.extend(self.nonce_pool.read().transactions_by_sender(sender)); + for store in self.lane_stores() { + transactions.extend(store.pool.read().transactions_by_sender(sender)); + } transactions } @@ -1406,13 +2052,16 @@ where ) -> Vec>> { let mut transactions = self.protocol_pool.get_pending_transactions_with_predicate(&mut predicate); - transactions.extend( - self.nonce_pool - .read() - .pending_transactions() - .into_iter() - .filter(|transaction| predicate(transaction)), - ); + for store in self.lane_stores() { + transactions.extend( + store + .pool + .read() + .pending_transactions() + .into_iter() + .filter(|transaction| predicate(transaction)), + ); + } transactions } @@ -1421,7 +2070,9 @@ where sender: Address, ) -> Vec>> { let mut transactions = self.protocol_pool.get_pending_transactions_by_sender(sender); - transactions.extend(self.nonce_pool.read().pending_transactions_by_sender(sender)); + for store in self.lane_stores() { + transactions.extend(store.pool.read().pending_transactions_by_sender(sender)); + } transactions } @@ -1430,7 +2081,9 @@ where sender: Address, ) -> Vec>> { let mut transactions = self.protocol_pool.get_queued_transactions_by_sender(sender); - transactions.extend(self.nonce_pool.read().queued_transactions_by_sender(sender)); + for store in self.lane_stores() { + transactions.extend(store.pool.read().queued_transactions_by_sender(sender)); + } transactions } @@ -1438,7 +2091,14 @@ where &self, sender: Address, ) -> Option>> { - self.protocol_pool.get_highest_transaction_by_sender(sender) + let protocol = self.protocol_pool.get_highest_transaction_by_sender(sender); + let validity = self.validity.pool.read().lane_highest_transaction(&(sender, U256::ZERO)); + match (protocol, validity) { + (Some(protocol), Some(validity)) => { + Some(if validity.nonce() > protocol.nonce() { validity } else { protocol }) + } + (protocol, validity) => protocol.or(validity), + } } fn get_highest_consecutive_transaction_by_sender( @@ -1446,7 +2106,32 @@ where sender: Address, on_chain_nonce: u64, ) -> Option>> { - self.protocol_pool.get_highest_consecutive_transaction_by_sender(sender, on_chain_nonce) + // The run may alternate between the protocol pool and the validity + // pool's protocol-nonce lane, so follow it across both until neither + // holds the next nonce. + let validity = self.validity.pool.read(); + let lane_id = (sender, U256::ZERO); + let mut next = on_chain_nonce; + let mut last = None; + loop { + let mut advanced = false; + if let Some(transaction) = + self.protocol_pool.get_highest_consecutive_transaction_by_sender(sender, next) + && transaction.nonce() >= next + { + next = transaction.nonce().checked_add(1)?; + last = Some(transaction); + advanced = true; + } + if let Some(transaction) = validity.highest_consecutive_from(&lane_id, next) { + next = transaction.nonce().checked_add(1)?; + last = Some(transaction); + advanced = true; + } + if !advanced { + return last; + } + } } fn get_transaction_by_sender_and_nonce( @@ -1454,7 +2139,9 @@ where sender: Address, nonce: u64, ) -> Option>> { - self.protocol_pool.get_transaction_by_sender_and_nonce(sender, nonce) + self.protocol_pool + .get_transaction_by_sender_and_nonce(sender, nonce) + .or_else(|| self.validity.pool.read().lane_transaction(&(sender, U256::ZERO), nonce)) } fn get_transactions_by_origin( @@ -1462,13 +2149,16 @@ where origin: TransactionOrigin, ) -> Vec>> { let mut transactions = self.protocol_pool.get_transactions_by_origin(origin); - transactions.extend( - self.nonce_pool - .read() - .all_transactions() - .into_iter() - .filter(|transaction| transaction.origin == origin), - ); + for store in self.lane_stores() { + transactions.extend( + store + .pool + .read() + .all_transactions() + .into_iter() + .filter(|transaction| transaction.origin == origin), + ); + } transactions } @@ -1477,20 +2167,23 @@ where origin: TransactionOrigin, ) -> Vec>> { let mut transactions = self.protocol_pool.get_pending_transactions_by_origin(origin); - transactions.extend( - self.nonce_pool - .read() - .pending_transactions() - .into_iter() - .filter(|transaction| transaction.origin == origin), - ); + for store in self.lane_stores() { + transactions.extend( + store + .pool + .read() + .pending_transactions() + .into_iter() + .filter(|transaction| transaction.origin == origin), + ); + } transactions } fn unique_senders(&self) -> AddressSet { let mut senders = self.protocol_pool.unique_senders(); - for sender in self.nonce_pool.read().unique_senders() { - senders.insert(sender); + for store in self.lane_stores() { + senders.extend(store.pool.read().unique_senders()); } senders } @@ -1569,15 +2262,13 @@ where fn best_transactions_with_attributes_and_parking( &self, attributes: BestTransactionsAttributes, + hold: Option>>, ) -> Box> { let base_fee = attributes.basefee; - let merged = MergeBestTransactions::new( - self.protocol_pool.best_transactions_with_attributes(attributes), - Box::new(self.nonce_pool.read().best_transactions(self.ordering.clone(), base_fee)), - self.ordering.clone(), - base_fee, - ); - Box::new(ParkedBestTransactions::new(merged, self.ordering.clone(), base_fee)) + let merged = self.merged_best_transactions(attributes); + Box::new( + ParkedBestTransactions::new(merged, self.ordering.clone(), base_fee).with_hold(hold), + ) } } @@ -1604,12 +2295,18 @@ where let now = update.timestamp(); let block_number = update.number(); let mined_transactions = update.mined_transactions.clone(); + let reorg = update.update_kind == PoolUpdateKind::Reorg; + let changed_nonces: Vec<(Address, u64)> = update + .changed_accounts + .iter() + .map(|account| (account.address, account.nonce)) + .collect(); // Free mined capacity atomically with admission before the heavier pool // maintenance. The transaction is already canonical, so it no longer // consumes an admission slot even until its pool entry is pruned below. { let _admission_guard = self.protocol_admission_lock.lock(); - if update.update_kind == PoolUpdateKind::Reorg { + if reorg { // The new branch can rebuild an earlier block or restart its flashblock indices. *self.published_flashblock.write() = None; } @@ -1623,32 +2320,62 @@ where } } self.protocol_pool.on_canonical_state_change(update); - { - let mut nonce_pool = self.nonce_pool.write(); - let pruned = nonce_pool.prune_mined(&mined_transactions); - // The nonce-free validity window is in milliseconds, evaluated - // against `block.timestamp * 1000`. - let expired = nonce_pool.remove_expired_nonce_free(now.saturating_mul(1_000)); - let mut listeners = self.listeners.write(); + // The nonce-free validity window is in milliseconds, evaluated + // against `block.timestamp * 1000`. + let now_ms = now.saturating_mul(1_000); + let mut touched_lanes = Vec::new(); + for store in self.lane_stores() { + let mut pool = store.pool.write(); + let pruned = pool.prune_mined(&mined_transactions); + let expired = pool.remove_expired_nonce_free(now_ms); + let mut listeners = store.listeners.write(); if !pruned.removed.is_empty() { listeners.on_mined(&pruned.removed, block_hash); } + listeners.on_lane_update(&pruned.lanes); if !expired.is_empty() { listeners.on_discarded(&expired); } - // Sidecar maintenance lock order is nonce_pool -> listeners -> guard. + // Lane-store maintenance lock order is pool -> listeners -> guard. let mut guard = self.guard.write(); - for transaction in &pruned.removed { + for transaction in pruned.removed.iter().chain(&pruned.lanes.discarded).chain(&expired) + { guard.release(transaction.hash()); } - for transaction in &expired { - guard.release(transaction.hash()); + // Mined lane transactions moved their channel's canonical nonce past + // them; a validity lane on the same channel follows that move. + touched_lanes.extend(pruned.removed.iter().filter_map(|transaction| { + Self::lane_id_of(&transaction.transaction) + .map(|lane_id| (lane_id, Some(transaction.nonce().saturating_add(1)))) + })); + } + { + // Re-anchor protocol-nonce validity lanes to the new account nonces, + // and re-check every lane that relied on a standard pending run the + // protocol pool may have reshuffled on its own. On a reorg only the + // reported account nonces may move an anchor back; the mined + // transactions' nonces are lower bounds and stay forward-only. + let _admission_guard = self.protocol_admission_lock.lock(); + let account_lanes = changed_nonces + .into_iter() + .map(|(address, nonce)| ((address, U256::ZERO), Some(nonce))); + touched_lanes.extend( + self.validity.pool.read().lanes_ahead_of_state().keys().map(|lane| (*lane, None)), + ); + if reorg { + self.sync_validity_lanes(touched_lanes, false); + self.sync_validity_lanes(account_lanes, true); + } else { + touched_lanes.extend(account_lanes); + self.sync_validity_lanes(touched_lanes, false); } } self.expire_due_buckets(now); self.expire_by_block(block_number); self.reconcile_guard(); GuardMetrics::tracked().set(self.guard.read().len() as f64); + let (validity_transactions, validity_bytes) = self.validity_pool_occupancy(); + ValidityPoolMetrics::record_occupancy(validity_transactions, validity_bytes); } fn update_accounts(&self, accounts: Vec) { @@ -1664,7 +2391,7 @@ where } dropped }; - let invalidated = self.remove_dropped_across_pools(dropped); + let invalidated = self.remove_from_stores(dropped, RemoveMode::Exact); GuardMetrics::record_balance_update_invalidations(invalidated.len()); if !invalidated.is_empty() { debug!( @@ -1677,6 +2404,10 @@ where .flat_map(|account| self.protocol_pool.get_transactions_by_sender(account.address)) .map(|transaction| *transaction.hash()) .collect(); + let validity_lanes: Vec<_> = accounts + .iter() + .map(|account| ((account.address, U256::ZERO), Some(account.nonce))) + .collect(); self.protocol_pool.update_accounts(accounts); let discarded: Vec<_> = discard_candidates .into_iter() @@ -1686,6 +2417,10 @@ where for hash in discarded { guard.release(&hash); } + drop(guard); + // The protocol pool re-sorted these senders' runs and took their + // reported nonces as-is; validity lanes behind them follow. + self.sync_validity_lanes(validity_lanes, true); } fn delete_blob(&self, tx: B256) { @@ -1702,6 +2437,10 @@ where } #[derive(Debug)] +/// Event listeners for transactions in a lane store. +/// +/// Bounded listeners follow reth's listener semantics: a listener whose channel is full misses +/// that event but stays subscribed, and is removed only once its receiver is dropped. struct SidecarListeners { by_hash: HashMap>>, all_events: Vec>>, @@ -1777,18 +2516,14 @@ impl SidecarListeners { rx } - fn on_inserted(&mut self, nonce_pool: &TwoDNoncePool, outcome: &InsertOutcome) { + fn on_inserted(&mut self, pool: &TwoDNoncePool, outcome: &InsertOutcome) { let hash = outcome.outcome.hash; - let Some(transaction) = nonce_pool.get(&hash) else { + let Some(transaction) = pool.get(&hash) else { return; }; if let Some(replaced) = &outcome.replaced { - self.broadcast_hash_event(replaced.hash(), TransactionEvent::Replaced(hash)); - self.broadcast_all(FullTransactionEvent::Replaced { - transaction: Arc::clone(replaced), - replaced_by: hash, - }); + self.on_replaced(std::slice::from_ref(replaced), hash); } match &outcome.outcome.state { @@ -1802,11 +2537,41 @@ impl SidecarListeners { } } + self.on_lane_moved(&outcome.anchor); for promoted in &outcome.promoted { self.broadcast_pending_transaction(promoted); } } + /// Publishes that `replaced_by` superseded each of `transactions`. + fn on_replaced(&mut self, transactions: &[Arc>], replaced_by: TxHash) { + for transaction in transactions { + self.broadcast_hash_event(transaction.hash(), TransactionEvent::Replaced(replaced_by)); + self.broadcast_all(FullTransactionEvent::Replaced { + transaction: Arc::clone(transaction), + replaced_by, + }); + } + } + + /// Publishes a lane cursor move: removed transactions are discarded, and + /// the rest of the lane is re-announced where its subpool changed. + fn on_lane_update(&mut self, update: &LaneUpdate) { + self.on_discarded(&update.discarded); + self.on_lane_moved(update); + } + + fn on_lane_moved(&mut self, update: &LaneUpdate) { + for promoted in &update.promoted { + self.broadcast_pending_transaction(promoted); + } + for queued in &update.queued { + let hash = *queued.hash(); + self.broadcast_hash_event(&hash, TransactionEvent::Queued); + self.broadcast_all(FullTransactionEvent::Queued(hash, Some(QueuedReason::NonceGap))); + } + } + fn on_mined(&mut self, transactions: &[Arc>], block_hash: B256) { for transaction in transactions { let hash = *transaction.hash(); @@ -1834,7 +2599,7 @@ impl SidecarListeners { } fn broadcast_all(&mut self, event: FullTransactionEvent) { - self.all_events.retain(|listener| listener.try_send(event.clone()).is_ok()); + Self::send_to_open(&mut self.all_events, &event); } fn broadcast_pending_transaction(&mut self, transaction: &Arc>) { @@ -1846,19 +2611,31 @@ impl SidecarListeners { } fn broadcast_pending(&mut self, transaction: &Arc>) { - self.pending_all.retain(|listener| listener.try_send(*transaction.hash()).is_ok()); + let hash = *transaction.hash(); + Self::send_to_open(&mut self.pending_all, &hash); if transaction.propagate { - self.pending_propagate - .retain(|listener| listener.try_send(*transaction.hash()).is_ok()); + Self::send_to_open(&mut self.pending_propagate, &hash); } } fn broadcast_new(&mut self, event: NewTransactionEvent) { - self.new_all.retain(|listener| listener.try_send(event.clone()).is_ok()); + Self::send_to_open(&mut self.new_all, &event); if event.transaction.propagate { - self.new_propagate.retain(|listener| listener.try_send(event.clone()).is_ok()); + Self::send_to_open(&mut self.new_propagate, &event); } } + + /// Sends `item` to every listener without waiting, dropping only listeners whose receiver is + /// gone. + /// + /// A full channel drops the item but keeps the listener: a consumer that falls behind for a + /// while must not lose its subscription for good, since a merged listener stays open on the + /// protocol side and the consumer would never learn that sidecar events stopped. + fn send_to_open(listeners: &mut Vec>, item: &I) { + listeners.retain(|listener| { + !matches!(listener.try_send(item.clone()), Err(TrySendError::Closed(_))) + }); + } } fn merge_receivers( @@ -1910,10 +2687,10 @@ mod tests { use std::time::{Duration, Instant}; use alloy_consensus::{ - SignableTransaction, Transaction, TxEip1559, + SignableTransaction, Transaction, TxEip1559, TxEip7702, transaction::{Recovered, SignerRecoverable}, }; - use alloy_eips::eip2718::Encodable2718; + use alloy_eips::{eip2718::Encodable2718, eip7702::Authorization}; use alloy_primitives::{Bytes, TxKind, U256}; use alloy_signer::SignerSync; use alloy_signer_local::PrivateKeySigner; @@ -1926,12 +2703,15 @@ mod tests { use base_execution_evm::BaseEvmConfig; use base_test_utils::build_test_genesis_everest; use futures::{StreamExt, future::join_all}; - use reth_primitives_traits::SealedBlock; + use reth_primitives_traits::{InMemorySize, SealedBlock}; use reth_provider::test_utils::{ExtendedAccount, MockEthProvider}; use reth_tasks::Runtime; use reth_transaction_pool::{ - CanonicalStateUpdate, PoolConfig, PoolUpdateKind, PriceBumpConfig, TransactionOrigin, - blobstore::InMemoryBlobStore, error::PoolErrorKind, identifier::TransactionId, + CanonicalStateUpdate, PoolConfig, PoolUpdateKind, PriceBumpConfig, SubPoolLimit, + TransactionOrigin, + blobstore::InMemoryBlobStore, + error::{InvalidPoolTransactionError, PoolErrorKind}, + identifier::TransactionId, validate::EthTransactionValidatorBuilder, }; @@ -2312,7 +3092,7 @@ mod tests { ]; let (batched, pool) = assert_batch_matches_sequential(batch, &[sender.address()]).await; assert!(batched.iter().all(Result::is_ok)); - assert!(pool.nonce_pool.read().contains(&sidecar_hash)); + assert!(pool.sidecar.pool.read().contains(&sidecar_hash)); } /// Adds `batch` one transaction at a time to one fresh pool and as one @@ -2472,7 +3252,7 @@ mod tests { } let over = self_paid_eoa_8130(&sidecar_signer, U256::from(cap + 1), 0, 0, 1_000); let over_hash = *over.hash(); - let mut over_events = sidecar_pool.listeners.write().subscribe_hash(over_hash).0; + let mut over_events = sidecar_pool.sidecar.listeners.write().subscribe_hash(over_hash).0; let error = sidecar_pool .add_transaction(TransactionOrigin::Local, over) .await @@ -2505,7 +3285,7 @@ mod tests { admitted.push(*transaction.hash()); assert!(pool.add_transaction(TransactionOrigin::Local, transaction).await.is_ok()); } - assert!(admitted.iter().all(|hash| pool.nonce_pool.read().contains(hash))); + assert!(admitted.iter().all(|hash| pool.sidecar.pool.read().contains(hash))); let over = self_paid_eoa_8130( &signer, @@ -2634,13 +3414,14 @@ mod tests { let replacement = self_paid_eoa_8130(&signer, U256::from(1), 0, 0, 1_250); let replacement_hash = *replacement.hash(); - let mut replacement_events = pool.listeners.write().subscribe_hash(replacement_hash).0; + let mut replacement_events = + pool.sidecar.listeners.write().subscribe_hash(replacement_hash).0; let validated = pool.validator().validate_transaction(TransactionOrigin::Local, replacement).await; pool.validator().validator().clear_limit_class_cache(); let error = pool - .add_validated_sidecar_transaction(validated, TransactionOrigin::Local) + .add_validated_lane_transaction(PoolStore::Sidecar, validated, TransactionOrigin::Local) .expect_err("stale classification must reject the replacement"); assert!(error.to_string().contains("classification changed during validation")); assert!(pool.get(&replacement_hash).is_none()); @@ -2706,7 +3487,7 @@ mod tests { ); let hash = *transaction.hash(); pool.add_transaction(TransactionOrigin::Local, transaction).await.unwrap(); - assert!(pool.nonce_pool.read().contains(&hash)); + assert!(pool.sidecar.pool.read().contains(&hash)); assert!(pool.guard.read().contains(&hash)); let block = SealedBlock::seal_slow(BaseBlock { @@ -2723,7 +3504,7 @@ mod tests { }); assert!(pool.get(&hash).is_none()); - assert!(!pool.nonce_pool.read().contains(&hash)); + assert!(!pool.sidecar.pool.read().contains(&hash)); assert!(!pool.guard.read().contains(&hash)); } @@ -2766,7 +3547,7 @@ mod tests { }); let hash = *transaction.hash(); let admission = IntegrationPool::admission_for(&transaction).unwrap(); - pool.nonce_pool.write().insert_validated(valid_pool_transaction(transaction), 0).unwrap(); + pool.sidecar.pool.write().insert_validated(valid_pool_transaction(transaction), 0).unwrap(); pool.guard.write().try_admit(admission).unwrap(); pool.update_accounts(vec![ChangedAccount { @@ -2882,6 +3663,55 @@ mod tests { assert!(pool.get(&hash).is_some()); } + #[tokio::test] + async fn received_publication_frees_same_nonce_slot_without_publishing() { + let (pool, client) = build_integration_pool(); + let signer = signer(); + fund(&client, signer.address()); + let predicates = vec![ + ValidityPredicate::BlockNumber { + op: ValidityOperator::LessThanOrEqual, + value: U256::from(101), + }, + ValidityPredicate::FlashblockIndex { + op: ValidityOperator::LessThanOrEqual, + value: U256::from(2), + }, + ]; + let original = self_paid_eoa_8130(&signer, U256::ZERO, 0, 0, 1_000) + .with_validity_predicates(predicates.clone()); + let original_hash = *original.hash(); + pool.add_transaction(TransactionOrigin::Local, original).await.unwrap(); + + // Same nonce and fees as the original; the gas limit only makes it a distinct transaction. + let unbumped = || signed_8130(&signer, U256::ZERO, 0, 0, 1_000, 999_999); + + // A follower observes flashblock 1 of the deadline block: the deadline still holds, so an + // unbumped same-nonce replacement is priced against the original. + assert_eq!(pool.expire_published(101, 1), 0); + let early = unbumped(); + let err = pool.add_transaction(TransactionOrigin::Local, early).await.unwrap_err(); + assert!(matches!(err.kind, PoolErrorKind::ReplacementUnderpriced), "got {err:?}"); + assert!(pool.get(&original_hash).is_some()); + + // Observing the deadline flashblock evicts the original and frees the slot, and a stale + // position received afterwards is ignored. + assert_eq!(pool.expire_published(101, 2), 1); + assert!(pool.get(&original_hash).is_none()); + assert_eq!(pool.expire_published(101, 0), 0); + let replacement = unbumped(); + let replacement_hash = *replacement.hash(); + pool.add_transaction(TransactionOrigin::Local, replacement) + .await + .expect("unbumped replacement is admitted once the deadline flashblock is received"); + assert!(pool.get(&replacement_hash).is_some()); + + let late = self_paid_eoa_8130(&signer, U256::ZERO, 0, 0, 1_001) + .with_validity_predicates(predicates); + let error = pool.add_transaction(TransactionOrigin::Local, late).await.unwrap_err(); + assert!(error.to_string().contains("validity deadline expired"), "{error}"); + } + #[tokio::test] async fn published_position_rejects_late_expired_admission_and_preserves_future_block() { let (pool, client) = build_integration_pool(); @@ -3088,4 +3918,1341 @@ mod tests { assert_eq!(pool.guard.read().contains(&hash), remains_live); } } + + fn build_integration_pool_with_config( + config: PoolConfig, + ) -> (IntegrationPool, MockEthProvider>) { + let (pool, client) = build_integration_pool(); + let ordering = BaseOrdering::default(); + let validator = pool.protocol_pool.validator().clone(); + let blob_store = pool.protocol_pool.blob_store().clone(); + let pool = Pool::new(validator, ordering.clone(), blob_store, config); + (BaseTransactionPool::new(pool, ordering).with_guard_limits(GuardLimits::default()), client) + } + + fn sign_eoa_8130(signer: &PrivateKeySigner, tx: TxEip8130) -> BasePooledTransaction { + let signature = signer.sign_hash_sync(&tx.sender_signature_hash()).unwrap(); + let signed = + Eip8130Signed::new(tx, Bytes::from(signature.as_bytes().to_vec()), Bytes::new()); + let pooled = ConsensusPooledTransaction::Eip8130(signed); + BasePooledTransaction::from_pooled(Recovered::new_unchecked(pooled, signer.address())) + } + + fn is_pending(pool: &IntegrationPool, hash: TxHash) -> bool { + pool.pending_transactions().iter().any(|transaction| *transaction.hash() == hash) + } + + fn set_balance(pool: &IntegrationPool, address: Address, balance: U256) { + pool.update_accounts(vec![ChangedAccount { address, nonce: 0, balance }]); + } + + fn best_hashes(pool: &IntegrationPool) -> Vec { + pool.best_transactions().map(|transaction| *transaction.hash()).collect() + } + + // ----- Validity pool isolation ----- + + /// A block bound none of these tests reach, so the transaction rests until + /// something else removes it. + const RESTING_EXPIRY_BLOCK: u64 = 1_000; + + fn resting_predicates() -> Vec { + vec![ValidityPredicate::BlockNumber { + op: ValidityOperator::LessThanOrEqual, + value: U256::from(RESTING_EXPIRY_BLOCK), + }] + } + + fn validity_1559( + signer: &PrivateKeySigner, + nonce: u64, + max_fee_per_gas: u128, + ) -> BasePooledTransaction { + signed_1559_with_fees(signer, nonce, max_fee_per_gas, 0) + .with_validity_predicates(resting_predicates()) + } + + fn validity_8130( + signer: &PrivateKeySigner, + nonce_key: U256, + nonce_sequence: u64, + max_fee_per_gas: u128, + ) -> BasePooledTransaction { + self_paid_eoa_8130(signer, nonce_key, nonce_sequence, 0, max_fee_per_gas) + .with_validity_predicates(resting_predicates()) + } + + fn funded_signer( + client: &MockEthProvider>, + ) -> PrivateKeySigner { + let signer = signer(); + fund(client, signer.address()); + signer + } + + fn in_protocol_pool(pool: &IntegrationPool, hash: TxHash) -> bool { + pool.protocol_pool.get(&hash).is_some() + } + + fn in_sidecar_pool(pool: &IntegrationPool, hash: TxHash) -> bool { + pool.sidecar.pool.read().contains(&hash) + } + + fn in_validity_pool(pool: &IntegrationPool, hash: TxHash) -> bool { + pool.validity.pool.read().contains(&hash) + } + + fn is_queued(pool: &IntegrationPool, hash: TxHash) -> bool { + pool.queued_transactions().iter().any(|transaction| *transaction.hash() == hash) + } + + /// Asserts `hash` is held by exactly `store` and nothing else. + fn assert_store(pool: &IntegrationPool, hash: TxHash, store: PoolStore) { + assert_eq!(in_protocol_pool(pool, hash), store == PoolStore::Protocol, "protocol {hash}"); + assert_eq!(in_sidecar_pool(pool, hash), store == PoolStore::Sidecar, "sidecar {hash}"); + assert_eq!(in_validity_pool(pool, hash), store == PoolStore::Validity, "validity {hash}"); + assert!(pool.get(&hash).is_some(), "lookup {hash}"); + } + + fn assert_gone(pool: &IntegrationPool, hash: TxHash) { + assert!(pool.get(&hash).is_none(), "{hash} is still pooled"); + assert!(!in_protocol_pool(pool, hash) && !in_sidecar_pool(pool, hash)); + assert!(!in_validity_pool(pool, hash)); + assert!(!pool.guard.read().contains(&hash)); + } + + async fn add(pool: &IntegrationPool, transaction: BasePooledTransaction) -> TxHash { + let hash = *transaction.hash(); + pool.add_transaction(TransactionOrigin::External, transaction).await.unwrap(); + hash + } + + async fn add_err(pool: &IntegrationPool, transaction: BasePooledTransaction) -> PoolErrorKind { + pool.add_transaction(TransactionOrigin::External, transaction).await.unwrap_err().kind + } + + fn commit_block( + pool: &IntegrationPool, + number: u64, + changed_accounts: Vec, + mined_transactions: Vec, + ) { + canonical_update( + pool, + number, + changed_accounts, + mined_transactions, + PoolUpdateKind::Commit, + ); + } + + fn canonical_update( + pool: &IntegrationPool, + number: u64, + changed_accounts: Vec, + mined_transactions: Vec, + update_kind: PoolUpdateKind, + ) { + let block = SealedBlock::seal_slow(BaseBlock { + header: alloy_consensus::Header { + number, + gas_limit: pool.block_info().block_gas_limit, + timestamp: INTEGRATION_POOL_NOW_SECS + number * 2, + ..Default::default() + }, + body: Default::default(), + }); + pool.on_canonical_state_change(CanonicalStateUpdate { + new_tip: &block, + pending_block_base_fee: 0, + pending_block_blob_fee: None, + changed_accounts, + mined_transactions, + update_kind, + }); + } + + fn account_at(address: Address, nonce: u64) -> ChangedAccount { + ChangedAccount { address, nonce, balance: U256::from(1_000_000_000_000_000_000u64) } + } + + #[tokio::test] + async fn every_validity_transaction_kind_lands_in_the_validity_pool() { + let (pool, client) = build_integration_pool(); + let signers: Vec<_> = (0..7).map(|_| funded_signer(&client)).collect(); + let nonce_free_expiry = INTEGRATION_POOL_NOW_MS + 5_000; + + let eip1559 = add(&pool, validity_1559(&signers[0], 0, 1_000)).await; + let key0 = add(&pool, validity_8130(&signers[1], U256::ZERO, 0, 1_000)).await; + let channel = add(&pool, validity_8130(&signers[2], U256::from(7), 0, 1_000)).await; + let nonce_free = add( + &pool, + signed_nonce_free_tx(&signers[3], nonce_free_expiry, 1_000) + .with_validity_predicates(resting_predicates()), + ) + .await; + let plain_1559 = add(&pool, signed_1559(&signers[4], 0)).await; + let plain_channel = + add(&pool, self_paid_eoa_8130(&signers[5], U256::from(7), 0, 0, 1_000)).await; + let plain_nonce_free = + add(&pool, signed_nonce_free_tx(&signers[6], nonce_free_expiry, 1_000)).await; + + for hash in [eip1559, key0, channel, nonce_free] { + assert_store(&pool, hash, PoolStore::Validity); + assert!(is_pending(&pool, hash), "{hash} should be pending"); + } + assert_store(&pool, plain_1559, PoolStore::Protocol); + assert_store(&pool, plain_channel, PoolStore::Sidecar); + assert_store(&pool, plain_nonce_free, PoolStore::Sidecar); + + let expected_bytes: usize = [eip1559, key0, channel, nonce_free] + .iter() + .map(|hash| pool.get(hash).unwrap().transaction.size()) + .sum(); + assert_eq!(pool.validity_pool_occupancy(), (4, expected_bytes)); + let size = pool.pool_size(); + assert_eq!((size.pending, size.queued, size.total), (7, 0, 7)); + assert_eq!(pool.all_transactions().pending.len(), 7); + assert_eq!(best_hashes(&pool).len(), 7); + } + + #[tokio::test] + async fn validity_transaction_extends_the_pending_standard_run() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let standard0 = add(&pool, signed_1559(&signer, 0)).await; + let standard1 = add(&pool, signed_1559(&signer, 1)).await; + let validity2 = add(&pool, validity_1559(&signer, 2, 1_000)).await; + let validity3 = add(&pool, validity_1559(&signer, 3, 1_000)).await; + + assert_store(&pool, validity2, PoolStore::Validity); + assert!(is_pending(&pool, validity2) && is_pending(&pool, validity3)); + assert_eq!(best_hashes(&pool), vec![standard0, standard1, validity2, validity3]); + assert_eq!( + pool.get_highest_consecutive_transaction_by_sender(signer.address(), 0) + .map(|transaction| *transaction.hash()), + Some(validity3) + ); + assert_eq!( + pool.get_highest_transaction_by_sender(signer.address()).map(|tx| *tx.hash()), + Some(validity3) + ); + assert_eq!( + pool.get_transaction_by_sender_and_nonce(signer.address(), 2).map(|tx| *tx.hash()), + Some(validity2) + ); + assert_eq!( + pool.get_pending_transaction_by_sender_and_nonce(signer.address(), 3) + .map(|tx| *tx.hash()), + Some(validity3) + ); + assert_eq!(pool.get_transactions_by_sender(signer.address()).len(), 4); + assert_eq!(pool.get_pending_transactions_by_sender(signer.address()).len(), 4); + } + + #[tokio::test] + async fn validity_transaction_behind_a_gap_waits_for_the_standard_run() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let validity2 = validity_1559(&signer, 2, 1_000); + let validity2_hash = *validity2.hash(); + let mut events = pool + .add_transaction_and_subscribe(TransactionOrigin::External, validity2) + .await + .unwrap(); + assert!(matches!(events.next().await, Some(TransactionEvent::Queued))); + assert!(is_queued(&pool, validity2_hash) && !is_pending(&pool, validity2_hash)); + assert!(best_hashes(&pool).is_empty()); + assert!(pool.get_highest_consecutive_transaction_by_sender(signer.address(), 0).is_none()); + + let standard0 = add(&pool, signed_1559(&signer, 0)).await; + assert!(is_queued(&pool, validity2_hash), "nonce 1 is still missing"); + assert_eq!(best_hashes(&pool), vec![standard0]); + + let standard1 = add(&pool, signed_1559(&signer, 1)).await; + assert!(matches!(events.next().await, Some(TransactionEvent::Pending))); + assert!(is_pending(&pool, validity2_hash)); + assert_eq!(best_hashes(&pool), vec![standard0, standard1, validity2_hash]); + assert_eq!( + pool.get_highest_consecutive_transaction_by_sender(signer.address(), 0) + .map(|tx| *tx.hash()), + Some(validity2_hash) + ); + } + + #[tokio::test] + async fn validity_lane_fills_its_own_gap() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let validity1 = add(&pool, validity_1559(&signer, 1, 1_000)).await; + assert!(is_queued(&pool, validity1)); + let validity0 = add(&pool, validity_1559(&signer, 0, 1_000)).await; + assert!(is_pending(&pool, validity0) && is_pending(&pool, validity1)); + assert_eq!(best_hashes(&pool), vec![validity0, validity1]); + let size = pool.pool_size(); + assert_eq!((size.pending, size.queued), (2, 0)); + } + + #[tokio::test] + async fn standard_transaction_behind_a_resting_validity_transaction_stays_queued() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let validity0 = add(&pool, validity_1559(&signer, 0, 1_000)).await; + let standard1 = add(&pool, signed_1559(&signer, 1)).await; + + assert_store(&pool, standard1, PoolStore::Protocol); + assert!(is_queued(&pool, standard1), "the protocol pool sees a nonce gap"); + let size = pool.pool_size(); + assert_eq!((size.pending, size.queued), (1, 1)); + // The merge still releases it, right behind the validity run it waits for. + assert_eq!(best_hashes(&pool), vec![validity0, standard1]); + + // Dropping the validity transaction leaves the standard one gapped. + pool.remove_transactions(vec![validity0]); + assert_gone(&pool, validity0); + assert!(is_queued(&pool, standard1)); + + let standard0 = add(&pool, signed_1559(&signer, 0)).await; + assert_eq!(best_hashes(&pool), vec![standard0, standard1]); + } + + #[tokio::test] + async fn mining_the_standard_prefix_keeps_the_validity_successor_pending() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let standard0 = add(&pool, signed_1559(&signer, 0)).await; + let validity1 = add(&pool, validity_1559(&signer, 1, 1_000)).await; + let validity2 = add(&pool, validity_1559(&signer, 2, 1_000)).await; + assert!(is_pending(&pool, validity1)); + + commit_block(&pool, 1, vec![account_at(signer.address(), 1)], vec![standard0]); + assert_gone(&pool, standard0); + assert!(is_pending(&pool, validity1) && is_pending(&pool, validity2)); + assert_eq!( + pool.validity.pool.read().lane_cursor(&(signer.address(), U256::ZERO)), + Some(LaneCursor::at(1)) + ); + assert_eq!(best_hashes(&pool), vec![validity1, validity2]); + + // Mining a validity transaction prunes it and advances its own lane. + commit_block(&pool, 2, vec![account_at(signer.address(), 2)], vec![validity1]); + assert_gone(&pool, validity1); + assert!(is_pending(&pool, validity2)); + assert_eq!(pool.validity_pool_occupancy().0, 1); + + // A canonical nonce past the lane drops what it left behind. + commit_block(&pool, 3, vec![account_at(signer.address(), 3)], Vec::new()); + assert_gone(&pool, validity2); + assert_eq!(pool.validity_pool_occupancy(), (0, 0)); + } + + #[tokio::test] + async fn dropping_the_standard_prefix_queues_the_validity_successor() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let standard0 = add(&pool, signed_1559(&signer, 0)).await; + let validity1 = validity_1559(&signer, 1, 1_000); + let validity1_hash = *validity1.hash(); + let mut events = pool + .add_transaction_and_subscribe(TransactionOrigin::External, validity1) + .await + .unwrap(); + assert!(matches!(events.next().await, Some(TransactionEvent::Pending))); + + pool.remove_transactions(vec![standard0]); + assert!(matches!(events.next().await, Some(TransactionEvent::Queued))); + assert!(is_queued(&pool, validity1_hash) && !is_pending(&pool, validity1_hash)); + assert!(best_hashes(&pool).is_empty()); + + let standard0 = add(&pool, signed_1559(&signer, 0)).await; + assert!(matches!(events.next().await, Some(TransactionEvent::Pending))); + assert_eq!(best_hashes(&pool), vec![standard0, validity1_hash]); + } + + #[tokio::test] + async fn removing_standard_descendants_queues_but_keeps_validity_successors() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let standard0 = add(&pool, signed_1559(&signer, 0)).await; + let validity1 = add(&pool, validity_1559(&signer, 1, 1_000)).await; + let standard2 = add(&pool, signed_1559(&signer, 2)).await; + assert!(is_queued(&pool, standard2), "reth sees a gap at nonce 1"); + assert_eq!(best_hashes(&pool), vec![standard0, validity1, standard2]); + + let removed: Vec<_> = pool + .remove_transactions_and_descendants(vec![standard0]) + .iter() + .map(|tx| *tx.hash()) + .collect(); + assert_eq!(removed, vec![standard0, standard2]); + assert_store(&pool, validity1, PoolStore::Validity); + assert!(is_queued(&pool, validity1)); + + // Removing a validity transaction with descendants cascades within its lane. + let validity2 = add(&pool, validity_1559(&signer, 2, 1_000)).await; + let removed: Vec<_> = pool + .remove_transactions_and_descendants(vec![validity1]) + .iter() + .map(|tx| *tx.hash()) + .collect(); + assert_eq!(removed, vec![validity1, validity2]); + assert_eq!(pool.validity_pool_occupancy(), (0, 0)); + } + + #[tokio::test] + async fn balance_drop_that_parks_the_standard_run_queues_validity_successors() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let standard0 = add(&pool, signed_1559(&signer, 0)).await; + let validity1 = add(&pool, validity_1559(&signer, 1, 1_000)).await; + assert_eq!(best_hashes(&pool), vec![standard0, validity1]); + + set_balance(&pool, signer.address(), U256::ZERO); + assert!(is_queued(&pool, standard0), "reth parks the insolvent standard transaction"); + assert!(is_queued(&pool, validity1), "the validity successor follows it"); + assert!(best_hashes(&pool).is_empty()); + + set_balance(&pool, signer.address(), U256::from(1_000_000_000_000_000_000u64)); + assert_eq!(best_hashes(&pool), vec![standard0, validity1]); + } + + #[tokio::test] + async fn protocol_nonce_slot_is_exclusive_across_stores_under_the_price_bump() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let validity = validity_1559(&signer, 0, 1_000); + let validity_hash = *validity.hash(); + let mut validity_events = pool + .add_transaction_and_subscribe(TransactionOrigin::External, validity) + .await + .unwrap(); + assert!(matches!(validity_events.next().await, Some(TransactionEvent::Pending))); + + // The unbumped standard copy is the same signed transaction, so it is a + // duplicate rather than a replacement. + let error = add_err(&pool, signed_1559_with_fees(&signer, 0, 1_000, 0)).await; + assert!(matches!(error, PoolErrorKind::AlreadyImported), "{error:?}"); + for underpriced in [1_001, 1_099] { + let error = add_err(&pool, signed_1559_with_fees(&signer, 0, underpriced, 0)).await; + assert!(matches!(error, PoolErrorKind::ReplacementUnderpriced), "{error:?}"); + assert_store(&pool, validity_hash, PoolStore::Validity); + } + + let standard = add(&pool, signed_1559_with_fees(&signer, 0, 1_100, 0)).await; + assert_store(&pool, standard, PoolStore::Protocol); + assert_gone(&pool, validity_hash); + assert!(matches!( + validity_events.next().await, + Some(TransactionEvent::Replaced(replaced_by)) if replaced_by == standard + )); + assert_eq!(pool.block_expiry.read().len(), 0, "the replaced expiry entry is dropped"); + + let error = add_err(&pool, validity_1559(&signer, 0, 1_200)).await; + assert!(matches!(error, PoolErrorKind::ReplacementUnderpriced), "{error:?}"); + assert_store(&pool, standard, PoolStore::Protocol); + + let bumped = add(&pool, validity_1559(&signer, 0, 1_210)).await; + assert_store(&pool, bumped, PoolStore::Validity); + assert_gone(&pool, standard); + assert_eq!(pool.block_expiry.read().len(), 1); + assert_eq!(best_hashes(&pool), vec![bumped]); + + let error = add_err(&pool, validity_1559(&signer, 0, 1_210)).await; + assert!(matches!(error, PoolErrorKind::AlreadyImported), "{error:?}"); + let error = add_err(&pool, signed_1559_with_fees(&signer, 0, 1_100, 0)).await; + assert!(matches!(error, PoolErrorKind::ReplacementUnderpriced), "{error:?}"); + let size = pool.pool_size(); + assert_eq!((size.pending, size.total), (1, 1)); + } + + #[tokio::test] + async fn cross_store_replacement_cuts_the_validity_lane_behind_the_newcomer() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let standard0 = add(&pool, signed_1559(&signer, 0)).await; + let validity1 = add(&pool, validity_1559(&signer, 1, 1_000)).await; + let validity2 = add(&pool, validity_1559(&signer, 2, 1_000)).await; + assert_eq!(best_hashes(&pool), vec![standard0, validity1, validity2]); + + // A standard transaction replacing validity nonce 1 leaves nonce 2 pending + // behind the new standard run. + let standard1 = add(&pool, signed_1559_with_fees(&signer, 1, 1_100, 0)).await; + assert_gone(&pool, validity1); + assert!(is_pending(&pool, validity2)); + assert_eq!(best_hashes(&pool), vec![standard0, standard1, validity2]); + + // A validity transaction replacing standard nonce 0 parks standard nonce 1 + // in reth and, with it, validity nonce 2. The merge still bridges the + // standard transaction behind the validity head; validity nonce 2 waits + // for the next canonical update. + let validity0 = add(&pool, validity_1559(&signer, 0, 1_100)).await; + assert_gone(&pool, standard0); + assert!(is_queued(&pool, standard1)); + assert!(is_queued(&pool, validity2)); + assert_eq!(best_hashes(&pool), vec![validity0, standard1]); + } + + #[tokio::test] + async fn channel_validity_transactions_follow_the_sidecar_run() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let channel = U256::from(3); + let lane = (signer.address(), channel); + + let sidecar0 = add(&pool, self_paid_eoa_8130(&signer, channel, 0, 0, 1_000)).await; + let validity1 = add(&pool, validity_8130(&signer, channel, 1, 1_000)).await; + assert_store(&pool, sidecar0, PoolStore::Sidecar); + assert_store(&pool, validity1, PoolStore::Validity); + assert!(is_pending(&pool, validity1)); + assert_eq!( + pool.validity.pool.read().lane_cursor(&lane), + Some(LaneCursor { state_nonce: 0, next_nonce: 1 }) + ); + + // A sidecar successor behind the validity transaction is gapped. + let sidecar2 = add(&pool, self_paid_eoa_8130(&signer, channel, 2, 0, 1_000)).await; + assert!(is_queued(&pool, sidecar2)); + assert_eq!(best_hashes(&pool), vec![sidecar0, validity1, sidecar2]); + + // Same-slot admission across the sidecar boundary follows the price bump. + let error = add_err(&pool, self_paid_eoa_8130(&signer, channel, 1, 0, 1_050)).await; + assert!(matches!(error, PoolErrorKind::ReplacementUnderpriced), "{error:?}"); + let sidecar1 = add(&pool, self_paid_eoa_8130(&signer, channel, 1, 0, 1_100)).await; + assert_gone(&pool, validity1); + assert_eq!(best_hashes(&pool), vec![sidecar0, sidecar1, sidecar2]); + assert!(pool.validity.pool.read().lane_cursor(&lane).is_none()); + + // And back: a validity transaction takes the sidecar head. + let validity0 = add(&pool, validity_8130(&signer, channel, 0, 1_100)).await; + assert_gone(&pool, sidecar0); + assert_store(&pool, validity0, PoolStore::Validity); + assert!(is_queued(&pool, sidecar1) && is_queued(&pool, sidecar2)); + assert_eq!(best_hashes(&pool), vec![validity0, sidecar1, sidecar2]); + } + + #[tokio::test] + async fn a_cross_store_replacement_evicted_on_insert_keeps_the_incumbent_reserved() { + let (pool, client) = build_integration_pool(); + let pool = pool.with_validity_pool_limits(ValidityPoolLimits::inherit().with_max_txs(1)); + let signer = funded_signer(&client); + let channel = U256::from(3); + let incumbent = add(&pool, self_paid_eoa_8130(&signer, channel, 0, 0, 1_000)).await; + assert!(pool.guard.read().contains(&incumbent)); + let resting = add(&pool, validity_1559(&funded_signer(&client), 0, 5_000)).await; + + // The bumped validity copy outbids the sidecar incumbent but is the + // cheapest member of the full validity pool, so it is evicted on insert. + let newcomer = validity_8130(&signer, channel, 0, 1_100); + let newcomer_hash = *newcomer.hash(); + let error = add_err(&pool, newcomer).await; + assert!(matches!(error, PoolErrorKind::DiscardedOnInsert), "{error:?}"); + assert_store(&pool, incumbent, PoolStore::Sidecar); + assert_store(&pool, resting, PoolStore::Validity); + assert!(pool.guard.read().contains(&incumbent), "the incumbent keeps its reservation"); + assert!(!pool.guard.read().contains(&newcomer_hash)); + + // A replacement that survives eviction takes over the reservation. + let replacement = add(&pool, validity_8130(&signer, channel, 0, 6_000)).await; + assert_store(&pool, replacement, PoolStore::Validity); + assert_gone(&pool, incumbent); + assert!(pool.guard.read().contains(&replacement)); + assert!(!pool.guard.read().contains(&incumbent)); + } + + #[tokio::test] + async fn mined_sidecar_head_promotes_the_validity_successor_on_its_channel() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let channel = U256::from(3); + let sidecar0 = add(&pool, self_paid_eoa_8130(&signer, channel, 0, 0, 1_000)).await; + let validity1 = add(&pool, validity_8130(&signer, channel, 1, 1_000)).await; + let validity2 = add(&pool, validity_8130(&signer, channel, 2, 1_000)).await; + + commit_block(&pool, 1, vec![account_at(signer.address(), 0)], vec![sidecar0]); + assert_gone(&pool, sidecar0); + assert!(is_pending(&pool, validity1) && is_pending(&pool, validity2)); + assert_eq!( + pool.validity.pool.read().lane_cursor(&(signer.address(), channel)), + Some(LaneCursor::at(1)) + ); + } + + #[tokio::test] + async fn nonce_free_validity_transactions_are_isolated_and_expire() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let expiry = INTEGRATION_POOL_NOW_MS + 5_000; + let validity = signed_nonce_free_tx(&signer, expiry, 1_000) + .with_validity_predicates(resting_predicates()); + let validity_hash = *validity.hash(); + let mut events = pool + .add_transaction_and_subscribe(TransactionOrigin::External, validity) + .await + .unwrap(); + assert!(matches!(events.next().await, Some(TransactionEvent::Pending))); + assert_store(&pool, validity_hash, PoolStore::Validity); + assert!(pool.guard.read().contains(&validity_hash)); + assert_eq!(best_hashes(&pool), vec![validity_hash]); + + // Resubmission is rejected from the validity pool, not duplicated in the sidecar. + let error = add_err( + &pool, + signed_nonce_free_tx(&signer, expiry, 1_000) + .with_validity_predicates(resting_predicates()), + ) + .await; + assert!(matches!(error, PoolErrorKind::AlreadyImported), "{error:?}"); + + // The same replay id without predicates is the sidecar's slot: it must + // bump the validity pool's copy, which it then replaces. + let error = add_err(&pool, signed_nonce_free_tx(&signer, expiry, 1_050)).await; + assert!(matches!(error, PoolErrorKind::ReplacementUnderpriced), "{error:?}"); + let plain = add(&pool, signed_nonce_free_tx(&signer, expiry, 1_100)).await; + assert!(matches!( + events.next().await, + Some(TransactionEvent::Replaced(replaced_by)) if replaced_by == plain + )); + assert_gone(&pool, validity_hash); + assert_store(&pool, plain, PoolStore::Sidecar); + + // And back into the validity pool under the same rule. + let bumped = signed_nonce_free_tx(&signer, expiry, 1_210) + .with_validity_predicates(resting_predicates()); + let bumped_hash = *bumped.hash(); + let mut events = + pool.add_transaction_and_subscribe(TransactionOrigin::External, bumped).await.unwrap(); + assert!(matches!(events.next().await, Some(TransactionEvent::Pending))); + assert_gone(&pool, plain); + assert_store(&pool, bumped_hash, PoolStore::Validity); + + // Time passes beyond `valid_before`: canonical maintenance expires it. + commit_block(&pool, 5, Vec::new(), Vec::new()); + assert!(matches!(events.next().await, Some(TransactionEvent::Discarded))); + assert_gone(&pool, bumped_hash); + assert_eq!(pool.validity_pool_occupancy(), (0, 0)); + } + + #[tokio::test] + async fn a_standard_flood_cannot_evict_validity_transactions() { + let config = + PoolConfig { pending_limit: SubPoolLimit::new(2, usize::MAX), ..PoolConfig::default() }; + let (pool, client) = build_integration_pool_with_config(config); + let mut validity = Vec::new(); + for _ in 0..2 { + let signer = funded_signer(&client); + validity.push(add(&pool, validity_1559(&signer, 0, 1_000)).await); + } + + let mut standard = Vec::new(); + for _ in 0..6 { + let signer = funded_signer(&client); + let transaction = signed_1559_with_fees(&signer, 0, 10_000, 5_000); + standard.push(*transaction.hash()); + // reth discards the worst standard transaction, which may be the newcomer. + let _ = pool.add_transaction(TransactionOrigin::External, transaction).await; + } + + for hash in &validity { + assert_store(&pool, *hash, PoolStore::Validity); + assert!(is_pending(&pool, *hash)); + } + let surviving_standard = + standard.iter().filter(|hash| in_protocol_pool(&pool, **hash)).count(); + assert!(surviving_standard <= 2, "reth enforces its own pending limit"); + assert_eq!(pool.validity_pool_occupancy().0, 2); + } + + #[tokio::test] + async fn a_mixed_batch_keeps_each_senders_submission_order() { + let (pool, client) = build_integration_pool(); + let standalone = funded_signer(&client); + let bridged = funded_signer(&client); + let replaced = funded_signer(&client); + let batch = vec![ + signed_1559(&standalone, 0), + validity_1559(&bridged, 0, 1_000), + signed_1559(&bridged, 1), + validity_1559(&replaced, 0, 1_000), + // Replaces the validity transaction before it only if admitted after it. + signed_1559_with_fees(&replaced, 0, 2_000, 0), + ]; + let hashes: Vec<_> = batch.iter().map(|transaction| *transaction.hash()).collect(); + + let results = pool.add_transactions(TransactionOrigin::External, batch).await; + let admitted: Vec<_> = + results.into_iter().map(|result| result.expect("admitted").hash).collect(); + assert_eq!(admitted, hashes, "results follow the submission order"); + + assert_store(&pool, hashes[0], PoolStore::Protocol); + assert!(is_pending(&pool, hashes[0])); + assert_store(&pool, hashes[1], PoolStore::Validity); + assert_store(&pool, hashes[2], PoolStore::Protocol); + assert!(is_queued(&pool, hashes[2])); + assert_gone(&pool, hashes[3]); + assert_store(&pool, hashes[4], PoolStore::Protocol); + } + + #[tokio::test] + async fn a_validity_flood_cannot_evict_standard_transactions() { + let config = + PoolConfig { pending_limit: SubPoolLimit::new(2, usize::MAX), ..PoolConfig::default() }; + let (pool, client) = build_integration_pool_with_config(config); + let mut standard = Vec::new(); + for _ in 0..2 { + let signer = funded_signer(&client); + standard.push(add(&pool, signed_1559(&signer, 0)).await); + } + + let mut validity = Vec::new(); + for fee in [5_000, 6_000, 7_000, 8_000] { + let signer = funded_signer(&client); + validity.push(add(&pool, validity_1559(&signer, 0, fee)).await); + } + // A newcomer cheaper than everything resting is refused outright. + let cheap_signer = funded_signer(&client); + let error = add_err(&pool, validity_1559(&cheap_signer, 0, 1_000)).await; + assert!(matches!(error, PoolErrorKind::DiscardedOnInsert), "{error:?}"); + + for hash in &standard { + assert_store(&pool, *hash, PoolStore::Protocol); + assert!(is_pending(&pool, *hash)); + } + assert_eq!(pool.validity_pool_occupancy().0, 2); + assert!(!in_validity_pool(&pool, validity[0]) && !in_validity_pool(&pool, validity[1])); + assert!(in_validity_pool(&pool, validity[2]) && in_validity_pool(&pool, validity[3])); + assert_gone(&pool, validity[0]); + let size = pool.pool_size(); + assert_eq!((size.pending, size.total), (4, 4)); + } + + #[tokio::test] + async fn validity_pool_limits_override_the_inherited_pending_limit() { + let (pool, client) = build_integration_pool(); + let pool = pool.with_validity_pool_limits(ValidityPoolLimits::inherit().with_max_txs(1)); + let incumbent_signer = funded_signer(&client); + let incumbent = add(&pool, validity_1559(&incumbent_signer, 0, 2_000)).await; + + let cheap_signer = funded_signer(&client); + let error = add_err(&pool, validity_1559(&cheap_signer, 0, 1_000)).await; + assert!(matches!(error, PoolErrorKind::DiscardedOnInsert), "{error:?}"); + assert_store(&pool, incumbent, PoolStore::Validity); + + let rich_signer = funded_signer(&client); + let rich = add(&pool, validity_1559(&rich_signer, 0, 3_000)).await; + assert_store(&pool, rich, PoolStore::Validity); + assert_gone(&pool, incumbent); + assert_eq!(pool.validity_pool_occupancy().0, 1); + + // The protocol pool keeps its own, much larger, limit. + for _ in 0..3 { + let signer = funded_signer(&client); + add(&pool, signed_1559(&signer, 0)).await; + } + assert_eq!(pool.pool_size().pending, 4); + } + + #[tokio::test] + async fn validity_pool_byte_limit_evicts_the_cheapest_transaction() { + let (pool, client) = build_integration_pool(); + let probe = validity_1559(&funded_signer(&client), 0, 1_000); + let one_transaction = probe.size(); + let pool = pool.with_validity_pool_limits( + ValidityPoolLimits::inherit().with_max_size(one_transaction), + ); + let cheap = add(&pool, probe).await; + let rich = add(&pool, validity_1559(&funded_signer(&client), 0, 2_000)).await; + assert_gone(&pool, cheap); + assert_store(&pool, rich, PoolStore::Validity); + assert_eq!(pool.validity_pool_occupancy(), (1, one_transaction)); + } + + #[tokio::test] + async fn validity_pool_account_slots_cap_transactions_per_sender() { + let (pool, client) = build_integration_pool(); + let pool = + pool.with_validity_pool_limits(ValidityPoolLimits::inherit().with_max_account_slots(2)); + let signer = funded_signer(&client); + add(&pool, validity_1559(&signer, 1, 1_000)).await; + add(&pool, validity_1559(&signer, 2, 1_000)).await; + let error = add_err(&pool, validity_1559(&signer, 3, 1_000)).await; + assert!( + matches!(error, PoolErrorKind::SpammerExceededCapacity(address) if address == signer.address()), + "{error:?}" + ); + // Filling the lane head is exempt, as it is in reth: it unblocks what is held. + let head = add(&pool, validity_1559(&signer, 0, 1_000)).await; + assert!(is_pending(&pool, head)); + assert_eq!(pool.validity_pool_occupancy().0, 3); + assert_eq!(pool.pool_size().pending, 3); + } + + #[tokio::test] + async fn validity_pool_eviction_drops_the_block_expiry_entry() { + let (pool, client) = build_integration_pool(); + let pool = pool.with_validity_pool_limits(ValidityPoolLimits::inherit().with_max_txs(1)); + let mut previous = None; + for fee in 1..=5u128 { + let signer = funded_signer(&client); + let hash = add(&pool, validity_1559(&signer, 0, fee * 1_000)).await; + if let Some(previous) = previous { + assert_gone(&pool, previous); + } + previous = Some(hash); + assert_eq!(pool.validity_pool_occupancy().0, 1); + assert_eq!(pool.block_expiry.read().len(), 1, "evicted entries leave the index"); + } + + // A newcomer discarded on insert leaves no entry behind either. + let cheap_signer = funded_signer(&client); + let error = add_err(&pool, validity_1559(&cheap_signer, 0, 500)).await; + assert!(matches!(error, PoolErrorKind::DiscardedOnInsert), "{error:?}"); + assert_eq!(pool.block_expiry.read().len(), 1); + } + + #[tokio::test] + async fn block_expiry_frees_the_validity_pool_slot() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let validity = signed_1559(&signer, 0).with_validity_predicates(vec![ + ValidityPredicate::BlockNumber { + op: ValidityOperator::LessThanOrEqual, + value: U256::from(5), + }, + ]); + let validity_hash = *validity.hash(); + let mut events = pool + .add_transaction_and_subscribe(TransactionOrigin::External, validity) + .await + .unwrap(); + assert!(matches!(events.next().await, Some(TransactionEvent::Pending))); + assert_store(&pool, validity_hash, PoolStore::Validity); + assert_eq!(pool.block_expiry.read().len(), 1); + + commit_block(&pool, 4, Vec::new(), Vec::new()); + assert_store(&pool, validity_hash, PoolStore::Validity); + commit_block(&pool, 5, Vec::new(), Vec::new()); + assert!(matches!(events.next().await, Some(TransactionEvent::Discarded))); + assert_gone(&pool, validity_hash); + assert_eq!(pool.block_expiry.read().len(), 0); + assert_eq!(pool.validity_pool_occupancy(), (0, 0)); + + // The freed slot admits an unbumped standard transaction. + let standard = add(&pool, signed_1559(&signer, 0)).await; + assert_store(&pool, standard, PoolStore::Protocol); + } + + #[tokio::test] + async fn published_flashblock_expires_validity_pool_members() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let validity = signed_1559(&signer, 0).with_validity_predicates(vec![ + ValidityPredicate::BlockNumber { + op: ValidityOperator::LessThanOrEqual, + value: U256::from(100), + }, + ValidityPredicate::FlashblockIndex { + op: ValidityOperator::LessThanOrEqual, + value: U256::from(1), + }, + ]); + let validity_hash = add(&pool, validity).await; + let standard1 = add(&pool, signed_1559(&signer, 1)).await; + assert_store(&pool, validity_hash, PoolStore::Validity); + assert!(is_queued(&pool, standard1)); + + assert_eq!(publish(&pool, 100, 1), 1); + assert_gone(&pool, validity_hash); + assert!(is_queued(&pool, standard1), "the standard successor still waits for nonce 0"); + let standard0 = add(&pool, signed_1559(&signer, 0)).await; + assert_eq!(best_hashes(&pool), vec![standard0, standard1]); + } + + #[tokio::test] + async fn update_accounts_re_anchors_validity_lanes_to_the_new_nonce() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let validity0 = add(&pool, validity_1559(&signer, 0, 1_000)).await; + let validity1 = add(&pool, validity_1559(&signer, 1, 1_000)).await; + let standard2 = add(&pool, signed_1559(&signer, 2)).await; + assert!(is_queued(&pool, standard2)); + + pool.update_accounts(vec![account_at(signer.address(), 1)]); + assert_gone(&pool, validity0); + assert!(is_pending(&pool, validity1)); + assert!(is_queued(&pool, standard2)); + + pool.update_accounts(vec![account_at(signer.address(), 2)]); + assert_gone(&pool, validity1); + assert_eq!(best_hashes(&pool), vec![standard2]); + assert_eq!(pool.validity_pool_occupancy(), (0, 0)); + } + + #[tokio::test] + async fn remove_transactions_by_sender_clears_every_store() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let other = funded_signer(&client); + let standard = add(&pool, signed_1559(&signer, 0)).await; + let validity = add(&pool, validity_1559(&signer, 1, 1_000)).await; + let channel_validity = add(&pool, validity_8130(&signer, U256::from(9), 0, 1_000)).await; + let sidecar = add(&pool, self_paid_eoa_8130(&signer, U256::from(8), 0, 0, 1_000)).await; + let others_validity = add(&pool, validity_1559(&other, 0, 1_000)).await; + + let removed: std::collections::HashSet<_> = pool + .remove_transactions_by_sender(signer.address()) + .iter() + .map(|tx| *tx.hash()) + .collect(); + assert_eq!( + removed, + std::collections::HashSet::from([standard, validity, channel_validity, sidecar]) + ); + for hash in [standard, validity, channel_validity, sidecar] { + assert_gone(&pool, hash); + } + assert_store(&pool, others_validity, PoolStore::Validity); + assert_eq!(pool.validity_pool_occupancy().0, 1); + } + + #[tokio::test] + async fn best_transactions_merge_ranks_across_stores_and_parks_invalid_lanes() { + let (pool, client) = build_integration_pool(); + let standard_signer = funded_signer(&client); + let validity_signer = funded_signer(&client); + let cheap_standard = add(&pool, signed_1559_with_fees(&standard_signer, 0, 1_000, 1)).await; + let rich_validity = add(&pool, validity_1559(&validity_signer, 0, 1_000)).await; + let validity_successor = add(&pool, validity_1559(&validity_signer, 1, 1_000)).await; + // `validity_1559` carries no priority fee, so the ordering ranks it below + // a tipping standard transaction only when the tip is higher. + let tipping_standard = + add(&pool, signed_1559_with_fees(&standard_signer, 1, 1_000, 500)).await; + + let order = best_hashes(&pool); + assert_eq!(order.len(), 4); + assert!( + order.iter().position(|hash| *hash == cheap_standard) + < order.iter().position(|hash| *hash == tipping_standard), + "nonce order within a sender is preserved: {order:?}" + ); + assert!( + order.iter().position(|hash| *hash == rich_validity) + < order.iter().position(|hash| *hash == validity_successor) + ); + + // Marking a validity transaction invalid parks the rest of its lane. + let mut best = pool.best_transactions(); + let mut yielded = Vec::new(); + while let Some(transaction) = best.next() { + if *transaction.hash() == rich_validity { + best.mark_invalid( + &transaction, + InvalidPoolTransactionError::Consensus( + reth_primitives_traits::transaction::error::InvalidTransactionError::TxTypeNotSupported, + ), + ); + } + yielded.push(*transaction.hash()); + } + assert!(yielded.contains(&rich_validity)); + assert!(!yielded.contains(&validity_successor)); + assert!(yielded.contains(&cheap_standard) && yielded.contains(&tipping_standard)); + } + + /// The reviewer's case: a validity successor that outbids the standard + /// transaction in front of it must still come out behind it. + #[tokio::test] + async fn validity_successor_outbidding_the_standard_run_waits_for_it() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let other = funded_signer(&client); + let standard0 = add(&pool, signed_1559_with_fees(&signer, 0, 1_000, 1)).await; + let rich_validity1 = add( + &pool, + signed_1559_with_fees(&signer, 1, 1_000, 900) + .with_validity_predicates(resting_predicates()), + ) + .await; + let rich_validity2 = add( + &pool, + signed_1559_with_fees(&signer, 2, 1_000, 800) + .with_validity_predicates(resting_predicates()), + ) + .await; + let unrelated = add(&pool, signed_1559_with_fees(&other, 0, 1_000, 500)).await; + assert!(is_pending(&pool, rich_validity1) && is_pending(&pool, rich_validity2)); + + // Priority alone would rank the validity transactions first. + assert_eq!( + best_hashes(&pool), + vec![unrelated, standard0, rich_validity1, rich_validity2], + "the standard predecessor is released before its validity successors" + ); + + // Once the builder drops the standard predecessor, its successors stay parked. + let mut best = pool.best_transactions(); + let mut yielded = Vec::new(); + while let Some(transaction) = best.next() { + if *transaction.hash() == standard0 { + best.mark_invalid(&transaction, InvalidPoolTransactionError::Underpriced); + } + yielded.push(*transaction.hash()); + } + assert_eq!(yielded, vec![unrelated, standard0]); + + // Lanes anchored at canonical state need no predecessor and keep their own order. + pool.remove_transactions(vec![standard0]); + let validity0 = add( + &pool, + signed_1559_with_fees(&signer, 0, 1_000, 1) + .with_validity_predicates(resting_predicates()), + ) + .await; + assert_eq!(best_hashes(&pool), vec![unrelated, validity0, rich_validity1, rich_validity2]); + } + + /// The builder's case: a predicate-delayed validity transaction must not + /// keep the user's standard successors out of the block once it executes. + #[tokio::test] + async fn standard_descendants_behind_a_validity_run_are_released_after_it() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let other = funded_signer(&client); + let validity0 = add( + &pool, + signed_1559_with_fees(&signer, 0, 1_000, 100) + .with_validity_predicates(resting_predicates()), + ) + .await; + let standard1 = add(&pool, signed_1559_with_fees(&signer, 1, 1_000, 900)).await; + let standard2 = add(&pool, signed_1559_with_fees(&signer, 2, 1_000, 950)).await; + // Nonce 3 is missing: nonce 4 is not part of the bridged run. + let standard4 = add(&pool, signed_1559_with_fees(&signer, 4, 1_000, 999)).await; + let unrelated = add(&pool, signed_1559_with_fees(&other, 0, 1_000, 500)).await; + assert!(is_queued(&pool, standard1) && is_queued(&pool, standard2)); + + assert_eq!(best_hashes(&pool), vec![unrelated, validity0, standard1, standard2]); + + // A dropped validity head takes its standard descendants with it. + let mut best = pool.best_transactions(); + let mut yielded = Vec::new(); + while let Some(transaction) = best.next() { + if *transaction.hash() == validity0 { + best.mark_invalid(&transaction, InvalidPoolTransactionError::Underpriced); + } + yielded.push(*transaction.hash()); + } + assert_eq!(yielded, vec![unrelated, validity0]); + + // A validity run that is itself gapped releases nothing behind it. + pool.remove_transactions(vec![validity0]); + let validity3 = add(&pool, validity_1559(&signer, 3, 1_000)).await; + assert!(is_queued(&pool, validity3) && is_queued(&pool, standard4)); + assert_eq!(best_hashes(&pool), vec![unrelated]); + } + + /// The reviewer's second case: a waiting validity head must not starve + /// the validity transactions of every other sender. + #[tokio::test] + async fn a_waiting_validity_head_does_not_starve_other_validity_senders() { + let (pool, client) = build_integration_pool(); + let waiting = funded_signer(&client); + let others: Vec<_> = (0..3).map(|_| funded_signer(&client)).collect(); + let standard0 = add(&pool, signed_1559_with_fees(&waiting, 0, 1_000, 1)).await; + let waiting_validity = add( + &pool, + signed_1559_with_fees(&waiting, 1, 1_000, 900) + .with_validity_predicates(resting_predicates()), + ) + .await; + let mut other_validity = Vec::new(); + for (index, other) in others.iter().enumerate() { + other_validity.push( + add( + &pool, + signed_1559_with_fees(other, 0, 1_000, 500 - index as u128) + .with_validity_predicates(resting_predicates()), + ) + .await, + ); + } + + let mut expected = other_validity.clone(); + expected.extend([standard0, waiting_validity]); + assert_eq!(best_hashes(&pool), expected); + + // A builder that stops before the low-tip predecessor still gets every other lane. + let first_three: Vec<_> = + pool.best_transactions().take(3).map(|transaction| *transaction.hash()).collect(); + assert_eq!(first_three, other_validity); + } + + /// The standard descendants may be in their store before the validity + /// transaction arrives, and the validity lane may come and go. + #[tokio::test] + async fn standard_descendants_admitted_before_the_validity_run_are_bridged_too() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let standard1 = add(&pool, signed_1559(&signer, 1)).await; + let standard2 = add(&pool, signed_1559(&signer, 2)).await; + assert!(best_hashes(&pool).is_empty()); + + let validity0 = add(&pool, validity_1559(&signer, 0, 1_000)).await; + assert_eq!(best_hashes(&pool), vec![validity0, standard1, standard2]); + + // A fresh validity lane meets the waiting descendants again. + pool.remove_transactions(vec![validity0]); + assert!(best_hashes(&pool).is_empty()); + let validity0 = add(&pool, validity_1559(&signer, 0, 1_100)).await; + assert_eq!(best_hashes(&pool), vec![validity0, standard1, standard2]); + + // The shared flag follows the standard store's hold on the lane: it is + // dropped once nothing standard is left and set again by the next admission. + let lane = (signer.address(), U256::ZERO); + assert!(pool.validity.pool.read().is_shared_lane(&lane)); + pool.remove_transactions(vec![standard1, standard2]); + assert!(!pool.validity.pool.read().is_shared_lane(&lane)); + assert_eq!(best_hashes(&pool), vec![validity0]); + let standard1 = add(&pool, signed_1559(&signer, 1)).await; + let standard2 = add(&pool, signed_1559(&signer, 2)).await; + assert!(pool.validity.pool.read().is_shared_lane(&lane)); + assert_eq!(best_hashes(&pool), vec![validity0, standard1, standard2]); + + // Mining the validity transaction promotes the descendants in reth itself. + commit_block(&pool, 1, vec![account_at(signer.address(), 1)], vec![validity0]); + assert_gone(&pool, validity0); + assert!(is_pending(&pool, standard1) && is_pending(&pool, standard2)); + assert_eq!(best_hashes(&pool), vec![standard1, standard2]); + } + + #[tokio::test] + async fn channel_validity_successor_waits_for_the_sidecar_head() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let channel = U256::from(4); + let sidecar0 = add(&pool, self_paid_eoa_8130(&signer, channel, 0, 0, 1_000)).await; + let rich_validity1 = add( + &pool, + signed_8130(&signer, channel, 1, 0, 1_000, 1_000_000) + .with_validity_predicates(resting_predicates()), + ) + .await; + // Give the validity successor the higher priority through a tip. + let rich_validity1 = { + pool.remove_transactions(vec![rich_validity1]); + let tipped = TxEip8130 { + chain_id: test_chain_id(), + sender: None, + nonce_key: channel, + nonce_sequence: 1, + valid_after: 0, + valid_before: 0, + max_priority_fee_per_gas: 900, + max_fee_per_gas: 1_000, + gas_limit: 1_000_000, + account_changes: Vec::new(), + calls: Vec::new(), + metadata: Bytes::new(), + payer: None, + }; + add( + &pool, + sign_eoa_8130(&signer, tipped).with_validity_predicates(resting_predicates()), + ) + .await + }; + assert!(is_pending(&pool, rich_validity1)); + assert_eq!(best_hashes(&pool), vec![sidecar0, rich_validity1]); + } + + #[tokio::test] + async fn pending_listener_announces_validity_transactions() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let mut pending = pool.pending_transactions_listener(); + let mut new_transactions = pool.new_transactions_listener(); + let validity = add(&pool, validity_1559(&signer, 0, 1_000)).await; + assert_eq!(pending.recv().await, Some(validity)); + let event = new_transactions.recv().await.unwrap(); + assert_eq!(*event.transaction.hash(), validity); + assert!(matches!(event.subpool, SubPool::Pending)); + } + + #[tokio::test] + async fn reorg_mined_validity_transaction_is_pruned_and_lane_follows() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let validity0 = add(&pool, validity_1559(&signer, 0, 1_000)).await; + let validity1 = add(&pool, validity_1559(&signer, 1, 1_000)).await; + + canonical_update( + &pool, + 1, + vec![account_at(signer.address(), 1)], + vec![validity0], + PoolUpdateKind::Reorg, + ); + assert_gone(&pool, validity0); + assert!(is_pending(&pool, validity1)); + assert!(pool.published_flashblock.read().is_none()); + } + + #[tokio::test] + async fn a_reorg_lowering_the_account_nonce_moves_the_validity_lane_back() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let lane = (signer.address(), U256::ZERO); + let validity2 = add(&pool, validity_1559(&signer, 2, 1_000)).await; + canonical_update( + &pool, + 1, + vec![account_at(signer.address(), 2)], + vec![], + PoolUpdateKind::Commit, + ); + assert!(is_pending(&pool, validity2)); + + // A commit never moves the anchor back. + canonical_update( + &pool, + 2, + vec![account_at(signer.address(), 1)], + vec![], + PoolUpdateKind::Commit, + ); + assert_eq!(pool.validity.pool.read().lane_cursor(&lane), Some(LaneCursor::at(2))); + + // The new branch never mined nonces 0 and 1: the lane waits behind the gap. + canonical_update( + &pool, + 1, + vec![account_at(signer.address(), 0)], + vec![], + PoolUpdateKind::Reorg, + ); + assert_eq!(pool.validity.pool.read().lane_cursor(&lane), Some(LaneCursor::at(0))); + assert!(is_queued(&pool, validity2)); + assert!(best_hashes(&pool).is_empty()); + + // Re-injected standard transactions close the gap. + let standard0 = add(&pool, signed_1559(&signer, 0)).await; + let standard1 = add(&pool, signed_1559(&signer, 1)).await; + assert!(is_pending(&pool, validity2)); + assert_eq!(best_hashes(&pool), vec![standard0, standard1, validity2]); + } + + #[tokio::test] + async fn an_account_resync_lowering_the_nonce_moves_the_validity_lane_back() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let lane = (signer.address(), U256::ZERO); + let validity1 = add(&pool, validity_1559(&signer, 1, 1_000)).await; + pool.update_accounts(vec![account_at(signer.address(), 1)]); + assert!(is_pending(&pool, validity1)); + + pool.update_accounts(vec![account_at(signer.address(), 0)]); + assert_eq!(pool.validity.pool.read().lane_cursor(&lane), Some(LaneCursor::at(0))); + assert!(is_queued(&pool, validity1)); + } + + /// Funds `signer` as an EIP-7702 delegated account (its code is the + /// `0xef0100 || target` delegation designator). + fn delegate( + client: &MockEthProvider>, + signer: &PrivateKeySigner, + ) { + let mut designator = vec![0xef, 0x01, 0x00]; + designator.extend_from_slice(Address::repeat_byte(0xDE).as_slice()); + client.add_account( + signer.address(), + ExtendedAccount::new(0, U256::from(1_000_000_000_000_000_000u64)) + .with_bytecode(Bytes::from(designator)), + ); + } + + fn signed_7702(signer: &PrivateKeySigner, nonce: u64) -> BasePooledTransaction { + let authorization = Authorization { + chain_id: U256::from(test_chain_id()), + address: Address::repeat_byte(0xDE), + nonce: nonce + 1, + }; + let auth_signature = signer.sign_hash_sync(&authorization.signature_hash()).unwrap(); + let tx = TxEip7702 { + chain_id: test_chain_id(), + nonce, + gas_limit: 100_000, + max_fee_per_gas: 1_000, + max_priority_fee_per_gas: 0, + to: Address::repeat_byte(0xEE), + value: U256::ZERO, + access_list: Default::default(), + authorization_list: vec![authorization.into_signed(auth_signature)], + input: Bytes::new(), + }; + let signature = signer.sign_hash_sync(&tx.signature_hash()).unwrap(); + let envelope = BaseTxEnvelope::Eip7702(tx.into_signed(signature)); + let recovered = envelope.clone().try_into_recovered().unwrap(); + BasePooledTransaction::new(recovered, envelope.encode_2718_len()) + } + + fn is_delegated_validity_rejection(error: &PoolErrorKind) -> bool { + matches!(error, PoolErrorKind::InvalidTransaction(_)) + && format!("{error:?}").contains("DelegatedValidityTransaction") + } + + #[tokio::test] + async fn a_delegated_sender_cannot_submit_validity_transactions() { + let (pool, client) = build_integration_pool(); + let signer = signer(); + delegate(&client, &signer); + + let error = add_err(&pool, validity_1559(&signer, 0, 1_000)).await; + assert!(is_delegated_validity_rejection(&error), "{error:?}"); + assert_eq!(pool.validity_pool_occupancy().0, 0); + + // Its standard transactions still go to reth under the delegated-account limits. + let standard = add(&pool, signed_1559(&signer, 0)).await; + assert_store(&pool, standard, PoolStore::Protocol); + } + + #[tokio::test] + async fn an_eip7702_transaction_cannot_carry_validity_predicates() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let transaction = signed_7702(&signer, 0).with_validity_predicates(resting_predicates()); + + let error = add_err(&pool, transaction).await; + assert!(is_delegated_validity_rejection(&error), "{error:?}"); + assert_eq!(pool.validity_pool_occupancy().0, 0); + } + + #[tokio::test] + async fn a_sender_delegated_after_admission_loses_its_validity_transactions() { + let (pool, client) = build_integration_pool(); + let signer = funded_signer(&client); + let other = funded_signer(&client); + let standard = add(&pool, signed_1559(&signer, 0)).await; + let validity1 = add(&pool, validity_1559(&signer, 1, 1_000)).await; + let validity2 = add(&pool, validity_1559(&signer, 2, 1_000)).await; + let unrelated = add(&pool, validity_1559(&other, 0, 1_000)).await; + + let removed = pool.apply_state_diff(&[AccountStateDiff { + address: signer.address(), + code_changed: true, + ..Default::default() + }]); + let mut removed: Vec<_> = removed.iter().map(|tx| *tx.hash()).collect(); + removed.sort(); + let mut expected = vec![validity1, validity2]; + expected.sort(); + assert_eq!(removed, expected); + assert_gone(&pool, validity1); + assert_gone(&pool, validity2); + assert_store(&pool, standard, PoolStore::Protocol); + assert_store(&pool, unrelated, PoolStore::Validity); + + // Once delegated, the sender cannot submit new ones either. + delegate(&client, &signer); + let error = add_err(&pool, validity_1559(&signer, 1, 1_000)).await; + assert!(is_delegated_validity_rejection(&error), "{error:?}"); + } } diff --git a/crates/execution/txpool/src/store.rs b/crates/execution/txpool/src/store.rs new file mode 100644 index 0000000000..c8fbf83512 --- /dev/null +++ b/crates/execution/txpool/src/store.rs @@ -0,0 +1,63 @@ +//! Routing of pooled transactions to the store that holds them. + +use crate::BasePooledTx; + +/// The store inside [`crate::BaseTransactionPool`] that owns a transaction. +/// +/// Every transaction carrying validity predicates lives in the validity pool, +/// whatever its type or nonce mode. Among the rest, EIP-8130 transactions +/// sequenced on a finite nonce channel or nonce-free live in the sidecar, and +/// everything sequenced on the account's protocol nonce lives in the reth +/// protocol pool. A `(sender, nonce key, nonce)` slot is held by at most one +/// store at a time. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub enum PoolStore { + /// The reth `Pool` holding standard, protocol-nonce transactions. + Protocol, + /// The EIP-8130 sidecar holding standard channelized and nonce-free transactions. + Sidecar, + /// The validity pool holding every transaction with validity predicates. + Validity, +} + +impl PoolStore { + /// Every store, in the order the merged best-transaction iterator uses. + pub const ALL: [Self; 3] = [Self::Protocol, Self::Sidecar, Self::Validity]; + + /// Returns the store `transaction` belongs in. + pub fn of(transaction: &T) -> Self { + if !transaction.validity_conditions().is_empty() { + Self::Validity + } else if transaction.is_eip8130_sidecar_transaction() { + Self::Sidecar + } else { + Self::Protocol + } + } + + /// Position of this store in [`Self::ALL`]. + pub const fn index(self) -> usize { + match self { + Self::Protocol => 0, + Self::Sidecar => 1, + Self::Validity => 2, + } + } + + /// Whether this store keeps its own, non-protocol-pool lanes. + pub const fn is_lane_store(self) -> bool { + !matches!(self, Self::Protocol) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn index_matches_all_order() { + for (index, store) in PoolStore::ALL.into_iter().enumerate() { + assert_eq!(store.index(), index); + } + } +} diff --git a/crates/execution/txpool/src/transaction.rs b/crates/execution/txpool/src/transaction.rs index 3c9a0520b7..f693284a2c 100644 --- a/crates/execution/txpool/src/transaction.rs +++ b/crates/execution/txpool/src/transaction.rs @@ -127,8 +127,8 @@ impl BasePooledTransaction { /// Sets the validity predicates required for this transaction's inclusion. /// - /// Conditions are compiled once into timing bounds and separate state groups. - /// The submitted batch is retained only for lossless builder forwarding. + /// Conditions are compiled once into timing bounds and separate state groups; + /// each predicate is stored exactly once and re-combined for builder forwarding. #[must_use] pub fn with_validity_predicates( mut self, @@ -140,10 +140,9 @@ impl BasePooledTransaction { self } - /// Returns the submitted predicates for lossless builder forwarding. + /// Returns every submitted predicate, regrouped by kind, for builder forwarding. /// Pool and builder checks must use [`Self::validity_conditions`] instead. - #[must_use] - pub fn validity_predicates(&self) -> &[crate::ValidityPredicate] { + pub fn validity_predicates(&self) -> impl Iterator { self.validity_conditions().submitted() } @@ -417,14 +416,6 @@ pub trait BasePooledTx: PoolTransaction + DataAvailabilitySized { /// Returns the EIP-2718 encoded bytes of the transaction. fn encoded_2718(&self) -> Cow<'_, Bytes>; - /// Returns submitted predicates for builder forwarding, not repeated evaluation. - /// - /// Derived from [`Self::validity_conditions`] so forwarding and enforcement - /// use the same ingested metadata. - fn validity_predicates(&self) -> &[crate::ValidityPredicate] { - self.validity_conditions().submitted() - } - /// Returns the compiled conditions used for pool and builder enforcement. /// /// Implementations must supply their ingested metadata, or explicitly return @@ -453,6 +444,12 @@ pub trait BasePooledTx: PoolTransaction + DataAvailabilitySized { None } + /// Returns whether this is a nonce-free EIP-8130 transaction (`nonce_key == + /// NONCE_KEY_MAX`), which has a replay id instead of a nonce lane. + fn is_eip8130_nonce_free(&self) -> bool { + self.eip8130_replay_id().is_some() + } + /// Returns the invalidation watch set computed during validation, if set. /// /// Defaults to `None` for implementers that do not track invalidation @@ -538,6 +535,11 @@ where Some(signed.tx().replay_id(self.sender())) } + fn is_eip8130_nonce_free(&self) -> bool { + self.as_eip8130() + .is_some_and(|signed| signed.tx().nonce_key == Eip8130Constants::NONCE_KEY_MAX) + } + fn watch_set(&self) -> Option<&crate::WatchSet> { self.watch_set.get() } @@ -925,7 +927,7 @@ mod tests { } #[test] - fn validity_clones_share_lossless_batch_count_and_allocated_size() { + fn validity_clones_share_partitioned_batch_count_and_allocated_size() { let transaction = eip8130_pooled(U256::ZERO); let bare_size = transaction.size(); let balance = ValidityPredicate::Balance { @@ -938,13 +940,13 @@ mod tests { value: U256::from(9), }; let mut submitted = Vec::with_capacity(17); - submitted.extend([balance.clone(), block.clone(), block, balance]); - let expected = submitted.clone(); + submitted.extend([balance.clone(), block.clone(), block.clone(), balance.clone()]); + // Forwarding yields timing predicates first, then state partitions. + let expected = [block.clone(), block, balance.clone(), balance]; let transaction = transaction.with_validity_predicates(submitted); let cloned = transaction.clone(); assert!(core::ptr::eq(transaction.validity_conditions(), cloned.validity_conditions())); - assert_eq!(cloned.validity_predicates(), expected); - assert_eq!(BasePooledTx::validity_predicates(&cloned), expected); + assert_eq!(cloned.validity_predicates().cloned().collect::>(), expected); assert_eq!(cloned.validity_conditions().len(), 4); assert_eq!(cloned.validity_conditions().block().min(), Some(U256::from(10))); assert_eq!(transaction.size(), cloned.size()); @@ -954,10 +956,10 @@ mod tests { + core::mem::size_of::() + transaction.validity_conditions().heap_size() ); - assert!( - transaction.validity_conditions().heap_size() - >= 19 * core::mem::size_of::(), - "submitted spare capacity plus the two state predicates must be counted" + assert_eq!( + transaction.validity_conditions().heap_size(), + 4 * core::mem::size_of::(), + "each predicate is stored once; submitted spare capacity is not retained" ); } @@ -971,11 +973,7 @@ mod tests { let transaction = eip8130_pooled(U256::ZERO).with_validity_predicates(vec![predicate.clone()]); - assert_eq!(transaction.validity_predicates(), core::slice::from_ref(&predicate)); - assert_eq!( - BasePooledTx::validity_predicates(&transaction), - core::slice::from_ref(&predicate) - ); + assert_eq!(transaction.validity_predicates().collect::>(), [&predicate]); assert!(transaction.validity_conditions().contains(&predicate)); let cloned = transaction.clone(); assert!(core::ptr::eq(transaction.validity_conditions(), cloned.validity_conditions())); diff --git a/crates/execution/txpool/src/two_d_nonce_pool.rs b/crates/execution/txpool/src/two_d_nonce_pool.rs index adfdf1b2ea..d42c8a96fa 100644 --- a/crates/execution/txpool/src/two_d_nonce_pool.rs +++ b/crates/execution/txpool/src/two_d_nonce_pool.rs @@ -1,41 +1,144 @@ -//! Sidecar storage and iteration for channelized and nonce-free EIP-8130 transactions. +//! Lane storage and iteration for transactions kept outside the protocol +//! pool: the EIP-8130 sidecar (finite nonce channels and nonce-free +//! transactions) and the validity pool, which also holds protocol-nonce lanes. use std::{ - collections::{BTreeMap, BinaryHeap, HashSet}, + cmp::Reverse, + collections::{BTreeMap, BTreeSet, BinaryHeap, HashSet}, + ops::Bound, sync::Arc, + time::Instant, }; use alloy_primitives::{ - Address, TxHash, U256, - map::{B256Map, HashMap}, + Address, B256, TxHash, U256, + map::{AddressMap, B256Map, HashMap}, }; use base_common_consensus::Eip8130Constants; use reth_primitives_traits::transaction::error::InvalidTransactionError; use reth_transaction_pool::{ - AddedTransactionOutcome, BestTransactions, PoolResult, PriceBumpConfig, TransactionOrdering, - ValidPoolTransaction, + AddedTransactionOutcome, BestTransactions, LocalTransactionConfig, PoolConfig, PoolResult, + PoolSize, PriceBumpConfig, SubPoolLimit, TransactionOrdering, ValidPoolTransaction, error::{InvalidPoolTransactionError, PoolError, PoolErrorKind}, - identifier::{SenderIdentifiers, TransactionId}, + identifier::{SenderId, TransactionId}, pool::{AddedTransactionState, QueuedReason}, }; -use crate::{BasePooledTx, BestTransactionPriority}; +use crate::{BasePooledTx, BestTransactionPriority, best::MergeSource}; -type LaneId = (Address, U256); +/// A finite channel: `(sender, nonce_key)`. +pub(crate) type LaneId = (Address, U256); + +/// Where a lane's cursor stands relative to canonical state. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct LaneCursor { + /// The channel nonce as validation or canonical maintenance last observed it. + pub state_nonce: u64, + /// The first nonce this lane can execute. Equal to `state_nonce` unless + /// another store holds the pending run from `state_nonce` up to here. + pub next_nonce: u64, +} + +impl LaneCursor { + /// A cursor sitting at the canonical nonce. + pub(crate) const fn at(state_nonce: u64) -> Self { + Self { state_nonce, next_nonce: state_nonce } + } +} #[derive(Debug)] struct NonceLane { + /// Canonical nonce the lane was last anchored to. + state_nonce: u64, + /// First executable nonce; see [`LaneCursor::next_nonce`]. next_nonce: u64, + /// Length of the consecutive pending run from `next_nonce`. Kept in step + /// by every mutation, so pending and queued splits never walk the lane. + pending_len: u64, transactions: BTreeMap>>, } -impl Default for NonceLane { - fn default() -> Self { - Self { next_nonce: 0, transactions: BTreeMap::new() } +impl NonceLane { + /// A lane at `cursor` holding the members of `transactions` at or past + /// its executable cursor; members below it are dropped. No member sits + /// below `next_nonce` afterwards either: insertion rejects such nonces and + /// [`Self::set_next_nonce`] drops them. + fn new( + cursor: LaneCursor, + mut transactions: BTreeMap>>, + ) -> Self { + let transactions = transactions.split_off(&cursor.next_nonce); + let mut lane = Self { + state_nonce: cursor.state_nonce, + next_nonce: cursor.next_nonce, + pending_len: 0, + transactions, + }; + lane.extend_pending_run(); + lane + } + + /// The nonce `offset` places past `next_nonce`, unless that overflows. + const fn nonce_at(&self, offset: u64) -> Option { + self.next_nonce.checked_add(offset) + } + + /// Grows the pending run over every member directly following it. + fn extend_pending_run(&mut self) { + while self + .nonce_at(self.pending_len) + .is_some_and(|nonce| self.transactions.contains_key(&nonce)) + { + self.pending_len += 1; + } + } + + fn is_pending(&self, nonce: u64) -> bool { + nonce.checked_sub(self.next_nonce).is_some_and(|offset| offset < self.pending_len) + } + + fn insert( + &mut self, + nonce: u64, + transaction: Arc>, + ) -> Option>> { + let replaced = self.transactions.insert(nonce, transaction); + if self.nonce_at(self.pending_len) == Some(nonce) { + self.extend_pending_run(); + } + replaced + } + + fn remove(&mut self, nonce: u64) -> Option>> { + let removed = self.transactions.remove(&nonce)?; + if self.is_pending(nonce) { + self.pending_len = nonce - self.next_nonce; + } + Some(removed) + } + + /// Moves the executable cursor, returning the members left below it. + fn set_next_nonce(&mut self, next_nonce: u64) -> BTreeMap>> { + let retained = self.transactions.split_off(&next_nonce); + let stale = std::mem::replace(&mut self.transactions, retained); + self.next_nonce = next_nonce; + self.pending_len = 0; + self.extend_pending_run(); + stale + } + + /// Key bound of the first member at or past `next_nonce + offset`; empty + /// when that lies past `u64::MAX`. + fn offset_start(&self, offset: u64) -> Bound { + self.nonce_at(offset).map_or(Bound::Excluded(u64::MAX), Bound::Included) + } + + /// Key bound right after the consecutive pending run; open when the run + /// reaches `u64::MAX`. + fn run_end(&self) -> Bound { + self.nonce_at(self.pending_len).map_or(Bound::Unbounded, Bound::Excluded) } -} -impl NonceLane { fn live_transactions(&self) -> impl Iterator>> { self.transactions.range(self.next_nonce..).map(|(_, transaction)| transaction) } @@ -43,22 +146,66 @@ impl NonceLane { fn consecutive_pending_transactions( &self, ) -> impl Iterator>> { - self.live_transactions() - .enumerate() - .take_while(|(offset, transaction)| { - self.next_nonce - .checked_add(*offset as u64) - .is_some_and(|expected| transaction.nonce() == expected) - }) + self.transactions + .range((Bound::Included(self.next_nonce), self.run_end())) .map(|(_, transaction)| transaction) } - fn consecutive_pending_len(&self) -> usize { - self.consecutive_pending_transactions().count() + const fn consecutive_pending_len(&self) -> usize { + self.pending_len as usize } fn queued_transactions(&self) -> impl Iterator>> { - self.live_transactions().skip(self.consecutive_pending_len()) + self.transactions + .range((self.offset_start(self.pending_len), Bound::Unbounded)) + .map(|(_, transaction)| transaction) + } + + /// Members queued behind the pending run. Insertion rejects nonces below + /// `next_nonce` and moving it drops them, so every member is live. + fn queued_len(&self) -> usize { + self.transactions.len() - self.consecutive_pending_len() + } + + /// Whether every live member is in the pending run. + fn is_fully_pending(&self) -> bool { + self.transactions.last_key_value().is_none_or(|(nonce, _)| self.is_pending(*nonce)) + } + + /// The nonce right after the lane's consecutive pending run, or `None` + /// when nothing is pending. + fn pending_end(&self) -> Option { + (self.pending_len > 0).then(|| self.next_nonce.saturating_add(self.pending_len)) + } + + const fn cursor(&self) -> LaneCursor { + LaneCursor { state_nonce: self.state_nonce, next_nonce: self.next_nonce } + } +} + +/// Transactions whose lane state changed when a lane cursor moved. +#[derive(Debug)] +pub(crate) struct LaneUpdate { + /// Removed because their nonce is below the new cursor and can never execute. + pub discarded: Vec>>, + /// Joined the lane's consecutive pending run. + pub promoted: Vec>>, + /// Left the lane's consecutive pending run because the cursor moved back + /// behind a gap. + pub queued: Vec>>, +} + +impl Default for LaneUpdate { + fn default() -> Self { + Self { discarded: Vec::new(), promoted: Vec::new(), queued: Vec::new() } + } +} + +impl LaneUpdate { + pub(crate) fn extend(&mut self, other: Self) { + self.discarded.extend(other.discarded); + self.promoted.extend(other.promoted); + self.queued.extend(other.queued); } } @@ -67,6 +214,10 @@ impl NonceLane { pub(crate) struct InsertOutcome { pub outcome: AddedTransactionOutcome, pub replaced: Option>>, + /// Lane changes from anchoring the cursor to the validation state. These + /// stand even if the insertion is rolled back. + pub anchor: LaneUpdate, + /// Queued successors the inserted transaction made pending. pub promoted: Vec>>, } @@ -74,6 +225,142 @@ pub(crate) struct InsertOutcome { #[derive(Debug)] pub(crate) struct PruneMinedOutcome { pub removed: Vec>>, + /// Lane changes caused by advancing cursors past the mined nonces. + pub lanes: LaneUpdate, +} + +/// Admission and capacity limits for the sidecar, mirroring the protocol pool. +#[derive(Debug, Clone)] +pub(crate) struct SidecarLimits { + /// Count and byte caps across all sidecar transactions. + pub capacity: SubPoolLimit, + /// Transactions one non-local sender may hold beyond its lane heads. + pub max_account_slots: usize, + /// Minimum `max_fee_per_gas` accepted, in wei. + pub minimal_protocol_basefee: u64, + /// Senders and origins exempt from the per-sender slot cap. + pub local_transactions: LocalTransactionConfig, +} + +impl SidecarLimits { + /// Applies the protocol pool's fee floor, slot cap and pending subpool cap. + pub(crate) fn from_pool_config(config: &PoolConfig) -> Self { + Self { + capacity: config.pending_limit, + max_account_slots: config.max_account_slots, + minimal_protocol_basefee: config.minimal_protocol_basefee, + local_transactions: config.local_transactions_config.clone(), + } + } + + /// Limits that never reject or evict. + pub(crate) fn unlimited() -> Self { + Self { + capacity: SubPoolLimit::new(usize::MAX, usize::MAX), + max_account_slots: usize::MAX, + minimal_protocol_basefee: 0, + local_transactions: LocalTransactionConfig::default(), + } + } +} + +/// Running totals checked against [`SidecarLimits`] and the sender ids held by +/// live transactions. +#[derive(Debug, Default)] +struct SidecarUsage { + size: usize, + per_sender: HashMap, + sender_ids: SidecarSenderIds, +} + +impl SidecarUsage { + fn add(&mut self, transaction: &ValidPoolTransaction) { + self.size += transaction.transaction.size(); + *self.per_sender.entry(transaction.sender()).or_default() += 1; + } + + fn remove(&mut self, transaction: &ValidPoolTransaction) { + self.size = self.size.saturating_sub(transaction.transaction.size()); + let sender = transaction.sender(); + if let Some(count) = self.per_sender.get_mut(&sender) { + *count -= 1; + if *count == 0 { + self.per_sender.remove(&sender); + } + } + self.sender_ids.release(transaction.sender_id()); + for id in transaction.authority_ids.iter().flatten() { + self.sender_ids.release(*id); + } + } + + fn sender_count(&self, sender: &Address) -> usize { + self.per_sender.get(sender).copied().unwrap_or_default() + } +} + +/// Sender ids for sidecar transactions, counted by the live transactions that +/// reference each address as their sender or payer. +/// +/// Reth's `SenderIdentifiers` never releases an id, so it would keep an entry +/// for every address the sidecar has ever seen. Here an id is released with +/// its last reference, and a returning address gets a fresh id. +#[derive(Debug, Default)] +pub struct SidecarSenderIds { + next_id: u64, + ids: AddressMap<(SenderId, usize)>, + addresses: HashMap, +} + +impl SidecarSenderIds { + /// Returns the id for `address` and records one more reference to it. + pub fn acquire(&mut self, address: Address) -> SenderId { + let (id, references) = self.ids.entry(address).or_insert_with(|| { + let id = SenderId::from(self.next_id); + self.next_id = self.next_id.wrapping_add(1); + self.addresses.insert(id, address); + (id, 0) + }); + *references += 1; + *id + } + + /// Drops one reference to `id`, releasing the id with its last reference. + pub fn release(&mut self, id: SenderId) { + let Some(address) = self.addresses.get(&id).copied() else { + return; + }; + let Some((_, references)) = self.ids.get_mut(&address) else { + return; + }; + *references -= 1; + if *references == 0 { + self.ids.remove(&address); + self.addresses.remove(&id); + } + } + + /// Returns the number of addresses that currently hold an id. + pub fn len(&self) -> usize { + self.ids.len() + } + + /// Returns `true` when no address holds an id. + pub fn is_empty(&self) -> bool { + self.ids.is_empty() + } +} + +/// Eviction rank of a candidate (a lane tail or a nonce-free transaction); the +/// smallest is evicted first. Field order is the ranking order: queued before +/// pending, non-local before local, lowest fee cap, then most recently received. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +struct EvictionKey { + pending: bool, + local: bool, + max_fee_per_gas: u128, + received: Reverse, + hash: TxHash, } /// EIP-8130 sidecar for finite non-zero nonce channels and nonce-free transactions. @@ -86,19 +373,174 @@ pub(crate) struct TwoDNoncePool { lanes: HashMap>, nonce_free: B256Map>>, hashes: B256Map>>, - senders: SenderIdentifiers, price_bump_config: PriceBumpConfig, + limits: SidecarLimits, + usage: SidecarUsage, + /// Whether transactions sequenced on the account's protocol nonce (nonce + /// key zero, including non-EIP-8130 types) are stored in `(sender, 0)` + /// lanes. The EIP-8130 sidecar rejects them; the validity pool accepts them. + protocol_nonce_lanes: bool, + /// Every eviction candidate, ordered by [`EvictionKey`]. Kept in step with + /// `lanes` and `nonce_free` so eviction never scans the pool. + eviction_order: BTreeSet, + /// The key each lane's tail currently holds in `eviction_order`. + lane_candidates: HashMap, + /// Lanes whose executable cursor sits past their canonical anchor, with + /// that anchor. Kept in step with `lanes` so a build never scans them. + ahead_of_state: HashMap, + /// Lanes another store also holds transactions in, flagged by the owner + /// when the two meet. Cleared with the lane; a flag may outlive the other + /// store's transactions, so readers verify before relying on it. + shared_lanes: HashSet, } impl TwoDNoncePool { - /// Creates a new 2D nonce sidecar pool. + /// Creates a new 2D nonce sidecar pool without admission limits. pub(crate) fn new(price_bump_config: PriceBumpConfig) -> Self { Self { lanes: HashMap::default(), nonce_free: B256Map::default(), hashes: B256Map::default(), - senders: SenderIdentifiers::default(), price_bump_config, + limits: SidecarLimits::unlimited(), + usage: SidecarUsage::default(), + protocol_nonce_lanes: false, + eviction_order: BTreeSet::new(), + lane_candidates: HashMap::default(), + ahead_of_state: HashMap::default(), + shared_lanes: HashSet::default(), + } + } + + /// Applies admission and capacity limits. Call before inserting. + #[must_use] + pub(crate) fn with_limits(mut self, limits: SidecarLimits) -> Self { + self.limits = limits; + self + } + + /// Also stores transactions sequenced on the account's protocol nonce, in + /// `(sender, 0)` lanes. Call before inserting. + #[must_use] + pub(crate) const fn with_protocol_nonce_lanes(mut self) -> Self { + self.protocol_nonce_lanes = true; + self + } + + /// The nonce key of the lane `transaction` is sequenced in: its EIP-8130 + /// channel, or zero for a protocol-nonce transaction when this pool stores + /// those. `None` for nonce-free transactions and for protocol-nonce + /// transactions the pool does not accept. + fn lane_key(&self, transaction: &T) -> Option { + if transaction.is_eip8130_nonce_free() { + return None; + } + transaction + .eip8130_nonce_channel_key() + .or_else(|| self.protocol_nonce_lanes.then_some(U256::ZERO)) + } + + /// Returns the summed `size()` of every pooled transaction. + pub(crate) const fn size(&self) -> usize { + self.usage.size + } + + /// Returns the number of pooled transactions. + pub(crate) fn len(&self) -> usize { + self.hashes.len() + } + + /// Rejects a new transaction the protocol pool would refuse: a fee cap below + /// the protocol floor, or a transaction from a non-local sender that already + /// fills its slots. Replacements never add a slot. + /// + /// Reth exempts an account's next nonce from the slot cap. A sender can open + /// any number of lanes, each with its own head, so only the head of a lane + /// already holding queued transactions is exempt; a head on a fresh lane + /// counts against the cap. + fn ensure_admissible( + &self, + transaction: &ValidPoolTransaction, + is_replacement: bool, + fills_existing_lane_head: bool, + ) -> PoolResult<()> { + let hash = *transaction.hash(); + let fee_cap = transaction.max_fee_per_gas(); + if fee_cap < u128::from(self.limits.minimal_protocol_basefee) { + return Err(PoolError::new( + hash, + PoolErrorKind::FeeCapBelowMinimumProtocolFeeCap(fee_cap), + )); + } + let sender = transaction.sender(); + if !is_replacement + && !fills_existing_lane_head + && !self.limits.local_transactions.is_local(transaction.origin, &sender) + && self.usage.sender_count(&sender) >= self.limits.max_account_slots + { + return Err(PoolError::new(hash, PoolErrorKind::SpammerExceededCapacity(sender))); + } + Ok(()) + } + + /// Evicts the lowest-priority transactions until the sidecar fits its + /// capacity, returning them so callers can release their guard slots. + /// + /// Only lane tails and nonce-free transactions are candidates, so eviction + /// never opens a nonce gap. Queued tails (behind a nonce gap) go first: they + /// cannot execute and cost nothing to hold, so ranking them by fee alone + /// would let gapped spam push out pending transactions. See [`EvictionKey`] + /// for the full order. Each eviction is `O(log n)`. + pub(crate) fn evict_over_capacity(&mut self) -> Vec>> { + let mut evicted = Vec::new(); + while self.limits.capacity.is_exceeded(self.hashes.len(), self.usage.size) + && let Some(hash) = self.eviction_candidate() + && let Some(transaction) = self.remove_hash(hash) + { + evicted.push(transaction); + } + evicted + } + + fn eviction_candidate(&self) -> Option { + self.eviction_order.first().map(|key| key.hash) + } + + fn eviction_key(&self, transaction: &ValidPoolTransaction, pending: bool) -> EvictionKey { + EvictionKey { + pending, + local: self + .limits + .local_transactions + .is_local(transaction.origin, &transaction.sender()), + max_fee_per_gas: transaction.max_fee_per_gas(), + received: Reverse(transaction.timestamp), + hash: *transaction.hash(), + } + } + + /// Re-ranks `lane_id`'s tail after the lane changed. Must follow every lane + /// mutation: the tail, and whether it is pending, depend on the whole lane. + /// Brings every per-lane index in line with `lane_id` after it changed: + /// its eviction candidate, its ahead-of-state anchor and its shared flag. + fn refresh_lane_indexes(&mut self, lane_id: LaneId) { + if let Some(previous) = self.lane_candidates.remove(&lane_id) { + self.eviction_order.remove(&previous); + } + let Some(lane) = self.lanes.get(&lane_id) else { + self.ahead_of_state.remove(&lane_id); + self.shared_lanes.remove(&lane_id); + return; + }; + if lane.next_nonce > lane.state_nonce { + self.ahead_of_state.insert(lane_id, lane.state_nonce); + } else { + self.ahead_of_state.remove(&lane_id); + } + if let Some((_, tail)) = lane.transactions.last_key_value() { + let key = self.eviction_key(tail, lane.is_fully_pending()); + self.eviction_order.insert(key); + self.lane_candidates.insert(lane_id, key); } } @@ -109,49 +551,72 @@ impl TwoDNoncePool { /// Returns the number of pending and queued transactions. pub(crate) fn pending_and_queued_txn_count(&self) -> (usize, usize) { - let mut pending = 0; - let mut queued = 0; + let (pending, queued) = self.lanes.values().fold((0, 0), |(pending, queued), lane| { + (pending + lane.consecutive_pending_len(), queued + lane.queued_len()) + }); + (pending + self.nonce_free.len(), queued) + } + + /// Counts and byte sizes of the pending and queued sets, in one pass over + /// the transactions and without cloning anything. + pub(crate) fn pool_size(&self) -> PoolSize { + let mut size = PoolSize::default(); for lane in self.lanes.values() { - let pending_in_lane = lane.consecutive_pending_len(); - let live_in_lane = lane.live_transactions().count(); - pending += pending_in_lane; - queued += live_in_lane.saturating_sub(pending_in_lane); + for transaction in lane.transactions.values() { + if lane.is_pending(transaction.nonce()) { + size.pending += 1; + size.pending_size += transaction.transaction.size(); + } else { + size.queued += 1; + size.queued_size += transaction.transaction.size(); + } + } } - pending += self.nonce_free.len(); - (pending, queued) + for transaction in self.nonce_free.values() { + size.pending += 1; + size.pending_size += transaction.transaction.size(); + } + size.total = size.pending + size.queued; + size + } + + /// Every pending transaction, lane runs first and nonce-free ones last. + pub(crate) fn pending_transactions_iter( + &self, + ) -> impl Iterator>> { + self.lanes + .values() + .flat_map(NonceLane::consecutive_pending_transactions) + .chain(self.nonce_free.values()) + } + + /// Every queued transaction. + pub(crate) fn queued_transactions_iter( + &self, + ) -> impl Iterator>> { + self.lanes.values().flat_map(NonceLane::queued_transactions) + } + + /// Every transaction, lane members first and nonce-free ones last. + pub(crate) fn all_transactions_iter( + &self, + ) -> impl Iterator>> { + self.lanes.values().flat_map(NonceLane::live_transactions).chain(self.nonce_free.values()) } /// Returns all pending transactions. pub(crate) fn pending_transactions(&self) -> Vec>> { - let mut transactions = Vec::new(); - for lane in self.lanes.values() { - for transaction in lane.consecutive_pending_transactions() { - transactions.push(Arc::clone(transaction)); - } - } - transactions.extend(self.nonce_free.values().cloned()); - transactions + self.pending_transactions_iter().cloned().collect() } /// Returns all queued transactions. pub(crate) fn queued_transactions(&self) -> Vec>> { - let mut transactions = Vec::new(); - for lane in self.lanes.values() { - for transaction in lane.queued_transactions() { - transactions.push(Arc::clone(transaction)); - } - } - transactions + self.queued_transactions_iter().cloned().collect() } /// Returns all transactions in the sidecar. pub(crate) fn all_transactions(&self) -> Vec>> { - let mut transactions = Vec::new(); - for lane in self.lanes.values() { - transactions.extend(lane.live_transactions().cloned()); - } - transactions.extend(self.nonce_free.values().cloned()); - transactions + self.all_transactions_iter().cloned().collect() } /// Returns all transaction hashes in the sidecar. @@ -169,6 +634,15 @@ impl TwoDNoncePool { self.hashes.get(hash).cloned() } + /// Returns the hashes of `sender`'s transactions. A sender with none costs + /// one map lookup; otherwise this walks the lanes. + pub(crate) fn sender_transaction_hashes(&self, sender: &Address) -> Vec { + if self.usage.sender_count(sender) == 0 { + return Vec::new(); + } + self.transactions_by_sender(*sender).iter().map(|transaction| *transaction.hash()).collect() + } + /// Returns transactions for the given sender. pub(crate) fn transactions_by_sender( &self, @@ -226,23 +700,47 @@ impl TwoDNoncePool { .collect() } - /// Returns or creates the sender id for the given address. - pub(crate) fn sender_id_or_create( - &mut self, - address: Address, - ) -> reth_transaction_pool::identifier::SenderId { - self.senders.sender_id_or_create(address) + /// Assigns the transaction's sender id and, for a sponsored transaction, + /// its payer's authority id, each holding a reference until the transaction + /// leaves the sidecar. + /// + /// The payer is the one admission resolved. Ids already on the transaction + /// are replaced, so a re-inserted transaction never carries ids that were + /// released. + fn acquire_ids(&mut self, transaction: &mut ValidPoolTransaction) { + let sender = transaction.sender(); + let sender_id = self.usage.sender_ids.acquire(sender); + transaction.transaction_id = TransactionId::new(sender_id, transaction.nonce()); + transaction.authority_ids = transaction + .transaction + .limit_class() + .filter(|class| class.payer != sender) + .map(|class| vec![self.usage.sender_ids.acquire(class.payer)]); } - /// Inserts a validated sidecar EIP-8130 transaction. + /// Inserts a validated transaction, anchoring its lane at `state_nonce`. /// /// Nonce-free transactions replace only another transaction with the same - /// replay id. Finite-channel transactions retain their lane-local sequence - /// replacement semantics. + /// replay id. Lane transactions retain their lane-local sequence + /// replacement semantics. The pool assigns the transaction's sender and + /// authority ids. + #[cfg(test)] pub(crate) fn insert_validated( &mut self, - mut transaction: ValidPoolTransaction, + transaction: ValidPoolTransaction, state_nonce: u64, + ) -> PoolResult> { + self.insert_validated_at(transaction, LaneCursor::at(state_nonce)) + } + + /// [`Self::insert_validated`] with the lane anchored at `cursor`, whose + /// `next_nonce` may sit past `state_nonce` when another store holds the + /// pending run in between. A lane transaction below `cursor.next_nonce` + /// is rejected as already executed. + pub(crate) fn insert_validated_at( + &mut self, + mut transaction: ValidPoolTransaction, + cursor: LaneCursor, ) -> PoolResult> { let hash = *transaction.hash(); if self.contains(&hash) { @@ -250,9 +748,7 @@ impl TwoDNoncePool { } if let Some(replay_id) = transaction.transaction.eip8130_replay_id() { - let sender_id = self.senders.sender_id_or_create(transaction.sender()); - transaction.transaction_id = TransactionId::new(sender_id, transaction.nonce()); - let transaction = Arc::new(transaction); + self.ensure_admissible(&transaction, self.nonce_free.contains_key(&replay_id), false)?; let replaced = if let Some(existing) = self.nonce_free.get(&replay_id) { if existing.is_replacement_underpriced(&transaction, &self.price_bump_config) { return Err(PoolError::new(hash, PoolErrorKind::ReplacementUnderpriced)); @@ -261,93 +757,300 @@ impl TwoDNoncePool { } else { None }; + self.acquire_ids(&mut transaction); + let transaction = Arc::new(transaction); if let Some(existing) = &replaced { self.hashes.remove(existing.hash()); + self.usage.remove(existing); + self.eviction_order.remove(&self.eviction_key(existing, true)); } self.nonce_free.insert(replay_id, Arc::clone(&transaction)); - self.hashes.insert(hash, transaction); + self.usage.add(&transaction); + self.eviction_order.insert(self.eviction_key(&transaction, true)); + self.hashes.insert(hash, Arc::clone(&transaction)); return Ok(InsertOutcome { outcome: AddedTransactionOutcome { hash, state: AddedTransactionState::Pending }, replaced, + anchor: LaneUpdate::default(), promoted: Vec::new(), }); } let sender = transaction.sender(); - let nonce_key = transaction.transaction.eip8130_nonce_channel_key().ok_or_else(|| { + let nonce_key = self.lane_key(&transaction.transaction).ok_or_else(|| { PoolError::other(hash, "2D nonce pool only accepts channelized EIP-8130 transactions") })?; let lane_id = (sender, nonce_key); - let sender_id = self.senders.sender_id_or_create(sender); let nonce = transaction.nonce(); - transaction.transaction_id = TransactionId::new(sender_id, nonce); - let transaction = Arc::new(transaction); - let lane = self.lanes.entry(lane_id).or_insert_with(|| NonceLane { - next_nonce: state_nonce, - transactions: BTreeMap::new(), - }); - // Keep the lane anchored to the state view used by validation. This may - // move backward after a reorg lowers the on-chain channel nonce, allowing - // now-valid transactions to be accepted instead of treating them as - // already executed under the pre-reorg lane cursor. - if state_nonce != lane.next_nonce { - lane.next_nonce = state_nonce; - } - let pending_len_before = lane.consecutive_pending_len(); - - if nonce < lane.next_nonce { + let existing_lane = self.lanes.get(&lane_id).filter(|lane| !lane.transactions.is_empty()); + let is_replacement = + existing_lane.is_some_and(|lane| lane.transactions.contains_key(&nonce)); + let fills_existing_lane_head = existing_lane.is_some() && nonce == cursor.next_nonce; + self.ensure_admissible(&transaction, is_replacement, fills_existing_lane_head)?; + if nonce < cursor.next_nonce { return Err(PoolError::new( hash, PoolErrorKind::InvalidTransaction(InvalidPoolTransactionError::Consensus( InvalidTransactionError::NonceNotConsistent { tx: nonce, - state: lane.next_nonce, + state: cursor.next_nonce, }, )), )); } - - let replaced: Option>> = - if let Some(existing) = lane.transactions.get(&nonce) { + let replaced = self + .lanes + .get(&lane_id) + .and_then(|lane| lane.transactions.get(&nonce)) + .map(|existing| { if existing.is_replacement_underpriced(&transaction, &self.price_bump_config) { - return Err(PoolError::new(hash, PoolErrorKind::ReplacementUnderpriced)); + Err(PoolError::new(hash, PoolErrorKind::ReplacementUnderpriced)) + } else { + Ok(Arc::clone(existing)) } - Some(Arc::clone(existing)) - } else { - None - }; + }) + .transpose()?; + self.acquire_ids(&mut transaction); + let transaction = Arc::new(transaction); - lane.transactions.insert(nonce, Arc::clone(&transaction)); + // Keep the lane anchored to the state view used by validation. This may + // move backward after a reorg lowers the on-chain channel nonce, allowing + // now-valid transactions to be accepted instead of treating them as + // already executed under the pre-reorg lane cursor. + let anchor = self.set_lane_cursor(lane_id, cursor); + let lane = + self.lanes.entry(lane_id).or_insert_with(|| NonceLane::new(cursor, BTreeMap::new())); + let pending_len_before = lane.pending_len; + + lane.insert(nonce, Arc::clone(&transaction)); self.hashes.insert(hash, Arc::clone(&transaction)); + self.usage.add(&transaction); if let Some(replaced) = &replaced { let replaced_hash = *replaced.hash(); self.hashes.remove(&replaced_hash); + self.usage.remove(replaced); } - let pending_len_after = lane.consecutive_pending_len(); - let state = if lane - .next_nonce - .checked_add(pending_len_after as u64) - .is_none_or(|boundary| nonce < boundary) - { + let state = if lane.is_pending(nonce) { AddedTransactionState::Pending } else { AddedTransactionState::Queued(QueuedReason::NonceGap) }; - let promoted = if matches!(state, AddedTransactionState::Pending) { - lane.consecutive_pending_transactions() - .skip(pending_len_before) - .filter(|candidate| *candidate.hash() != hash) - .cloned() - .collect() - } else { - Vec::new() + // Members the newcomer closed a gap in front of join the pending run with it. + let promoted = lane + .transactions + .range((lane.offset_start(pending_len_before), lane.run_end())) + .map(|(_, candidate)| candidate) + .filter(|candidate| *candidate.hash() != hash) + .cloned() + .collect(); + + self.refresh_lane_indexes(lane_id); + Ok(InsertOutcome { + outcome: AddedTransactionOutcome { hash, state }, + replaced, + anchor, + promoted, + }) + } + + /// Moves a lane's canonical anchor forward to `nonce`, leaving it alone + /// when the anchor is already there or past it. The executable cursor + /// never moves back below where it stood. + /// + /// Canonical maintenance runs independently of validation, so a late update + /// for an older block can report a nonce an insert has already anchored + /// past. Only validation anchoring and authoritative account nonces (a + /// reorg or reth's account resync) move a cursor backward, through + /// [`Self::set_lane_cursor`]. + pub(crate) fn advance_lane_nonce(&mut self, lane_id: LaneId, nonce: u64) -> LaneUpdate { + match self.lanes.get(&lane_id) { + Some(lane) if lane.state_nonce < nonce => self.set_lane_cursor( + lane_id, + LaneCursor { state_nonce: nonce, next_nonce: lane.next_nonce.max(nonce) }, + ), + _ => LaneUpdate::default(), + } + } + + /// Moves a lane's cursor to its canonical `nonce`, as validation or a + /// canonical state change observed it. + #[cfg(test)] + pub(crate) fn set_lane_nonce(&mut self, lane_id: LaneId, nonce: u64) -> LaneUpdate { + self.set_lane_cursor(lane_id, LaneCursor::at(nonce)) + } + + /// Moves a lane to `cursor`, in either direction. + /// + /// Transactions below the new executable cursor can never execute here, so + /// they are removed. The returned update also lists the transactions that + /// joined the consecutive pending run and those that left it because the + /// cursor moved back behind a gap, so listeners can follow the lane's state. + pub(crate) fn set_lane_cursor(&mut self, lane_id: LaneId, cursor: LaneCursor) -> LaneUpdate { + let mut update = LaneUpdate::default(); + let Some(lane) = self.lanes.get_mut(&lane_id) else { + return update; + }; + if lane.cursor() == cursor { + return update; + } + lane.state_nonce = cursor.state_nonce; + if lane.next_nonce == cursor.next_nonce { + self.refresh_lane_indexes(lane_id); + return update; + } + // A pending run is a contiguous nonce range, so only members between + // the two runs' starts or between their ends can have changed state; + // the shared middle stays pending and is never visited. Where the two + // spans overlap, the runs are disjoint and the overlap lies in the gap + // between them, so nothing there is reported. + let (old_next_nonce, old_pending_len) = (lane.next_nonce, lane.pending_len); + let old_end = lane.run_end(); + let was_pending = |nonce: u64| { + nonce.checked_sub(old_next_nonce).is_some_and(|offset| offset < old_pending_len) + }; + let stale = lane.set_next_nonce(cursor.next_nonce); + for transaction in stale.into_values() { + self.hashes.remove(transaction.hash()); + self.usage.remove(&transaction); + update.discarded.push(transaction); + } + let new_end = lane.run_end(); + let starts = old_next_nonce.min(lane.next_nonce)..old_next_nonce.max(lane.next_nonce); + let ends = match (old_end, new_end) { + (Bound::Excluded(a), Bound::Excluded(b)) => { + Some((Bound::Included(a.min(b)), Bound::Excluded(a.max(b)))) + } + (Bound::Excluded(end), Bound::Unbounded) | (Bound::Unbounded, Bound::Excluded(end)) => { + Some((Bound::Included(end), Bound::Unbounded)) + } + _ => None, }; + let changed = lane + .transactions + .range(starts) + .chain(ends.into_iter().flat_map(|ends| lane.transactions.range(ends))); + for (nonce, transaction) in changed { + match (was_pending(*nonce), lane.is_pending(*nonce)) { + (false, true) => update.promoted.push(Arc::clone(transaction)), + (true, false) => update.queued.push(Arc::clone(transaction)), + _ => {} + } + } + if lane.transactions.is_empty() { + self.lanes.remove(&lane_id); + } + self.refresh_lane_indexes(lane_id); + update + } + + /// Returns where `lane_id`'s cursor stands, or `None` when the lane is empty. + pub(crate) fn lane_cursor(&self, lane_id: &LaneId) -> Option { + self.lanes.get(lane_id).map(NonceLane::cursor) + } + + /// Returns the nonce right after `lane_id`'s consecutive pending run, or + /// `None` when the lane has nothing pending. + pub(crate) fn lane_pending_end(&self, lane_id: &LaneId) -> Option { + self.lanes.get(lane_id).and_then(NonceLane::pending_end) + } + + /// Returns the live transaction at `nonce` in `lane_id`. + pub(crate) fn lane_transaction( + &self, + lane_id: &LaneId, + nonce: u64, + ) -> Option>> { + let lane = self.lanes.get(lane_id)?; + (nonce >= lane.next_nonce).then(|| lane.transactions.get(&nonce).cloned())? + } + + /// Returns the transaction at `nonce` in `lane_id` if it sits in the lane's + /// consecutive pending run. + pub(crate) fn lane_pending_transaction( + &self, + lane_id: &LaneId, + nonce: u64, + ) -> Option>> { + let lane = self.lanes.get(lane_id)?; + let pending_end = lane.pending_end()?; + (lane.next_nonce..pending_end) + .contains(&nonce) + .then(|| lane.transactions.get(&nonce).cloned())? + } + + /// Returns the live transaction with the highest nonce in `lane_id`. + pub(crate) fn lane_highest_transaction( + &self, + lane_id: &LaneId, + ) -> Option>> { + self.lanes.get(lane_id)?.live_transactions().last().cloned() + } - Ok(InsertOutcome { outcome: AddedTransactionOutcome { hash, state }, replaced, promoted }) + /// Returns the last transaction of the consecutive run that starts at + /// `from` in `lane_id`, if `from` itself is live there. + pub(crate) fn highest_consecutive_from( + &self, + lane_id: &LaneId, + from: u64, + ) -> Option>> { + let lane = self.lanes.get(lane_id)?; + if from < lane.next_nonce { + return None; + } + let mut expected = from; + let mut last = None; + for (nonce, transaction) in lane.transactions.range(from..) { + if *nonce != expected { + break; + } + last = Some(Arc::clone(transaction)); + expected = expected.checked_add(1)?; + } + last + } + + /// Returns the nonce-free transaction stored under `replay_id`. + pub(crate) fn nonce_free_transaction( + &self, + replay_id: &B256, + ) -> Option>> { + self.nonce_free.get(replay_id).cloned() + } + + /// Lanes whose executable cursor sits past their canonical anchor, i.e. + /// those relying on another store's pending run, with that anchor. + pub(crate) const fn lanes_ahead_of_state(&self) -> &HashMap { + &self.ahead_of_state + } + + /// Flags `lane_id` as one another store also holds transactions in. No-op + /// for a lane this pool does not hold. + pub(crate) fn mark_shared_lane(&mut self, lane_id: LaneId) { + if self.lanes.contains_key(&lane_id) { + self.shared_lanes.insert(lane_id); + } + } + + /// Clears the shared flag once the other store holds nothing in `lane_id`. + pub(crate) fn unmark_shared_lane(&mut self, lane_id: &LaneId) { + self.shared_lanes.remove(lane_id); + } + + /// Whether `lane_id` is flagged as shared with another store. + pub(crate) fn is_shared_lane(&self, lane_id: &LaneId) -> bool { + self.shared_lanes.contains(lane_id) + } + + /// The shared lanes that currently have a pending run, with the nonce + /// right after that run. + pub(crate) fn shared_lanes_pending_end(&self) -> impl Iterator + '_ { + self.shared_lanes.iter().filter_map(|lane_id| { + self.lanes.get(lane_id).and_then(NonceLane::pending_end).map(|end| (*lane_id, end)) + }) } /// Removes the exact transactions by hash without advancing lane state. @@ -357,7 +1060,7 @@ impl TwoDNoncePool { ) -> Vec>> { let mut removed = Vec::new(); for hash in hashes { - if let Some(transaction) = self.remove_hash(*hash, false) { + if let Some(transaction) = self.remove_hash(*hash) { removed.push(transaction); } } @@ -374,9 +1077,9 @@ impl TwoDNoncePool { if self .hashes .get(hash) - .is_some_and(|transaction| transaction.transaction.eip8130_replay_id().is_some()) + .is_some_and(|transaction| transaction.transaction.is_eip8130_nonce_free()) { - if let Some(transaction) = self.remove_hash(*hash, false) { + if let Some(transaction) = self.remove_hash(*hash) { removed.push(transaction); } continue; @@ -384,7 +1087,7 @@ impl TwoDNoncePool { let Some(transaction) = self.hashes.get(hash) else { continue; }; - let Some(nonce_key) = transaction.transaction.eip8130_nonce_channel_key() else { + let Some(nonce_key) = self.lane_key(&transaction.transaction) else { continue; }; let lane_id = (transaction.sender(), nonce_key); @@ -410,8 +1113,8 @@ impl TwoDNoncePool { if self .hashes .get(hash) - .is_some_and(|transaction| transaction.transaction.eip8130_replay_id().is_some()) - && let Some(transaction) = self.remove_hash(*hash, false) + .is_some_and(|transaction| transaction.transaction.is_eip8130_nonce_free()) + && let Some(transaction) = self.remove_hash(*hash) { removed.push(transaction); } @@ -420,19 +1123,41 @@ impl TwoDNoncePool { .iter() .filter_map(|hash| { let transaction = self.hashes.get(hash)?; - let nonce_key = transaction.transaction.eip8130_nonce_channel_key()?; + let nonce_key = self.lane_key(&transaction.transaction)?; Some((transaction.sender(), nonce_key, transaction.nonce(), *hash)) }) .collect(); ordered_hashes.sort_unstable(); - for (_, _, _, hash) in ordered_hashes { - if let Some(transaction) = self.remove_hash(hash, true) { + // Mined nonces are consecutive per channel even where this pool holds + // none of the intermediate hashes, so each lane resumes after its + // highest mined nonce. + let mut mined_heads: HashMap = HashMap::default(); + for (sender, nonce_key, nonce, _) in &ordered_hashes { + mined_heads.insert((*sender, *nonce_key), *nonce); + } + let mined: HashSet = ordered_hashes.iter().map(|(.., hash)| *hash).collect(); + let mut lanes = LaneUpdate::default(); + for (lane_id, nonce) in mined_heads { + if let Some(next_nonce) = nonce.checked_add(1) { + let mut update = self.advance_lane_nonce(lane_id, next_nonce); + let (lane_mined, discarded) = update + .discarded + .into_iter() + .partition(|transaction| mined.contains(transaction.hash())); + update.discarded = discarded; + removed.extend::>(lane_mined); + lanes.extend(update); + } + } + // Mined hashes at or past an already-advanced cursor. + for (.., hash) in ordered_hashes { + if let Some(transaction) = self.remove_hash(hash) { removed.push(transaction); } } - PruneMinedOutcome { removed } + PruneMinedOutcome { removed, lanes } } /// Removes nonce-free transactions whose validity window has elapsed at @@ -468,7 +1193,8 @@ impl TwoDNoncePool { self.remove_transactions(&hashes) } - /// Returns a best-transactions iterator snapshot. + /// Returns a best-transactions iterator snapshot. It costs the pending + /// lanes, not every transaction. pub(crate) fn best_transactions( &self, ordering: O, @@ -477,41 +1203,37 @@ impl TwoDNoncePool { where O: TransactionOrdering, { - BestTwoDTransactions::new(&self.lanes, &self.nonce_free, ordering, base_fee) + BestTwoDTransactions::new( + &self.lanes, + &self.nonce_free, + self.protocol_nonce_lanes, + ordering, + base_fee, + ) } - fn remove_hash( - &mut self, - hash: TxHash, - advance_lane: bool, - ) -> Option>> { + fn remove_hash(&mut self, hash: TxHash) -> Option>> { if let Some(transaction) = self.hashes.get(&hash) && let Some(replay_id) = transaction.transaction.eip8130_replay_id() { let transaction = self.nonce_free.remove(&replay_id)?; self.hashes.remove(&hash); + self.usage.remove(&transaction); + self.eviction_order.remove(&self.eviction_key(&transaction, true)); return Some(transaction); } let transaction = self.hashes.get(&hash)?; - let nonce_key = transaction.transaction.eip8130_nonce_channel_key()?; + let nonce_key = self.lane_key(&transaction.transaction)?; let lane_id = (transaction.sender(), nonce_key); let nonce = transaction.nonce(); - let transaction = { - let lane = self.lanes.get_mut(&lane_id)?; - let transaction = lane.transactions.remove(&nonce)?; - if advance_lane - && nonce == lane.next_nonce - && let Some(next_nonce) = lane.next_nonce.checked_add(1) - { - lane.next_nonce = next_nonce; - } - transaction - }; + let transaction = self.lanes.get_mut(&lane_id)?.remove(nonce)?; if self.lanes.get(&lane_id).is_some_and(|lane| lane.transactions.is_empty()) { self.lanes.remove(&lane_id); } self.hashes.remove(&hash); + self.usage.remove(&transaction); + self.refresh_lane_indexes(lane_id); Some(transaction) } } @@ -526,19 +1248,49 @@ where O: TransactionOrdering, { lanes: Vec>, + /// Lane heads by priority. The priority names the transaction it was + /// computed for, so an entry that no longer matches its lane's current + /// member (the lane moved on, or was re-offered after a hold) is skipped. candidates: BinaryHeap<(BestTransactionPriority, usize)>, + /// Only lanes with more than one member: invalidating or holding a lane + /// after its single member was yielded has nothing left to act on. lane_indexes: HashMap, - nonce_free_indexes: HashMap, + /// Mirrors the pool's protocol-nonce lane mode for lane lookups. + protocol_nonce_lanes: bool, ordering: O, base_fee: u64, } +/// One lane's pending run as the snapshot walks it. #[derive(Debug)] struct LaneIterator { id: LaneId, - transactions: Vec>>, + head: Arc>, + /// Members after `head`, in nonce order. Empty for a one-member lane, so + /// most lanes cost no allocation beyond the head's reference. + tail: Vec>>, + /// Members yielded so far: `0` means `head` is next, `n` means `tail[n - 1]`. index: usize, invalidated: bool, + /// Held by the merge until its waiting head is yielded; see `MergeSource`. + held: bool, +} + +impl LaneIterator { + const fn new( + id: LaneId, + head: Arc>, + tail: Vec>>, + ) -> Self { + Self { id, head, tail, index: 0, invalidated: false, held: false } + } + + fn current(&self) -> Option<&Arc>> { + match self.index { + 0 => Some(&self.head), + n => self.tail.get(n - 1), + } + } } impl BestTwoDTransactions @@ -548,75 +1300,49 @@ where fn new( lanes: &HashMap>, nonce_free: &B256Map>>, + protocol_nonce_lanes: bool, ordering: O, base_fee: u64, ) -> Self { - let mut lanes: Vec<_> = lanes - .iter() - .filter_map(|(id, lane)| { - let mut next_nonce = lane.next_nonce; - let mut transactions = Vec::new(); - while let Some(transaction) = lane.transactions.get(&next_nonce) { - transactions.push(Arc::clone(transaction)); - let Some(incremented_nonce) = next_nonce.checked_add(1) else { - break; - }; - next_nonce = incremented_nonce; - } - (!transactions.is_empty()).then(|| LaneIterator { - id: *id, - transactions, - index: 0, - invalidated: false, - }) - }) - .collect(); - let finite_lane_count = lanes.len(); - lanes.extend(nonce_free.values().map(|transaction| LaneIterator { - id: (transaction.sender(), Eip8130Constants::NONCE_KEY_MAX), - transactions: vec![Arc::clone(transaction)], - index: 0, - invalidated: false, - })); - let lane_indexes = lanes[..finite_lane_count] - .iter() - .enumerate() - .map(|(index, lane)| (lane.id, index)) - .collect(); - let nonce_free_indexes = lanes[finite_lane_count..] + let mut snapshot = Vec::with_capacity(lanes.len() + nonce_free.len()); + snapshot.extend(lanes.iter().filter_map(|(id, lane)| { + let mut pending = lane.consecutive_pending_transactions(); + let head = Arc::clone(pending.next()?); + let mut tail = Vec::with_capacity(lane.consecutive_pending_len() - 1); + tail.extend(pending.cloned()); + Some(LaneIterator::new(*id, head, tail)) + })); + let lane_indexes = snapshot .iter() .enumerate() - .map(|(offset, lane)| (*lane.transactions[0].hash(), finite_lane_count + offset)) + .filter(|(_, lane)| !lane.tail.is_empty()) + .map(|(index, lane)| (lane.id, index)) .collect(); + snapshot.extend(nonce_free.values().map(|transaction| { + let id = (transaction.sender(), Eip8130Constants::NONCE_KEY_MAX); + LaneIterator::new(id, Arc::clone(transaction), Vec::new()) + })); let candidates = BinaryHeap::from( - lanes + snapshot .iter() .enumerate() .map(|(index, lane)| { - ( - BestTransactionPriority::new(&ordering, &lane.transactions[0], base_fee), - index, - ) + (BestTransactionPriority::new(&ordering, &lane.head, base_fee), index) }) .collect::>(), ); - Self { candidates, lanes, lane_indexes, nonce_free_indexes, ordering, base_fee } - } - - fn priority_key( - &self, - transaction: &Arc>, - ) -> BestTransactionPriority { - BestTransactionPriority::new(&self.ordering, transaction, self.base_fee) + Self { lanes: snapshot, candidates, lane_indexes, protocol_nonce_lanes, ordering, base_fee } } fn push_lane_head(&mut self, index: usize) { let lane = &self.lanes[index]; - if lane.invalidated || lane.index >= lane.transactions.len() { + if lane.invalidated || lane.held { return; } - let priority = self.priority_key(&lane.transactions[lane.index]); - self.candidates.push((priority, index)); + if let Some(current) = lane.current() { + let priority = BestTransactionPriority::new(&self.ordering, current, self.base_fee); + self.candidates.push((priority, index)); + } } } @@ -628,12 +1354,13 @@ where fn next(&mut self) -> Option { loop { - let (_, best_index) = self.candidates.pop()?; + let (priority, best_index) = self.candidates.pop()?; let lane = &mut self.lanes[best_index]; - if lane.invalidated { + let Some(current) = lane.current() else { continue }; + if lane.invalidated || lane.held || current.hash() != priority.hash() { continue; } - let transaction = Arc::clone(&lane.transactions[lane.index]); + let transaction = Arc::clone(current); lane.index += 1; self.push_lane_head(best_index); return Some(transaction); @@ -645,13 +1372,19 @@ impl BestTransactions for BestTwoDTransactions where O: TransactionOrdering, { + /// Parks the rest of `transaction`'s lane. Callers pass a transaction this + /// iterator yielded: a lane of one (every nonce-free transaction included) + /// has nothing left once yielded, so it is not indexed and this is a no-op. fn mark_invalid(&mut self, transaction: &Self::Item, _kind: InvalidPoolTransactionError) { - let index = if let Some(nonce_key) = transaction.transaction.eip8130_nonce_channel_key() { - self.lane_indexes.get(&(transaction.sender(), nonce_key)).copied() - } else { - self.nonce_free_indexes.get(transaction.hash()).copied() + let Some(nonce_key) = transaction + .transaction + .eip8130_nonce_channel_key() + .or_else(|| self.protocol_nonce_lanes.then_some(U256::ZERO)) + .filter(|_| !transaction.transaction.is_eip8130_nonce_free()) + else { + return; }; - if let Some(index) = index { + if let Some(index) = self.lane_indexes.get(&(transaction.sender(), nonce_key)).copied() { self.lanes[index].invalidated = true; } } @@ -661,6 +1394,26 @@ where fn set_skip_blobs(&mut self, _skip_blobs: bool) {} } +impl MergeSource for BestTwoDTransactions +where + O: TransactionOrdering, +{ + fn hold_lane(&mut self, lane_id: &LaneId) { + if let Some(index) = self.lane_indexes.get(lane_id).copied() { + self.lanes[index].held = true; + } + } + + fn release_lane(&mut self, lane_id: &LaneId) { + if let Some(index) = self.lane_indexes.get(lane_id).copied() + && self.lanes[index].held + { + self.lanes[index].held = false; + self.push_lane_head(index); + } + } +} + #[cfg(test)] mod tests { use std::{ @@ -668,18 +1421,23 @@ mod tests { time::Instant, }; - use alloy_consensus::{Transaction, transaction::Recovered}; - use alloy_primitives::Bytes; + use alloy_consensus::{ + SignableTransaction, Transaction, TxEip1559, + transaction::{Recovered, SignerRecoverable}, + }; + use alloy_eips::eip2718::Encodable2718; + use alloy_primitives::{Bytes, TxKind}; use alloy_signer::SignerSync; use alloy_signer_local::PrivateKeySigner; use base_common_chains::ChainConfig; use base_common_consensus::{ - BasePooledTransaction as ConsensusPooledTransaction, Eip8130Signed, TxEip8130, + BasePooledTransaction as ConsensusPooledTransaction, BaseTxEnvelope, Eip8130Signed, + TxEip8130, }; use reth_transaction_pool::{PoolTransaction, PriceBumpConfig, Priority, TransactionOrigin}; use super::*; - use crate::{BaseOrdering, BasePooledTransaction, ValidityOperator, ValidityPredicate}; + use crate::{BaseOrdering, BasePooledTransaction, LimitClass}; #[derive(Clone, Debug, Default)] struct CountingOrdering { @@ -824,18 +1582,23 @@ mod tests { .map(|nonce_key| { ( (signer.address(), U256::from(nonce_key)), - NonceLane { - next_nonce: 0, - transactions: BTreeMap::from([(0, Arc::clone(&transaction))]), - }, + NonceLane::new( + LaneCursor::at(0), + BTreeMap::from([(0, Arc::clone(&transaction))]), + ), ) }) .collect(); let setup_elapsed = setup_started.elapsed(); let snapshot_started = Instant::now(); - let best = - BestTwoDTransactions::new(&lanes, &B256Map::default(), BaseOrdering::coinbase_tip(), 0); + let best = BestTwoDTransactions::new( + &lanes, + &B256Map::default(), + false, + BaseOrdering::coinbase_tip(), + 0, + ); let snapshot_elapsed = snapshot_started.elapsed(); let drain_started = Instant::now(); @@ -915,30 +1678,6 @@ mod tests { assert_eq!(pool.pending_and_queued_txn_count(), (2, 0)); } - #[test] - fn validity_transaction_replaces_with_only_a_higher_max_fee() { - let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()); - let signer = signer(); - let predicate = ValidityPredicate::Balance { - address: Address::ZERO, - op: ValidityOperator::Equal, - value: U256::ZERO, - }; - let original = signed_nonce_free_tx(&signer, 1, 10, 100) - .with_validity_predicates(vec![predicate.clone()]); - let original = valid_pool_transaction(original); - let replacement = - signed_nonce_free_tx(&signer, 1, 0, 101).with_validity_predicates(vec![predicate]); - - pool.insert_validated(original, 0).unwrap(); - assert!( - pool.insert_validated(valid_pool_transaction(replacement), 0) - .unwrap() - .replaced - .is_some() - ); - } - #[test] fn nonce_free_removal_and_iteration_are_independent() { let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()); @@ -951,8 +1690,9 @@ mod tests { pool.insert_validated(low, 0).unwrap(); let mut best = pool.best_transactions(BaseOrdering::coinbase_tip(), 0); - let invalidated = pool.get(&high_hash).unwrap(); - best.mark_invalid(&invalidated, InvalidPoolTransactionError::Underpriced); + let yielded = best.next().unwrap(); + assert_eq!(*yielded.hash(), high_hash); + best.mark_invalid(&yielded, InvalidPoolTransactionError::Underpriced); assert_eq!(best.next().map(|tx| *tx.hash()), Some(low_hash)); assert_eq!(pool.remove_transactions_and_descendants(&[high_hash]).len(), 1); @@ -1061,6 +1801,108 @@ mod tests { ); } + fn hashes(transactions: &[Arc>]) -> Vec { + transactions.iter().map(|transaction| *transaction.hash()).collect() + } + + #[test] + fn higher_state_nonce_discards_transactions_below_the_cursor() { + let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()); + let signer = signer(); + let key = U256::from(4); + let stale: Vec<_> = (0..2) + .map(|nonce| valid_pool_transaction(signed_channel_tx(&signer, key, nonce, 1_000))) + .collect(); + let stale_hashes: Vec<_> = stale.iter().map(|transaction| *transaction.hash()).collect(); + for transaction in stale { + pool.insert_validated(transaction, 0).unwrap(); + } + + let next = valid_pool_transaction(signed_channel_tx(&signer, key, 2, 1_000)); + let outcome = pool.insert_validated(next, 2).unwrap(); + + assert_eq!(hashes(&outcome.anchor.discarded), stale_hashes); + assert!(stale_hashes.iter().all(|hash| !pool.contains(hash))); + assert_eq!(pool.all_hashes(), vec![outcome.outcome.hash]); + } + + #[test] + fn rejected_insert_leaves_the_lane_cursor_unchanged() { + let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()); + let signer = signer(); + let key = U256::from(4); + let head = valid_pool_transaction(signed_channel_tx(&signer, key, 0, 1_000)); + let head_hash = *head.hash(); + pool.insert_validated(head, 0).unwrap(); + + let too_low = valid_pool_transaction(signed_channel_tx(&signer, key, 1, 1_000)); + assert!(pool.insert_validated(too_low, 2).is_err()); + assert!(pool.contains(&head_hash)); + } + + #[test] + fn mined_nonce_past_a_gap_advances_the_cursor() { + let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()); + let signer = signer(); + let key = U256::from(5); + let [head, mined, next] = [0, 2, 3] + .map(|nonce| valid_pool_transaction(signed_channel_tx(&signer, key, nonce, 1_000))); + let (head_hash, mined_hash, next_hash) = (*head.hash(), *mined.hash(), *next.hash()); + for transaction in [head, mined, next] { + pool.insert_validated(transaction, 0).unwrap(); + } + + // The block also mines nonce 1, whose hash this pool never held. + let outcome = pool.prune_mined(&[head_hash, TxHash::repeat_byte(0x11), mined_hash]); + + assert_eq!(hashes(&outcome.removed), vec![head_hash, mined_hash]); + assert_eq!(hashes(&outcome.lanes.promoted), vec![next_hash]); + assert_eq!(hashes(&pool.pending_transactions()), vec![next_hash]); + } + + #[test] + fn mined_nonce_discards_a_different_hash_at_that_nonce() { + let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()); + let signer = signer(); + let key = U256::from(6); + let [mined, other, next, successor] = [0, 1, 2, 3] + .map(|nonce| valid_pool_transaction(signed_channel_tx(&signer, key, nonce, 1_000))); + let (mined_hash, other_hash, next_hash) = (*mined.hash(), *other.hash(), *next.hash()); + for transaction in [mined, other, next, successor] { + pool.insert_validated(transaction, 0).unwrap(); + } + + // The block mines nonces 0 to 2, nonce 1 under a hash this pool does not hold. + let outcome = pool.prune_mined(&[mined_hash, next_hash]); + + assert_eq!(hashes(&outcome.removed), vec![mined_hash, next_hash]); + assert_eq!(hashes(&outcome.lanes.discarded), vec![other_hash]); + assert!(!pool.contains(&other_hash)); + } + + #[test] + fn lower_state_nonce_reports_transactions_left_behind_a_gap_as_queued() { + let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()); + let signer = signer(); + let key = U256::from(7); + let pending: Vec<_> = (2..4) + .map(|nonce| valid_pool_transaction(signed_channel_tx(&signer, key, nonce, 1_000))) + .collect(); + let pending_hashes: Vec<_> = + pending.iter().map(|transaction| *transaction.hash()).collect(); + for transaction in pending { + pool.insert_validated(transaction, 2).unwrap(); + } + + // A reorg rewinds the channel to 1; nonce 1 is not pooled. + let later = valid_pool_transaction(signed_channel_tx(&signer, key, 5, 1_000)); + let outcome = pool.insert_validated(later, 1).unwrap(); + + assert_eq!(hashes(&outcome.anchor.queued), pending_hashes); + assert!(outcome.anchor.discarded.is_empty()); + assert!(pool.pending_transactions().is_empty()); + } + #[test] fn contiguous_lane_counts_full_run_as_pending() { let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()); @@ -1105,15 +1947,15 @@ mod tests { let queued = Arc::new(valid_pool_transaction(signed_channel_tx(&signer, U256::from(15), 10, 700))); - let lane = NonceLane { - next_nonce: 5, - transactions: BTreeMap::from([ + let lane = NonceLane::new( + LaneCursor::at(5), + BTreeMap::from([ (3, Arc::clone(&stale)), (5, Arc::clone(&first_pending)), (6, Arc::clone(&second_pending)), (10, Arc::clone(&queued)), ]), - }; + ); assert_eq!( lane.queued_transactions().map(|transaction| *transaction.hash()).collect::>(), @@ -1131,10 +1973,10 @@ mod tests { 1_000, ))); - let lane = NonceLane { - next_nonce: u64::MAX, - transactions: BTreeMap::from([(u64::MAX, Arc::clone(&transaction))]), - }; + let lane = NonceLane::new( + LaneCursor::at(u64::MAX), + BTreeMap::from([(u64::MAX, Arc::clone(&transaction))]), + ); assert_eq!( lane.consecutive_pending_transactions() @@ -1162,14 +2004,14 @@ mod tests { pool.hashes.insert(*queued.hash(), Arc::clone(&queued)); pool.lanes.insert( lane_id, - NonceLane { - next_nonce: 5, - transactions: BTreeMap::from([ + NonceLane::new( + LaneCursor::at(5), + BTreeMap::from([ (3, Arc::clone(&stale)), (5, Arc::clone(&pending)), (7, Arc::clone(&queued)), ]), - }, + ), ); assert_eq!( @@ -1200,16 +2042,21 @@ mod tests { let lane_id = (signer.address(), U256::from(17)); let lanes: HashMap<_, _> = [( lane_id, - NonceLane { - next_nonce: u64::MAX, - transactions: BTreeMap::from([(u64::MAX, Arc::clone(&transaction))]), - }, + NonceLane::new( + LaneCursor::at(u64::MAX), + BTreeMap::from([(u64::MAX, Arc::clone(&transaction))]), + ), )] .into_iter() .collect(); - let mut best = - BestTwoDTransactions::new(&lanes, &B256Map::default(), BaseOrdering::coinbase_tip(), 0); + let mut best = BestTwoDTransactions::new( + &lanes, + &B256Map::default(), + false, + BaseOrdering::coinbase_tip(), + 0, + ); assert_eq!(best.next().map(|transaction| *transaction.hash()), Some(*transaction.hash())); assert!(best.next().is_none()); } @@ -1222,8 +2069,7 @@ mod tests { valid_pool_transaction(signed_channel_tx(&signer, U256::from(18), u64::MAX, 1_000)); let lane_id = (signer.address(), U256::from(18)); - pool.lanes - .insert(lane_id, NonceLane { next_nonce: u64::MAX, transactions: BTreeMap::new() }); + pool.lanes.insert(lane_id, NonceLane::new(LaneCursor::at(u64::MAX), BTreeMap::new())); let outcome = pool.insert_validated(transaction, u64::MAX).unwrap(); @@ -1241,22 +2087,20 @@ mod tests { 1_000, ))); let head_hash = *head.hash(); - let stale = - Arc::new(valid_pool_transaction(signed_channel_tx(&signer, U256::from(19), 7, 900))); let lane_id = (signer.address(), U256::from(19)); pool.hashes.insert(head_hash, Arc::clone(&head)); pool.lanes.insert( lane_id, - NonceLane { - next_nonce: u64::MAX, - transactions: BTreeMap::from([(7, Arc::clone(&stale)), (u64::MAX, head)]), - }, + NonceLane::new(LaneCursor::at(u64::MAX), BTreeMap::from([(u64::MAX, head)])), ); - let _ = pool.prune_mined(&[head_hash]); + let removed = pool.prune_mined(&[head_hash]).removed; - assert_eq!(pool.lanes.get(&lane_id).map(|lane| lane.next_nonce), Some(u64::MAX)); + // Mining the last possible nonce empties the lane; its cursor never + // wraps back to a nonce that could accept a replay. + assert_eq!(removed.len(), 1); + assert!(pool.lanes.get(&lane_id).is_none_or(|lane| lane.next_nonce == u64::MAX)); } #[test] @@ -1347,6 +2191,63 @@ mod tests { assert_eq!(yielded_hashes[0], second_lane_head_hash); } + #[test] + fn a_queued_member_at_u64_max_counts_as_queued() { + let signer = signer(); + let head = + Arc::new(valid_pool_transaction(signed_channel_tx(&signer, U256::from(43), 0, 1_000))); + let last = Arc::new(valid_pool_transaction(signed_channel_tx( + &signer, + U256::from(43), + u64::MAX, + 1_000, + ))); + let lane = NonceLane::new( + LaneCursor::at(0), + BTreeMap::from([(0, Arc::clone(&head)), (u64::MAX, Arc::clone(&last))]), + ); + + assert_eq!(lane.consecutive_pending_len(), 1); + assert_eq!(lane.queued_len(), 1); + assert_eq!( + lane.queued_transactions().map(|transaction| *transaction.hash()).collect::>(), + vec![*last.hash()] + ); + } + + #[test] + fn a_held_lane_is_skipped_until_released() { + let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()); + let signer = signer(); + let held_lane = (signer.address(), U256::from(41)); + let rich: Vec<_> = (0..3) + .map(|sequence| { + valid_pool_transaction(signed_channel_tx(&signer, U256::from(41), sequence, 1_000)) + }) + .collect(); + let poor = valid_pool_transaction(signed_channel_tx(&signer, U256::from(42), 0, 100)); + let rich_hashes: Vec<_> = rich.iter().map(|tx| *tx.hash()).collect(); + let poor_hash = *poor.hash(); + for transaction in rich { + pool.insert_validated(transaction, 0).unwrap(); + } + pool.insert_validated(poor, 0).unwrap(); + + let mut best = pool.best_transactions(BaseOrdering::coinbase_tip(), 0); + assert_eq!(best.next().map(|tx| *tx.hash()), Some(rich_hashes[0])); + // Holding the lane after its head was pulled leaves the rest of it + // alone: the other lane is offered instead of the lane's next member. + best.hold_lane(&held_lane); + assert_eq!(best.next().map(|tx| *tx.hash()), Some(poor_hash)); + assert!(best.next().is_none()); + // Releasing picks the lane up where it stopped, in order. + best.release_lane(&held_lane); + assert_eq!( + best.map(|tx| *tx.hash()).collect::>(), + vec![rich_hashes[1], rich_hashes[2]] + ); + } + #[test] fn best_transactions_uses_effective_tip_across_sidecar_lanes() { let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()); @@ -1387,4 +2288,463 @@ mod tests { let mut best = pool.best_transactions(BaseOrdering::coinbase_tip(), 10); assert_eq!(best.next().map(|transaction| *transaction.hash()), Some(older_hash)); } + + #[test] + fn sender_slot_cap_counts_fresh_lane_heads() { + let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()) + .with_limits(SidecarLimits { max_account_slots: 2, ..SidecarLimits::unlimited() }); + let signer = signer(); + for nonce_key in 1..=2 { + let queued = signed_channel_tx(&signer, U256::from(nonce_key), 1, 1_000); + pool.insert_validated(valid_pool_transaction(queued), 0).unwrap(); + } + + let over_cap = signed_channel_tx(&signer, U256::from(3), 1, 1_000); + let error = pool.insert_validated(valid_pool_transaction(over_cap), 0).unwrap_err(); + assert!(matches!(error.kind, PoolErrorKind::SpammerExceededCapacity(_))); + let fresh_lane_head = signed_channel_tx(&signer, U256::from(3), 0, 1_000); + let error = pool.insert_validated(valid_pool_transaction(fresh_lane_head), 0).unwrap_err(); + assert!(matches!(error.kind, PoolErrorKind::SpammerExceededCapacity(_))); + + let gap_filling_head = signed_channel_tx(&signer, U256::from(1), 0, 1_000); + pool.insert_validated(valid_pool_transaction(gap_filling_head), 0).unwrap(); + let replacement = signed_channel_tx(&signer, U256::from(1), 1, 2_000); + let outcome = pool.insert_validated(valid_pool_transaction(replacement), 0).unwrap(); + assert!(outcome.replaced.is_some()); + } + + #[test] + fn slots_are_released_when_a_cursor_advance_discards_transactions() { + let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()) + .with_limits(SidecarLimits { max_account_slots: 2, ..SidecarLimits::unlimited() }); + let signer = signer(); + let nonce_key = U256::from(1); + for sequence in 0..2 { + let transaction = signed_channel_tx(&signer, nonce_key, sequence, 1_000); + pool.insert_validated(valid_pool_transaction(transaction), 0).unwrap(); + } + + let update = pool.advance_lane_nonce((signer.address(), nonce_key), 2); + assert_eq!(update.discarded.len(), 2); + + for sequence in 2..4 { + let transaction = signed_channel_tx(&signer, nonce_key, sequence, 1_000); + pool.insert_validated(valid_pool_transaction(transaction), 2).unwrap(); + } + } + + #[test] + fn eviction_takes_the_cheapest_lane_tail_without_opening_a_gap() { + let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()).with_limits(SidecarLimits { + capacity: SubPoolLimit::new(2, usize::MAX), + ..SidecarLimits::unlimited() + }); + let lane_signer = signer(); + let cheap_head = + valid_pool_transaction(signed_channel_tx(&lane_signer, U256::from(1), 0, 50)); + let cheap_head_hash = *cheap_head.hash(); + let rich_tail = + valid_pool_transaction(signed_channel_tx(&lane_signer, U256::from(1), 1, 5_000)); + let rich_tail_hash = *rich_tail.hash(); + let other = valid_pool_transaction(signed_channel_tx(&signer(), U256::from(1), 0, 1_000)); + let other_hash = *other.hash(); + pool.insert_validated(cheap_head, 0).unwrap(); + pool.insert_validated(rich_tail, 0).unwrap(); + pool.insert_validated(other, 0).unwrap(); + + let evicted: Vec<_> = + pool.evict_over_capacity().iter().map(|transaction| *transaction.hash()).collect(); + + assert_eq!(evicted, vec![other_hash]); + assert!(pool.contains(&cheap_head_hash)); + assert!(pool.contains(&rich_tail_hash)); + assert!(pool.evict_over_capacity().is_empty()); + } + + #[test] + fn full_sidecar_of_queued_spam_still_admits_a_cheaper_pending_head() { + let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()).with_limits(SidecarLimits { + capacity: SubPoolLimit::new(2, usize::MAX), + ..SidecarLimits::unlimited() + }); + let spam: Vec<_> = (0..2) + .map(|_| { + let gapped = valid_pool_transaction(signed_channel_tx( + &signer(), + U256::from(1), + 1, + 1_000_000, + )); + let hash = *gapped.hash(); + pool.insert_validated(gapped, 0).unwrap(); + hash + }) + .collect(); + assert_eq!(pool.pending_and_queued_txn_count(), (0, 2)); + let honest = valid_pool_transaction(signed_channel_tx(&signer(), U256::from(1), 0, 100)); + let honest_hash = *honest.hash(); + pool.insert_validated(honest, 0).unwrap(); + + let evicted: Vec<_> = + pool.evict_over_capacity().iter().map(|transaction| *transaction.hash()).collect(); + + assert_eq!(evicted.len(), 1); + assert!(spam.contains(&evicted[0]), "a queued transaction is evicted first"); + assert!(pool.contains(&honest_hash), "the pending head survives"); + } + + fn capped_pool(capacity: usize) -> TwoDNoncePool { + TwoDNoncePool::new(PriceBumpConfig::default()).with_limits(SidecarLimits { + capacity: SubPoolLimit::new(capacity, usize::MAX), + ..SidecarLimits::unlimited() + }) + } + + fn evicted_hashes(pool: &mut TwoDNoncePool) -> Vec { + pool.evict_over_capacity().iter().map(|transaction| *transaction.hash()).collect() + } + + #[test] + fn filling_a_gap_ranks_the_lane_tail_as_pending() { + let mut pool = capped_pool(2); + let lane_signer = signer(); + let gapped = + valid_pool_transaction(signed_channel_tx(&lane_signer, U256::from(1), 1, 1_000_000)); + let gapped_hash = *gapped.hash(); + let cheap = valid_pool_transaction(signed_channel_tx(&signer(), U256::from(1), 0, 100)); + let cheap_hash = *cheap.hash(); + pool.insert_validated(gapped, 0).unwrap(); + pool.insert_validated(cheap, 0).unwrap(); + pool.insert_validated( + valid_pool_transaction(signed_channel_tx(&lane_signer, U256::from(1), 0, 2_000)), + 0, + ) + .unwrap(); + + assert_eq!(evicted_hashes(&mut pool), vec![cheap_hash]); + assert!(pool.contains(&gapped_hash)); + } + + #[test] + fn moving_a_cursor_behind_a_gap_ranks_the_lane_tail_as_queued() { + let mut pool = capped_pool(2); + let lane_signer = signer(); + let rich = + valid_pool_transaction(signed_channel_tx(&lane_signer, U256::from(1), 5, 1_000_000)); + let rich_hash = *rich.hash(); + pool.insert_validated(rich, 5).unwrap(); + pool.set_lane_nonce((lane_signer.address(), U256::from(1)), 4); + let cheap = valid_pool_transaction(signed_channel_tx(&signer(), U256::from(1), 0, 100)); + let cheap_hash = *cheap.hash(); + pool.insert_validated(cheap, 0).unwrap(); + pool.insert_validated( + valid_pool_transaction(signed_channel_tx(&signer(), U256::from(1), 0, 200)), + 0, + ) + .unwrap(); + + assert_eq!(evicted_hashes(&mut pool), vec![rich_hash]); + assert!(pool.contains(&cheap_hash)); + } + + #[test] + fn evicting_a_lane_tail_exposes_the_next_one() { + let mut pool = capped_pool(0); + let lane_signer = signer(); + let head = valid_pool_transaction(signed_channel_tx(&lane_signer, U256::from(1), 0, 100)); + let head_hash = *head.hash(); + let tail = valid_pool_transaction(signed_channel_tx(&lane_signer, U256::from(1), 1, 100)); + let tail_hash = *tail.hash(); + pool.insert_validated(head, 0).unwrap(); + pool.insert_validated(tail, 0).unwrap(); + + assert_eq!(evicted_hashes(&mut pool), vec![tail_hash, head_hash]); + assert_eq!(pool.pending_and_queued_txn_count(), (0, 0)); + } + + #[test] + fn nonce_free_eviction_follows_replacement_and_removal() { + let mut pool = capped_pool(0); + let signer = signer(); + let replaced = valid_pool_transaction(signed_nonce_free_tx(&signer, 1, 0, 100)); + let replacement = valid_pool_transaction(signed_nonce_free_tx(&signer, 1, 0, 5_000)); + let replacement_hash = *replacement.hash(); + let removed = valid_pool_transaction(signed_nonce_free_tx(&signer, 2, 0, 200)); + let removed_hash = *removed.hash(); + let cheapest = valid_pool_transaction(signed_nonce_free_tx(&signer, 3, 0, 300)); + let cheapest_hash = *cheapest.hash(); + let richest = valid_pool_transaction(signed_nonce_free_tx(&signer, 4, 0, 9_000)); + let richest_hash = *richest.hash(); + for transaction in [replaced, replacement, removed, cheapest, richest] { + pool.insert_validated(transaction, 0).unwrap(); + } + assert_eq!(pool.remove_transactions(&[removed_hash]).len(), 1); + + assert_eq!(evicted_hashes(&mut pool), vec![cheapest_hash, replacement_hash, richest_hash]); + } + + fn sponsored_by( + transaction: BasePooledTransaction, + payer: Address, + ) -> ValidPoolTransaction { + transaction.set_limit_class(LimitClass { + sender: transaction.sender(), + payer, + classification_generation: 0, + sender_locked: false, + payer_locked: false, + payer_trusted: false, + payer_balance: U256::MAX, + max_cost: U256::ZERO, + }); + valid_pool_transaction(transaction) + } + + #[test] + fn sender_ids_are_released_when_transactions_are_removed_or_mined() { + const NOW_MS: u64 = 1_700_000_000_000; + let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()); + let [removed, mined, expired] = std::array::from_fn(|_| signer()); + let payer = Address::repeat_byte(0xA5); + pool.insert_validated( + sponsored_by(signed_channel_tx(&removed, U256::from(1), 0, 1_000), payer), + 0, + ) + .unwrap(); + let mined_tx = valid_pool_transaction(signed_channel_tx(&mined, U256::from(2), 0, 1_000)); + let mined_hash = *mined_tx.hash(); + pool.insert_validated(mined_tx, 0).unwrap(); + pool.insert_validated( + valid_pool_transaction(signed_nonce_free_tx(&expired, NOW_MS, 0, 1_000)), + 0, + ) + .unwrap(); + assert_eq!(pool.usage.sender_ids.len(), 4, "three senders and one payer"); + + pool.remove_transactions_by_sender(removed.address()); + assert_eq!(pool.usage.sender_ids.len(), 2, "the sender and its payer are released"); + pool.prune_mined(&[mined_hash]); + pool.remove_expired_nonce_free(NOW_MS); + + assert!(pool.all_transactions().is_empty()); + assert_eq!(pool.usage.sender_ids.len(), 0); + } + + #[test] + fn sender_ids_are_released_when_transactions_are_replaced_or_evicted() { + let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()).with_limits(SidecarLimits { + capacity: SubPoolLimit::new(1, usize::MAX), + ..SidecarLimits::unlimited() + }); + let [sender, evicted] = std::array::from_fn(|_| signer()); + let [first_payer, second_payer] = [Address::repeat_byte(0xA1), Address::repeat_byte(0xA2)]; + pool.insert_validated( + sponsored_by(signed_channel_tx(&sender, U256::from(1), 0, 1_000), first_payer), + 0, + ) + .unwrap(); + pool.insert_validated( + sponsored_by(signed_channel_tx(&sender, U256::from(1), 0, 2_000), second_payer), + 0, + ) + .unwrap(); + assert_eq!(pool.usage.sender_ids.len(), 2, "the replaced transaction's payer is released"); + + let underpriced = signed_channel_tx(&evicted, U256::from(1), 0, 1); + pool.insert_validated(valid_pool_transaction(underpriced), 0).unwrap(); + pool.evict_over_capacity(); + assert_eq!(pool.usage.sender_ids.len(), 2, "the evicted sender is released"); + + pool.set_lane_nonce((sender.address(), U256::from(1)), 1); + assert!(pool.all_transactions().is_empty()); + assert_eq!( + pool.usage.sender_ids.len(), + 0, + "a transaction below the lane cursor is released" + ); + } + + #[test] + fn removal_releases_sender_ids() { + let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()); + let sender = signer(); + let payer = Address::repeat_byte(0xA5); + let self_paid = valid_pool_transaction(signed_channel_tx(&sender, U256::from(1), 0, 1_000)); + let self_paid_hash = *self_paid.hash(); + pool.insert_validated(self_paid, 0).unwrap(); + let sponsored = sponsored_by(signed_channel_tx(&sender, U256::from(2), 0, 1_000), payer); + let sponsored_hash = *sponsored.hash(); + pool.insert_validated(sponsored, 0).unwrap(); + + pool.remove_transactions(&[self_paid_hash]); + assert_eq!(pool.usage.sender_ids.len(), 2, "the sponsored transaction keeps both ids"); + + pool.remove_transactions(&[sponsored_hash]); + assert_eq!(pool.usage.sender_ids.len(), 0); + } + + #[test] + fn rejected_insert_allocates_no_sender_id() { + let mut pool = TwoDNoncePool::new(PriceBumpConfig::default()); + let [existing, rejected] = std::array::from_fn(|_| signer()); + pool.insert_validated( + valid_pool_transaction(signed_channel_tx(&existing, U256::from(1), 0, 1_000)), + 0, + ) + .unwrap(); + + let below_cursor = signed_channel_tx(&rejected, U256::from(1), 0, 1_000); + assert!(pool.insert_validated(valid_pool_transaction(below_cursor), 1).is_err()); + let underpriced = signed_channel_tx(&existing, U256::from(1), 0, 1_001); + assert!(pool.insert_validated(valid_pool_transaction(underpriced), 0).is_err()); + + assert_eq!(pool.usage.sender_ids.len(), 1); + } + + fn signed_1559(signer: &PrivateKeySigner, nonce: u64, max_fee: u128) -> BasePooledTransaction { + let tx = TxEip1559 { + chain_id: test_chain_id(), + nonce, + gas_limit: 50_000, + max_fee_per_gas: max_fee, + max_priority_fee_per_gas: 0, + to: TxKind::Call(Address::repeat_byte(0xEE)), + value: U256::ZERO, + access_list: Default::default(), + input: Bytes::new(), + }; + let signature = signer.sign_hash_sync(&tx.signature_hash()).unwrap(); + let envelope = BaseTxEnvelope::Eip1559(tx.into_signed(signature)); + let recovered = envelope.clone().try_into_recovered().unwrap(); + BasePooledTransaction::new(recovered, envelope.encode_2718_len()) + } + + fn protocol_lane_pool() -> TwoDNoncePool { + TwoDNoncePool::new(PriceBumpConfig::default()).with_protocol_nonce_lanes() + } + + #[test] + fn protocol_nonce_transactions_need_protocol_lanes() { + let signer = signer(); + let mut channel_only = TwoDNoncePool::new(PriceBumpConfig::default()); + let rejected = valid_pool_transaction(signed_1559(&signer, 0, 1_000)); + let rejected_hash = *rejected.hash(); + let error = channel_only.insert_validated(rejected, 0).unwrap_err(); + assert_eq!(error.hash, rejected_hash); + assert_eq!(channel_only.len(), 0); + + // EIP-1559 and key-0 EIP-8130 transactions share the sender's lane 0. + let mut pool = protocol_lane_pool(); + let eip1559 = valid_pool_transaction(signed_1559(&signer, 0, 1_000)); + let key0 = valid_pool_transaction(signed_channel_tx(&signer, U256::ZERO, 1, 1_000)); + let (eip1559_hash, key0_hash) = (*eip1559.hash(), *key0.hash()); + pool.insert_validated(eip1559, 0).unwrap(); + pool.insert_validated(key0, 0).unwrap(); + let lane = (signer.address(), U256::ZERO); + assert_eq!(pool.lane_cursor(&lane), Some(LaneCursor::at(0))); + assert_eq!(hashes(&pool.pending_transactions()), vec![eip1559_hash, key0_hash]); + assert_eq!(pool.lane_pending_end(&lane), Some(2)); + assert_eq!( + pool.highest_consecutive_from(&lane, 0).map(|transaction| *transaction.hash()), + Some(key0_hash) + ); + assert_eq!(pool.len(), 2); + assert!(pool.size() > 0); + } + + #[test] + fn a_cursor_ahead_of_state_makes_the_run_behind_it_executable() { + let signer = signer(); + let lane = (signer.address(), U256::ZERO); + let mut pool = protocol_lane_pool(); + let cursor = LaneCursor { state_nonce: 0, next_nonce: 2 }; + + let behind = valid_pool_transaction(signed_1559(&signer, 1, 1_000)); + let error = pool.insert_validated_at(behind, cursor).unwrap_err(); + assert!( + matches!( + error.kind, + PoolErrorKind::InvalidTransaction(InvalidPoolTransactionError::Consensus( + InvalidTransactionError::NonceNotConsistent { tx: 1, state: 2 } + )) + ), + "{error:?}" + ); + assert!(pool.lane_cursor(&lane).is_none(), "a rejected insert creates no lane"); + + let head = valid_pool_transaction(signed_1559(&signer, 2, 1_000)); + let head_hash = *head.hash(); + let outcome = pool.insert_validated_at(head, cursor).unwrap(); + assert!(matches!(outcome.outcome.state, AddedTransactionState::Pending)); + assert_eq!(pool.lane_cursor(&lane), Some(cursor)); + assert_eq!(*pool.lanes_ahead_of_state(), HashMap::from_iter([(lane, 0)])); + assert_eq!(hashes(&pool.pending_transactions()), vec![head_hash]); + assert!(pool.lane_pending_transaction(&lane, 2).is_some()); + assert!(pool.highest_consecutive_from(&lane, 1).is_none(), "below the cursor"); + assert_eq!( + pool.highest_consecutive_from(&lane, 2).map(|transaction| *transaction.hash()), + Some(head_hash) + ); + } + + #[test] + fn set_lane_cursor_moves_the_executable_cursor_in_both_directions() { + let signer = signer(); + let lane = (signer.address(), U256::ZERO); + let mut pool = protocol_lane_pool(); + let ahead = LaneCursor { state_nonce: 0, next_nonce: 2 }; + let [second, third] = + [2, 3].map(|nonce| valid_pool_transaction(signed_1559(&signer, nonce, 1_000))); + let (second_hash, third_hash) = (*second.hash(), *third.hash()); + pool.insert_validated_at(second, ahead).unwrap(); + pool.insert_validated_at(third, ahead).unwrap(); + + // The run in front was mined up to the cursor: the lane no longer depends on it. + pool.set_lane_cursor(lane, LaneCursor::at(2)); + assert!(pool.lanes_ahead_of_state().is_empty()); + pool.set_lane_cursor(lane, ahead); + assert_eq!(*pool.lanes_ahead_of_state(), HashMap::from_iter([(lane, 0)])); + + // The standard run in front disappeared: nothing is executable, nothing is lost. + let update = pool.set_lane_cursor(lane, LaneCursor::at(0)); + assert_eq!(hashes(&update.queued), vec![second_hash, third_hash]); + assert!(update.discarded.is_empty() && update.promoted.is_empty()); + assert!(pool.pending_transactions().is_empty()); + assert!(pool.lanes_ahead_of_state().is_empty()); + assert_eq!(pool.lane_cursor(&lane), Some(LaneCursor::at(0))); + + // The run in front now reaches nonce 3: nonce 2 can never execute here. + let update = pool.set_lane_cursor(lane, LaneCursor { state_nonce: 0, next_nonce: 3 }); + assert_eq!(hashes(&update.discarded), vec![second_hash]); + assert_eq!(hashes(&update.promoted), vec![third_hash]); + assert!(!pool.contains(&second_hash)); + + // Canonical maintenance only moves the anchor forward and never below the cursor. + assert!(pool.advance_lane_nonce(lane, 2).discarded.is_empty()); + assert_eq!(pool.lane_cursor(&lane), Some(LaneCursor { state_nonce: 2, next_nonce: 3 })); + assert!(pool.advance_lane_nonce(lane, 1).discarded.is_empty()); + assert_eq!(pool.lane_cursor(&lane), Some(LaneCursor { state_nonce: 2, next_nonce: 3 })); + + // Past the lane's last transaction the lane is dropped entirely. + let update = pool.advance_lane_nonce(lane, 4); + assert_eq!(hashes(&update.discarded), vec![third_hash]); + assert!(pool.lane_cursor(&lane).is_none()); + assert_eq!(pool.len(), 0); + } + + #[test] + fn nonce_free_transactions_are_found_by_replay_id() { + let signer = signer(); + let mut pool = protocol_lane_pool(); + let transaction = valid_pool_transaction(signed_nonce_free_tx(&signer, 10_000, 0, 1_000)); + let replay_id = transaction.transaction.eip8130_replay_id().unwrap(); + let hash = *transaction.hash(); + pool.insert_validated(transaction, 0).unwrap(); + + assert_eq!(pool.nonce_free_transaction(&replay_id).map(|tx| *tx.hash()), Some(hash)); + assert!(pool.lanes_ahead_of_state().is_empty(), "nonce-free transactions have no lane"); + pool.remove_transactions(&[hash]); + assert!(pool.nonce_free_transaction(&replay_id).is_none()); + assert_eq!((pool.len(), pool.size()), (0, 0)); + } } diff --git a/crates/execution/txpool/src/validator.rs b/crates/execution/txpool/src/validator.rs index 007afbf2bd..cd77aa1e3b 100644 --- a/crates/execution/txpool/src/validator.rs +++ b/crates/execution/txpool/src/validator.rs @@ -74,6 +74,10 @@ pub enum BaseTxPoolError { /// Static validation label for the failure. reason: &'static str, }, + /// The transaction carries validity predicates but is an EIP-7702 + /// transaction or comes from an EIP-7702 delegated sender. + #[error("validity predicates are not allowed on EIP-7702 transactions or delegated senders")] + DelegatedValidityTransaction, } /// Resolved EIP-8130 actors and state data required to build the pool outcome. @@ -86,6 +90,8 @@ struct Eip8130ValidationState { payer_balance_after_auth: U256, sender_nonce: u64, sender_bytecode_hash: Option, + /// Whether the sender's code is an EIP-7702 delegation designator. + sender_delegated: bool, /// Payer-authentication gas metered on top of `gas_limit`. The execution /// path charges the operator fee on `gas_limit + payer_auth`, so admission /// must do the same to avoid admitting operator-fee-underfunded sponsored @@ -947,10 +953,48 @@ where bytecode_hash: state.sender_bytecode_hash, authorities: (state.payer != state.sender).then_some(vec![state.payer]), }; + let outcome = Self::reject_delegated_validity(outcome, state.sender_delegated); return self.apply_base_checks(outcome, state.payer_auth); } let outcome = self.inner.validate_one_with_state(origin, transaction, state); - self.apply_base_checks(outcome, 0) + // The inner validator admits a sender with code only when that code is + // an EIP-7702 delegation (EIP-3607), so any code hash here is one. + let sender_delegated = matches!( + &outcome, + TransactionValidationOutcome::Valid { bytecode_hash: Some(hash), .. } + if *hash != KECCAK_EMPTY + ); + self.apply_base_checks(Self::reject_delegated_validity(outcome, sender_delegated), 0) + } + + /// Rejects a transaction with validity predicates that is an EIP-7702 + /// transaction or whose sender is EIP-7702 delegated. + /// + /// Validity transactions live in their own pool, which does not enforce + /// reth's in-flight limits for delegated accounts and pending + /// authorizations. Keeping delegation out of validity transactions entirely + /// means those limits never need to span both pools. A sender delegated + /// after admission loses its validity transactions on the code change (see + /// `BaseTransactionPool::apply_state_diff`). + fn reject_delegated_validity( + outcome: TransactionValidationOutcome, + sender_delegated: bool, + ) -> TransactionValidationOutcome { + match outcome { + TransactionValidationOutcome::Valid { transaction, .. } + if !transaction.transaction().validity_conditions().is_empty() + && (sender_delegated + || transaction.transaction().authorization_list().is_some()) => + { + TransactionValidationOutcome::Invalid( + transaction.into_transaction(), + InvalidPoolTransactionError::other( + BaseTxPoolError::DelegatedValidityTransaction, + ), + ) + } + outcome => outcome, + } } /// Returns a low-cardinality sender authenticator label for metrics. @@ -1059,6 +1103,7 @@ where .map_err(|error| Self::state_read_error(error, "sender account read failed"))? .unwrap_or_default(); let protocol_nonce = sender_account.nonce; + let sender_delegated = Self::is_delegated(&*state, &sender_account)?; if is_create { Self::validate_eip8130_create_freshness(&*state, sender, &sender_account)?; } @@ -1304,6 +1349,7 @@ where payer_balance_after_auth: payer_account.balance.saturating_sub(payer_auth_charge), sender_nonce, sender_bytecode_hash: sender_account.bytecode_hash, + sender_delegated, payer_auth: intrinsic.payer_auth, watch_set, sender_locked, @@ -1521,6 +1567,20 @@ where }) } + /// Returns whether `account`'s code is an EIP-7702 delegation designator. + fn is_delegated( + state: &dyn StateProvider, + account: &reth_primitives_traits::Account, + ) -> Result { + let Some(code_hash) = account.bytecode_hash.filter(|hash| *hash != KECCAK_EMPTY) else { + return Ok(false); + }; + Ok(state + .bytecode_by_hash(&code_hash) + .map_err(|error| Self::state_read_error(error, "sender code read failed"))? + .is_some_and(|code| code.is_eip7702())) + } + fn account_has_code( state: &dyn StateProvider, address: Address, diff --git a/crates/execution/txpool/src/validity.rs b/crates/execution/txpool/src/validity.rs index 3a07c8a246..979f3ff41d 100644 --- a/crates/execution/txpool/src/validity.rs +++ b/crates/execution/txpool/src/validity.rs @@ -641,7 +641,7 @@ impl ValidatedTransactionExtensions for TransactionValidi fn extract(tx: &ValidPoolTransaction) -> Self { Self { - validity: tx.transaction.validity_predicates().to_vec(), + validity: tx.transaction.validity_predicates().cloned().collect(), validity_signature: tx.transaction.validity_signature(), } } @@ -1012,11 +1012,11 @@ mod tests { let transaction = extension.apply(transaction).unwrap(); - assert_eq!(transaction.validity_predicates(), expected); + assert_eq!(transaction.validity_predicates().cloned().collect::>(), expected); } #[test] - fn apply_compiles_timing_before_state_and_preserves_wire_order() { + fn apply_compiles_timing_before_state_and_forwards_timing_first() { let signed: BaseTransactionSigned = TxDeposit { source_hash: Default::default(), from: Address::ZERO, @@ -1042,7 +1042,7 @@ mod tests { op: ValidityOperator::GreaterThanOrEqual, value: U256::from(100), }; - // Submitted state-first; stored timing-first. + // Submitted state-first; stored and forwarded timing-first. let extension = TransactionValidity { validity: vec![state_predicate.clone(), timing_predicate.clone()], validity_signature: None, @@ -1051,8 +1051,8 @@ mod tests { let transaction = extension.apply(transaction).unwrap(); assert_eq!( - transaction.validity_predicates(), - [state_predicate.clone(), timing_predicate.clone()] + transaction.validity_predicates().cloned().collect::>(), + [timing_predicate.clone(), state_predicate.clone()] ); assert_eq!( transaction @@ -1280,7 +1280,8 @@ mod tests { // predicates; unlike the mempool ingress, `apply` must not reject them. let transaction = TransactionValidity::default().apply(transaction).unwrap(); - assert!(transaction.validity_predicates().is_empty()); + assert!(transaction.validity_conditions().is_empty()); + assert_eq!(transaction.validity_predicates().count(), 0); } #[test] diff --git a/crates/execution/txpool/src/validity_authorization.rs b/crates/execution/txpool/src/validity_authorization.rs index 25465075b9..6a073fd004 100644 --- a/crates/execution/txpool/src/validity_authorization.rs +++ b/crates/execution/txpool/src/validity_authorization.rs @@ -51,7 +51,7 @@ impl ValidityAuthorization { sidecar: TransactionValidity, mode: ValiditySignatureMode, ) -> Result { - mode.check(&sidecar.validity, sidecar.validity_signature.as_ref())?; + mode.check(!sidecar.validity.is_empty(), sidecar.validity_signature.as_ref())?; if mode != ValiditySignatureMode::Off && let Some(signature) = &sidecar.validity_signature { @@ -322,7 +322,11 @@ mod tests { let inbound = decoded.extensions.apply(inbound).unwrap(); assert_eq!(inbound.validity_signature(), Some(signature)); assert_eq!( - verify(&inbound, inbound.validity_predicates(), inbound.validity_signature().as_ref(),), + verify( + &inbound, + &inbound.validity_predicates().cloned().collect::>(), + inbound.validity_signature().as_ref(), + ), Ok(()) ); // Replacing predicates through the unsigned API must not retain stale authorization. @@ -331,7 +335,7 @@ mod tests { assert_eq!( verify( &unsigned, - unsigned.validity_predicates(), + &unsigned.validity_predicates().cloned().collect::>(), unsigned.validity_signature().as_ref(), ), Err(ValidityAuthorizationError::MissingSignature) @@ -514,7 +518,7 @@ mod tests { .unwrap(); let attached = tx.clone().with_validity(validity); assert_eq!(attached.validity_signature(), Some(signature)); - assert_eq!(attached.validity_predicates().len(), predicates.len()); + assert_eq!(attached.validity_predicates().count(), predicates.len()); } } diff --git a/crates/execution/txpool/src/validity_limits.rs b/crates/execution/txpool/src/validity_limits.rs new file mode 100644 index 0000000000..b12ced19e5 --- /dev/null +++ b/crates/execution/txpool/src/validity_limits.rs @@ -0,0 +1,94 @@ +//! Capacity limits for the validity pool. + +use reth_transaction_pool::{PoolConfig, SubPoolLimit}; + +use crate::two_d_nonce_pool::SidecarLimits; + +/// Operator-facing limits for the validity pool, the store that holds every +/// transaction carrying validity predicates. +/// +/// Each field falls back to the protocol pool's pending-subpool limit (or +/// account-slot limit) when unset, so by default the validity pool is as large +/// as the pending pool it was carved out of. Standard transactions and +/// validity transactions can then never evict one another: each side has its +/// own budget. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct ValidityPoolLimits { + /// Maximum number of validity transactions kept at once. + pub max_txs: Option, + /// Maximum summed `size()` of validity transactions kept at once, in bytes. + pub max_size: Option, + /// Maximum validity transactions per sender lane. + pub max_account_slots: Option, +} + +impl ValidityPoolLimits { + /// Limits that copy the protocol pool's pending limit in every dimension. + pub const fn inherit() -> Self { + Self { max_txs: None, max_size: None, max_account_slots: None } + } + + /// Caps the number of validity transactions. + #[must_use] + pub const fn with_max_txs(mut self, max_txs: usize) -> Self { + self.max_txs = Some(max_txs); + self + } + + /// Caps the summed size of validity transactions, in bytes. + #[must_use] + pub const fn with_max_size(mut self, max_size: usize) -> Self { + self.max_size = Some(max_size); + self + } + + /// Caps validity transactions per sender lane. + #[must_use] + pub const fn with_max_account_slots(mut self, max_account_slots: usize) -> Self { + self.max_account_slots = Some(max_account_slots); + self + } + + /// Resolves the limits against `config`, filling unset fields from its + /// pending subpool and account-slot limits. + pub(crate) fn resolve(self, config: &PoolConfig) -> SidecarLimits { + let pending = &config.pending_limit; + let capacity = SubPoolLimit { + max_txs: self.max_txs.unwrap_or(pending.max_txs), + max_size: self.max_size.unwrap_or(pending.max_size), + }; + let mut limits = SidecarLimits::from_pool_config(config); + limits.capacity = capacity; + limits.max_account_slots = self.max_account_slots.unwrap_or(config.max_account_slots); + limits + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn unset_fields_copy_the_pending_limit() { + let config = PoolConfig { + pending_limit: SubPoolLimit { max_txs: 123, max_size: 456 }, + max_account_slots: 7, + ..PoolConfig::default() + }; + let limits = ValidityPoolLimits::inherit().resolve(&config); + assert_eq!(limits.capacity, SubPoolLimit { max_txs: 123, max_size: 456 }); + assert_eq!(limits.max_account_slots, 7); + } + + #[test] + fn explicit_fields_override_the_pending_limit() { + let config = PoolConfig::default(); + let limits = ValidityPoolLimits::inherit() + .with_max_txs(5) + .with_max_size(6) + .with_max_account_slots(8) + .resolve(&config); + assert_eq!(limits.capacity, SubPoolLimit { max_txs: 5, max_size: 6 }); + assert_eq!(limits.max_account_slots, 8); + } +} diff --git a/crates/execution/txpool/src/validity_signature_mode.rs b/crates/execution/txpool/src/validity_signature_mode.rs index 15c4755b5d..35b989b584 100644 --- a/crates/execution/txpool/src/validity_signature_mode.rs +++ b/crates/execution/txpool/src/validity_signature_mode.rs @@ -4,7 +4,7 @@ use std::{fmt, str::FromStr}; use alloy_primitives::Signature; -use crate::{ValidityAuthorizationError, ValidityPredicate}; +use crate::ValidityAuthorizationError; /// Signature policy shared by forwarding ingress and builders. /// @@ -28,21 +28,13 @@ impl ValiditySignatureMode { /// signatures with [`crate::ValidityAuthorization`]. pub const fn check( self, - predicates: &[ValidityPredicate], + has_predicates: bool, signature: Option<&Signature>, ) -> Result<(), ValidityAuthorizationError> { - if matches!(self, Self::Off) { - return Ok(()); - } - if predicates.is_empty() { - return if signature.is_some() { - Err(ValidityAuthorizationError::UnexpectedSignature) - } else { - Ok(()) - }; - } - match (self, signature.is_some()) { - (Self::Required, false) => Err(ValidityAuthorizationError::MissingSignature), + match (self, has_predicates, signature.is_some()) { + (Self::Off, _, _) => Ok(()), + (_, false, true) => Err(ValidityAuthorizationError::UnexpectedSignature), + (Self::Required, true, false) => Err(ValidityAuthorizationError::MissingSignature), _ => Ok(()), } } @@ -93,6 +85,6 @@ mod tests { #[case] expected: Result<(), ValidityAuthorizationError>, ) { let signature = Signature::new(U256::ZERO, U256::ZERO, false); - assert_eq!(mode.check(&[], Some(&signature)), expected); + assert_eq!(mode.check(false, Some(&signature)), expected); } } diff --git a/crates/proof/zk/backend/src/succinct/cluster.rs b/crates/proof/zk/backend/src/succinct/cluster.rs index 864ab4a83c..3928400454 100644 --- a/crates/proof/zk/backend/src/succinct/cluster.rs +++ b/crates/proof/zk/backend/src/succinct/cluster.rs @@ -244,7 +244,7 @@ impl ClusterZkProver { async { // The upstream constructor consumes the endpoint string, and the proof config // needs to retain it for request submission. - ClusterServiceClient::new(cluster_rpc.clone()).await.map_err(|error| { + ClusterServiceClient::new(cluster_rpc.clone(), None).await.map_err(|error| { SuccinctZkProverBuildError::boxed_operation( "failed to create SP1 cluster client", error.into(), diff --git a/crates/proof/zk/programs/succinct/Cargo.lock b/crates/proof/zk/programs/succinct/Cargo.lock index f85e1d4929..6cc10843e1 100644 --- a/crates/proof/zk/programs/succinct/Cargo.lock +++ b/crates/proof/zk/programs/succinct/Cargo.lock @@ -620,7 +620,7 @@ checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" [[package]] name = "base-common-chains" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-chains", "alloy-eips", @@ -635,7 +635,7 @@ dependencies = [ [[package]] name = "base-common-consensus" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-consensus", "alloy-eips", @@ -658,7 +658,7 @@ dependencies = [ [[package]] name = "base-common-eip8130" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-primitives", "base-common-consensus", @@ -667,7 +667,7 @@ dependencies = [ [[package]] name = "base-common-evm" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-consensus", "alloy-eips", @@ -686,11 +686,11 @@ dependencies = [ [[package]] name = "base-common-flz" -version = "1.5.0" +version = "1.5.2" [[package]] name = "base-common-genesis" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-chains", "alloy-consensus", @@ -707,7 +707,7 @@ dependencies = [ [[package]] name = "base-common-l1-fees" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-primitives", "base-common-flz", @@ -716,7 +716,7 @@ dependencies = [ [[package]] name = "base-common-precompiles" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-evm", "alloy-primitives", @@ -731,7 +731,7 @@ dependencies = [ [[package]] name = "base-common-rpc-types-engine" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-consensus", "alloy-eips", @@ -747,7 +747,7 @@ dependencies = [ [[package]] name = "base-consensus-derive" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-consensus", "alloy-eips", @@ -768,7 +768,7 @@ dependencies = [ [[package]] name = "base-consensus-upgrades" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-eips", "alloy-primitives", @@ -777,11 +777,11 @@ dependencies = [ [[package]] name = "base-metrics" -version = "1.5.0" +version = "1.5.2" [[package]] name = "base-precompile-macros" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-primitives", "proc-macro2", @@ -791,7 +791,7 @@ dependencies = [ [[package]] name = "base-precompile-storage" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-evm", "alloy-primitives", @@ -805,7 +805,7 @@ dependencies = [ [[package]] name = "base-proof" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-consensus", "alloy-eips", @@ -838,7 +838,7 @@ dependencies = [ [[package]] name = "base-proof-driver" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-consensus", "alloy-evm", @@ -858,7 +858,7 @@ dependencies = [ [[package]] name = "base-proof-executor" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-consensus", "alloy-eips", @@ -880,7 +880,7 @@ dependencies = [ [[package]] name = "base-proof-mpt" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-primitives", "alloy-rlp", @@ -891,7 +891,7 @@ dependencies = [ [[package]] name = "base-proof-preimage" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-primitives", "async-trait", @@ -903,7 +903,7 @@ dependencies = [ [[package]] name = "base-proof-primitives" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-chains", "alloy-eips", @@ -929,7 +929,7 @@ dependencies = [ [[package]] name = "base-proof-zk-utils" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloy-consensus", "alloy-eips", @@ -965,7 +965,7 @@ dependencies = [ [[package]] name = "base-protocol" -version = "1.5.0" +version = "1.5.2" dependencies = [ "alloc-no-stdlib", "alloy-consensus", @@ -1849,7 +1849,7 @@ dependencies = [ [[package]] name = "k256" version = "0.13.4" -source = "git+https://github.com/sp1-patches/elliptic-curves?tag=patch-k256-13.4-sp1-6.2.0#41374de1febd88e67faa695a5641ae46460a8cb6" +source = "git+https://github.com/sp1-patches/elliptic-curves?tag=patch-k256-13.4-sp1-6.10.0#cd63f2b5a832e15fa98cedb73df2ce89bb1a032a" dependencies = [ "cfg-if", "ecdsa 0.16.9 (git+https://github.com/sp1-patches/signatures?tag=sp1-skip-verify-on-recovery)", @@ -1872,9 +1872,9 @@ dependencies = [ [[package]] name = "kzg-rs" -version = "0.2.8" +version = "0.2.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee8b4f55c3dedcfaa8668de1dfc8469e7a32d441c28edf225ed1f566fb32977d" +checksum = "433994161b1b86020801f5486deebcd90c1e654d5713e323ef7ef636445e9b9d" dependencies = [ "ff", "hex", @@ -2160,7 +2160,7 @@ dependencies = [ [[package]] name = "p256" version = "0.13.2" -source = "git+https://github.com/sp1-patches/elliptic-curves?tag=patch-p256-13.2-sp1-6.2.0#778543de72a8160a9ce253870da1efae6b18e6d3" +source = "git+https://github.com/sp1-patches/elliptic-curves?tag=patch-p256-13.2-sp1-6.10.0#f074fbff1b10b270f3c5ae255443d589c97b3175" dependencies = [ "ecdsa 0.16.9 (registry+https://github.com/rust-lang/crates.io-index)", "elliptic-curve", @@ -2424,7 +2424,7 @@ dependencies = [ [[package]] name = "primeorder" version = "0.13.1" -source = "git+https://github.com/sp1-patches/elliptic-curves?tag=patch-p256-13.2-sp1-6.2.0#778543de72a8160a9ce253870da1efae6b18e6d3" +source = "git+https://github.com/sp1-patches/elliptic-curves?tag=patch-p256-13.2-sp1-6.10.0#f074fbff1b10b270f3c5ae255443d589c97b3175" dependencies = [ "elliptic-curve", ] @@ -3131,7 +3131,7 @@ dependencies = [ [[package]] name = "sha2" version = "0.10.9" -source = "git+https://github.com/sp1-patches/RustCrypto-hashes?tag=patch-sha2-0.10.9-sp1-6.2.0#e48b656ebc806117554bb33c2f8687e4637e37ff" +source = "git+https://github.com/sp1-patches/RustCrypto-hashes?tag=patch-sha2-0.10.9-sp1-6.10.0#e48b656ebc806117554bb33c2f8687e4637e37ff" dependencies = [ "cfg-if", "cpufeatures 0.2.17", @@ -3204,9 +3204,9 @@ checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" [[package]] name = "slop-algebra" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "733d477823f02646fa8025c5e2d4227ed806ca33a469def68994f63122f9f106" +checksum = "6b44536b0d3267951634cd7f768c3ba73e0b64e05b2ca4783616d4e54cbf324d" dependencies = [ "itertools 0.14.0", "p3-field", @@ -3215,9 +3215,9 @@ dependencies = [ [[package]] name = "slop-bn254" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0207095beb89a56e3875a0ca43e5782a2efc8637124b43ce453df42410e12a4c" +checksum = "ae41b4d69a3cd4848f91f21e768cf4e623529dab91b0eb85d0fe80455c7a0d4e" dependencies = [ "ff", "p3-bn254-fr", @@ -3230,9 +3230,9 @@ dependencies = [ [[package]] name = "slop-challenger" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfc4121b9b84df835c8a6677a90dc9935562172a4a370758987407c793633334" +checksum = "aebbf4fde396e40eeb534981be75547ab89b4a348c4136961627c06d0cbcd51f" dependencies = [ "p3-challenger", "serde", @@ -3242,9 +3242,9 @@ dependencies = [ [[package]] name = "slop-koala-bear" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b27fec4aae6811dbf6a384cc7993e74621aede443356ef67c301c19c1f302865" +checksum = "627eb3ed4c0918f2dd2e70d2a1d312049ec3d2aa57fa0f7dbe2c3d9cd94993e4" dependencies = [ "lazy_static", "p3-koala-bear", @@ -3257,24 +3257,24 @@ dependencies = [ [[package]] name = "slop-poseidon2" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "13388d3c92bac6214ef7585869b968cf895d2715ebbb8b1a0e1464dd2ada30b2" +checksum = "51676e05a2891b15cd624b74c8aaa03cc10d5eb51542eeab5019294ad47c21fb" dependencies = [ "p3-poseidon2", ] [[package]] name = "slop-primitives" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93f569cac42b24e672f6b06ffde272d4086c2c2620f16d31c1d1ce88cddff681" +checksum = "91992a8acf6a9a4c52dee00142840d30c8cc9bbb3ff30ca3d56dcde428a591dd" [[package]] name = "slop-symmetric" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "693b50b55c3f5a2bb20dd3b975da79b80e4bf15b839b90f82e5dc7ba7262d3dc" +checksum = "9cb116ad23028ab41fe5228905a17bbeaf7b5fb7ec6790163bb8e8fef7092ffc" dependencies = [ "p3-symmetric", ] @@ -3290,9 +3290,9 @@ dependencies = [ [[package]] name = "sp1-lib" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d4bc4b2a4b6e5fd3105912c722f51de9d66e2440a44e4ddffaad04bfba42bdf" +checksum = "e8a36aae1b35d6eae35b479d23c818bb32c22e7c0e34cc1e1bc7c68319c94368" dependencies = [ "bincode", "elliptic-curve", @@ -3302,9 +3302,9 @@ dependencies = [ [[package]] name = "sp1-primitives" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d387e4babbee68d30774ef52aad88aad3607eda8145a0badfa823e1759187c0c" +checksum = "f2ad891be0e48ab9ca110486f122cf799a707e5ca327a75669cfc2792b086882" dependencies = [ "bincode", "blake3", @@ -3326,9 +3326,9 @@ dependencies = [ [[package]] name = "sp1-zkvm" -version = "6.8.1" +version = "6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "38ba654f5bcc242a388a82a9a2ab3578b73093b4f460357a1e255e9defe3abf9" +checksum = "4a82fd01010cb37e0a3b1ffdac814641dcb39040f5e4eef86bad5ee1edc9b54d" dependencies = [ "cfg-if", "critical-section", @@ -3345,9 +3345,9 @@ dependencies = [ [[package]] name = "sp1_bls12_381" -version = "0.8.0-sp1-6.0.0" +version = "0.8.0-sp1-6.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f23e41cd36168cc2e51e5d3e35ff0c34b204d945769a65591a76286d04b51e43" +checksum = "9b2ad1e32db3c05d37e4385e3e48795f14d687a5797bf2eee0a7753d7eaa9661" dependencies = [ "cfg-if", "ff", @@ -3816,14 +3816,14 @@ dependencies = [ [[patch.unused]] name = "sha3" version = "0.10.8" -source = "git+https://github.com/sp1-patches/RustCrypto-hashes?tag=patch-sha3-0.10.8-sp1-6.2.0#0a16ae7acd5cd5fbb432d884bd4aae2764a18cf7" +source = "git+https://github.com/sp1-patches/RustCrypto-hashes?tag=patch-sha3-0.10.8-sp1-6.10.0#0a16ae7acd5cd5fbb432d884bd4aae2764a18cf7" [[patch.unused]] name = "substrate-bn" version = "0.6.0" -source = "git+https://github.com/sp1-patches/bn?tag=patch-0.6.0-sp1-6.2.0-substrate-bn#b9cd95a749de1f20ac786178f9f8754f79a5ad55" +source = "git+https://github.com/sp1-patches/bn?tag=patch-0.6.0-sp1-6.10.0-substrate-bn#ab4d0085567695967cf34c23b3090f1ae02a03d1" [[patch.unused]] name = "tiny-keccak" version = "2.0.2" -source = "git+https://github.com/sp1-patches/tiny-keccak?tag=patch-2.0.2-sp1-6.2.0#c3f95bcc35b391101d0cf0abe91ea4c8423868b0" +source = "git+https://github.com/sp1-patches/tiny-keccak?tag=patch-2.0.2-sp1-6.10.0#8fca2536a39c02691f2877e838a2e5b016e892cc" diff --git a/crates/proof/zk/programs/succinct/Cargo.toml b/crates/proof/zk/programs/succinct/Cargo.toml index d1dacf6899..3f27663474 100644 --- a/crates/proof/zk/programs/succinct/Cargo.toml +++ b/crates/proof/zk/programs/succinct/Cargo.toml @@ -41,8 +41,8 @@ base-proof = { path = "../../../proof", default-features = false } base-proof-zk-utils = { path = "../../utils" } # SP1 (must match the cargo-prove Docker tag) -sp1-lib = { version = "=6.8.1", features = ["verify"] } -sp1-zkvm = { version = "=6.8.1", features = ["verify"] } +sp1-lib = { version = "=6.10.0", features = ["verify"] } +sp1-zkvm = { version = "=6.10.0", features = ["verify"] } # alloy (must match root Cargo.toml versions) alloy-consensus = { version = "2.0.5", default-features = false } @@ -76,9 +76,9 @@ codegen-units = 16 # the zkVM target (`target_os = "zkvm"`). On native builds the # patches are no-ops. [patch.crates-io] -tiny-keccak = { git = "https://github.com/sp1-patches/tiny-keccak", tag = "patch-2.0.2-sp1-6.2.0" } -p256 = { git = "https://github.com/sp1-patches/elliptic-curves", tag = "patch-p256-13.2-sp1-6.2.0" } -k256 = { git = "https://github.com/sp1-patches/elliptic-curves", tag = "patch-k256-13.4-sp1-6.2.0" } -substrate-bn = { git = "https://github.com/sp1-patches/bn", tag = "patch-0.6.0-sp1-6.2.0-substrate-bn" } -sha2 = { git = "https://github.com/sp1-patches/RustCrypto-hashes", package = "sha2", tag = "patch-sha2-0.10.9-sp1-6.2.0" } -sha3 = { git = "https://github.com/sp1-patches/RustCrypto-hashes", package = "sha3", tag = "patch-sha3-0.10.8-sp1-6.2.0" } +tiny-keccak = { git = "https://github.com/sp1-patches/tiny-keccak", tag = "patch-2.0.2-sp1-6.10.0" } +p256 = { git = "https://github.com/sp1-patches/elliptic-curves", tag = "patch-p256-13.2-sp1-6.10.0" } +k256 = { git = "https://github.com/sp1-patches/elliptic-curves", tag = "patch-k256-13.4-sp1-6.10.0" } +substrate-bn = { git = "https://github.com/sp1-patches/bn", tag = "patch-0.6.0-sp1-6.10.0-substrate-bn" } +sha2 = { git = "https://github.com/sp1-patches/RustCrypto-hashes", package = "sha2", tag = "patch-sha2-0.10.9-sp1-6.10.0" } +sha3 = { git = "https://github.com/sp1-patches/RustCrypto-hashes", package = "sha3", tag = "patch-sha3-0.10.8-sp1-6.10.0" } diff --git a/crates/utilities/reth-cli/README.md b/crates/utilities/reth-cli/README.md index d04f2abb3f..749f5c3e7b 100644 --- a/crates/utilities/reth-cli/README.md +++ b/crates/utilities/reth-cli/README.md @@ -5,6 +5,7 @@ Reth-specific CLI utilities for Base execution layer binaries. ## Overview - **`init_reth!`**: Initializes Reth's global version metadata for P2P identification and logging. +- **`init_engine!`**: Sets Reth engine persistence defaults sized for 200ms blocks. - **Snapshot manifests**: Provides the Base-owned archive generator shared by the execution node's `snapshot-manifest` command and the snapshotter sidecar. Existing archives can be reused without recompression after verifying their uncompressed BLAKE3 hashes. @@ -20,6 +21,7 @@ base-reth-cli = { git = "https://github.com/base/base" } fn main() { base_reth_cli::init_reth!(); base_reth_cli::init_snapshots!(); + base_reth_cli::init_engine!(); } ``` diff --git a/crates/utilities/reth-cli/src/engine.rs b/crates/utilities/reth-cli/src/engine.rs new file mode 100644 index 0000000000..d152f42adc --- /dev/null +++ b/crates/utilities/reth-cli/src/engine.rs @@ -0,0 +1,59 @@ +//! Reth engine persistence-default initialization utilities. + +use reth_node_core::args::DefaultEngineValues; + +/// Reth's defaults (7 / 5 / 16) are block counts sized for 2s blocks; these keep the same +/// time margins at Denim's 200ms block interval. +const PERSISTENCE_THRESHOLD: u64 = 70; +const MEMORY_BLOCK_BUFFER_TARGET: u64 = 70; +const PERSISTENCE_BACKPRESSURE_THRESHOLD: u64 = 160; + +/// Reth engine default initialization for Base execution layer binaries. +#[derive(Debug, Clone, Copy, Eq, PartialEq)] +pub struct Engine; + +impl Engine { + /// Initializes Reth's global engine persistence defaults. + /// + /// Explicit `--engine.*` flags still override these values. + /// + /// ### Panics + /// + /// Panics if the engine defaults were already initialized. + pub fn init_engine() { + DefaultEngineValues::default() + .with_persistence_threshold(PERSISTENCE_THRESHOLD) + .with_memory_block_buffer_target(MEMORY_BLOCK_BUFFER_TARGET) + .with_persistence_backpressure_threshold(PERSISTENCE_BACKPRESSURE_THRESHOLD) + .try_init() + .expect("failed to initialize engine defaults"); + } +} + +/// Initializes Reth's global engine persistence defaults. +/// +/// Use this in execution layer binaries (base-node-reth, base-builder) before parsing the CLI. +#[macro_export] +macro_rules! init_engine { + () => { + $crate::Engine::init_engine() + }; +} + +#[cfg(test)] +mod tests { + use reth_node_core::args::EngineArgs; + + use super::*; + + #[test] + fn engine_defaults_scale_for_denim_blocks() { + Engine::init_engine(); + + let args = EngineArgs::default(); + assert_eq!(args.persistence_threshold, 70); + assert_eq!(args.memory_block_buffer_target(), 70); + assert_eq!(args.persistence_backpressure_threshold(), 160); + args.validate().unwrap(); + } +} diff --git a/crates/utilities/reth-cli/src/lib.rs b/crates/utilities/reth-cli/src/lib.rs index 7b04351e53..16167705b0 100644 --- a/crates/utilities/reth-cli/src/lib.rs +++ b/crates/utilities/reth-cli/src/lib.rs @@ -10,6 +10,9 @@ mod version; pub use version::Version; +mod engine; +pub use engine::Engine; + mod snapshots; pub use snapshots::Snapshots; diff --git a/docs/transaction-events/README.md b/docs/transaction-events/README.md index bb3acaa166..4ebb1ae9ae 100644 --- a/docs/transaction-events/README.md +++ b/docs/transaction-events/README.md @@ -223,7 +223,8 @@ Forwarding: A mempool node forwards each pending transaction to every configured builder, and forwards it again every `--tx-forwarding-resend-after-ms` while it stays -pending. Each destination has its own pool reader and queue, so the mempool +pending. A transaction carrying validity predicates is resent only after its +forward to that builder failed. Each destination has its own pool reader and queue, so the mempool node does not journal the hand-off: it no longer emits `TXPOOL_BUILDER_CONSUMED` or `TXPOOL_BUILDER_FORWARD_SUCCESS`. Successful delivery is recorded by the receiving builder as diff --git a/etc/docs-index.toml b/etc/docs-index.toml index f22ebbcc17..f8938b83a3 100644 --- a/etc/docs-index.toml +++ b/etc/docs-index.toml @@ -156,7 +156,7 @@ digest = "1ccdbc3fb7" ["crates/batcher/comp/README.md"] summary = "base-comp: incremental Brotli channel compression (CompressionStream, BrotliLevel) producing protocol-compatible channels; batch encoding and framing live in batcher-encoder." -digest = "ce9421e523" +digest = "7d603b8774" ["crates/batcher/core/README.md"] summary = "base-batcher-core: async BatchDriver, SubmissionQueue for L1 transactions, and the DA throttle (Off, Step, Linear, Quadratic strategies) that publishes the DA limits the block builders apply." @@ -164,7 +164,7 @@ digest = "de7352c3a8" ["crates/batcher/encoder/README.md"] summary = "base-batcher-encoder: synchronous, no-I/O encoder from L2 blocks to L1 submissions. Covers channel closing rules, blob framing, metric labels, and BatchEncoder usage." -digest = "06022a9fa3" +digest = "e8624eb69c" ["crates/batcher/service/README.md"] summary = "base-batcher-service: BatcherConfig and BatcherService, which wire the encoder, block source, tx manager, and driver into a running batcher process." @@ -355,8 +355,8 @@ summary = "Proofs-history ExEx: versioned MDBX trie store serving historical eth digest = "13927f28e4" ["crates/execution/flashblocks-node/README.md"] -summary = "FlashblocksExtension for the Base node: install_ext usage (install last), CLI flags (--flashblocks-url, --max-pending-blocks-depth, cached-execution), and the state, subscriber, RPC components." -digest = "557e4ec2a8" +summary = "FlashblocksExtension for the Base node: install_ext usage (install last), CLI flags (--flashblocks-url, --max-pending-blocks-depth, cached-execution), state, subscriber, RPC, and pool deadline expiry." +digest = "bda32eab6a" ["crates/execution/flashblocks-node/tests/README.md"] summary = "Explains why flashblocks-node integration tests live in tests/ (public API only, single compilation unit with test utils) versus private-internals unit tests in src/. Read before adding tests." @@ -408,23 +408,23 @@ digest = "559e8e9f95" ["crates/execution/tx-forwarding/README.md"] summary = "base-tx-forwarding: forwards mempool txs to builder RPCs via per-destination queues fed by FIFO and tip-priority lanes; dedup, batching, resend; spawn_requests for custom ForwardRequest; CLI flags." -digest = "824cdc212c" +digest = "bd5709a547" ["crates/execution/txpool-rpc/README.md"] summary = "Txpool RPC admin/status APIs and validity submissions; shared staged signature modes, signature rejection metrics, predicate forwarding and sequencer proxy behavior." digest = "adef27d8c3" ["crates/execution/txpool-tracing/README.md"] -summary = "Txpool tracing extension: tracks mempool lifecycle events, txpool_transactionStatus RPC, enabling CLI flags, and reth_transaction_tracing_tx_event histogram. Use to debug tx latency or drops." -digest = "e58f6b44fb" +summary = "Txpool tracing extension: tracks mempool lifecycle events, txpool_transactionStatus RPC, enabling CLI flags, tx_event histogram and flow-labeled inclusion metrics. Use to debug tx latency or drops." +digest = "469e9f8628" ["crates/execution/txpool/PROPAGATION_PIPELINE.md"] summary = "Plan to forward mempool txs to builders via base_insertValidatedTransaction in 4 PRs (consumer, forwarder, builder endpoint, node integration). Only PR 1 is marked done. Names a crates/txpool/ path." digest = "ce42872576" ["crates/execution/txpool/README.md"] -summary = "Base txpool validation/ordering, builder wire extensions, staged EIP-712 predicate authorization, wallet signing contract, rejection metrics, and rollout/security limits." -digest = "f8413da3b9" +summary = "Base txpool validation/ordering, the separate validity pool, builder wire extensions, staged EIP-712 predicate authorization, wallet signing contract, rejection metrics, and rollout/security limits." +digest = "446353ed97" ["crates/infra/audit/README.md"] summary = "audit-archiver-lib: ingests Vector transaction-event/v1 batches into Postgres idempotently and serves JSON-RPC queries; covers the migrations and schema snapshot regeneration." @@ -615,8 +615,8 @@ summary = "One-line crate description: typed Optimism rollup JSON-RPC provider e digest = "d0aafc0327" ["crates/utilities/reth-cli/README.md"] -summary = "Reth-specific CLI utilities for Base EL binaries: init_reth! for version metadata, and the snapshot manifest archive generator shared by snapshot-manifest and the snapshotter sidecar." -digest = "4bc018ccf3" +summary = "Reth CLI utilities for Base EL binaries: init_reth! (version metadata), init_engine! (200ms-block persistence defaults), and the shared snapshot manifest archive generator." +digest = "72f0488bd4" ["crates/utilities/retry/README.md"] summary = "Shared RetryConfig wrapper over backon exponential retry builders, giving services common delay normalization, jitter, and bounded or unbounded retry limits. Use when adding retry logic." @@ -680,7 +680,7 @@ digest = "e87f3373b8" ["docs/transaction-events/README.md"] summary = "Spec for transaction-event/v1, the JSONL business event journal for transaction observability: config fields, envelope, event vocabulary, Vector to audit-archiver ingest, Postgres retention, examples." -digest = "24f8b4422b" +digest = "7a35820370" ["docs/transaction-events/tips-parity-checklist.md"] summary = "Checklist for verifying TIPS UI audit-archiver/Postgres responses match the S3-backed routes (block, txn, bundle, rejected) before switching TIPS_UI_QUERY_BACKEND=audit." diff --git a/etc/just/succinct.just b/etc/just/succinct.just index ad56b6035b..7c274d4760 100644 --- a/etc/just/succinct.just +++ b/etc/just/succinct.just @@ -1,5 +1,5 @@ _cargo_prove := "~/.sp1/bin/cargo-prove" -_sp1_tag := "v6.8.1" +_sp1_tag := "v6.10.0" _repo_root := justfile_directory() _succinct_dir := justfile_directory() / "crates/proof/zk/programs/succinct" @@ -12,7 +12,7 @@ build-elfs: build-range-elf build-agg-elf # Build the range program ELF (reproducible docker build) # -# SP1 v6.8.1's Docker rustc is 1.96 (`succinct-1.96.0-64bit-v2`), matching +# SP1 v6.10.0's Docker rustc is 1.96, matching # the workspace toolchain. Ignore package rust-version so the guest still # compiles if a dependency declares a newer one. build-range-elf: diff --git a/etc/upstream-pins/reth.toml b/etc/upstream-pins/reth.toml index 6d3b28bab2..f1aa026034 100644 --- a/etc/upstream-pins/reth.toml +++ b/etc/upstream-pins/reth.toml @@ -7,8 +7,8 @@ # To return to an official Reth tag, edit this file and run `just pin-reth`. repository = "https://github.com/base/reth" -reference = "base-v2.5.2.8" -rev = "7134be3b51cf9d9504f988bff3266a23085f0bd9" +reference = "base-v2.5.2.11" +rev = "9e380dd9b22f95649fb1b5e84c85449ef03f5dca" [upstream_base] tag = "v2.5.2" @@ -25,3 +25,15 @@ head = "f47fd63a3762d8add2e07f1d4afa9363e60d0853" [[patches]] pr = "https://github.com/paradigmxyz/reth/pull/26900" head = "8fce12d1ee3ffabac19cdc702c0df486e6754607" + +[[patches]] +pr = "https://github.com/base/reth/pull/27" +head = "c9f3652d80949c030eb890e5511d39ff74ee4990" + +[[patches]] +pr = "https://github.com/base/reth/pull/38" +head = "cf0d35dec240e190c4924eb5445e211e969b681c" + +[[patches]] +pr = "https://github.com/base/reth/pull/39" +head = "88aad0abfb3d912e2623c0036fea165897ef8403" diff --git a/llms-full.txt b/llms-full.txt index 115950c7f3..c090bc42ae 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -157,7 +157,7 @@ Find a doc through the index below instead of grepping the tree. After adding, m - [crates/execution/eip8130/README.md](crates/execution/eip8130/README.md): Native EIP-8130 account abstraction validation: enshrined authenticators (not precompiles) needing parity with contracts, validation layers, intrinsic gas formula, EIP-7623 floor, fee checks. - [crates/execution/evm/README.md](crates/execution/evm/README.md): BaseEvmConfig implementing reth ConfigureEvm, mapping timestamps to SpecId for upgrade-aware EVM environments, and re-exporting executor types. Use when changing Base EVM setup or fork specs. - [crates/execution/exex/README.md](crates/execution/exex/README.md): Proofs-history ExEx: versioned MDBX trie store serving historical eth_getProof without OOM, with pruner, debug_proofsSyncStatus, prune/unwind commands, metrics. Use to operate or debug proofs history. -- [crates/execution/flashblocks-node/README.md](crates/execution/flashblocks-node/README.md): FlashblocksExtension for the Base node: install_ext usage (install last), CLI flags (--flashblocks-url, --max-pending-blocks-depth, cached-execution), and the state, subscriber, RPC components. +- [crates/execution/flashblocks-node/README.md](crates/execution/flashblocks-node/README.md): FlashblocksExtension for the Base node: install_ext usage (install last), CLI flags (--flashblocks-url, --max-pending-blocks-depth, cached-execution), state, subscriber, RPC, and pool deadline expiry. - [crates/execution/flashblocks-node/tests/README.md](crates/execution/flashblocks-node/tests/README.md): Explains why flashblocks-node integration tests live in tests/ (public API only, single compilation unit with test utils) versus private-internals unit tests in src/. Read before adding tests. - [crates/execution/flashblocks/README.md](crates/execution/flashblocks/README.md): Flashblocks pending-state management: StateProcessor, subscriber, reconciler, the staleness invariant and depth limit, canonical reconciliation, and pending-aware RPC methods and subscriptions. - [crates/execution/metering/README.md](crates/execution/metering/README.md): Metering RPC: base_meterBundle, base_meterBlockByHash and base_meterBlockByNumber, plus the pseudo-opcode vocabulary for meteredOpcodes filters. Use to profile tx or block execution timing and gas. @@ -172,9 +172,9 @@ Find a doc through the index below instead of grepping the tree. After adding, m - [crates/execution/trie/src/db/schema.md](crates/execution/trie/src/db/schema.md): MDBX proof-history DB schema: versioned DupSort tables, tombstones, BlockChangeSet reverse index, proof window keys, cursor reads, append-only writes, block 0 backfill. For eth_getProof storage. - [crates/execution/tx-forwarding/README.md](crates/execution/tx-forwarding/README.md): base-tx-forwarding: forwards mempool txs to builder RPCs via per-destination queues fed by FIFO and tip-priority lanes; dedup, batching, resend; spawn_requests for custom ForwardRequest; CLI flags. - [crates/execution/txpool-rpc/README.md](crates/execution/txpool-rpc/README.md): Txpool RPC admin/status APIs and validity submissions; shared staged signature modes, signature rejection metrics, predicate forwarding and sequencer proxy behavior. -- [crates/execution/txpool-tracing/README.md](crates/execution/txpool-tracing/README.md): Txpool tracing extension: tracks mempool lifecycle events, txpool_transactionStatus RPC, enabling CLI flags, and reth_transaction_tracing_tx_event histogram. Use to debug tx latency or drops. +- [crates/execution/txpool-tracing/README.md](crates/execution/txpool-tracing/README.md): Txpool tracing extension: tracks mempool lifecycle events, txpool_transactionStatus RPC, enabling CLI flags, tx_event histogram and flow-labeled inclusion metrics. Use to debug tx latency or drops. - [crates/execution/txpool/PROPAGATION_PIPELINE.md](crates/execution/txpool/PROPAGATION_PIPELINE.md): Plan to forward mempool txs to builders via base_insertValidatedTransaction in 4 PRs (consumer, forwarder, builder endpoint, node integration). Only PR 1 is marked done. Names a crates/txpool/ path. -- [crates/execution/txpool/README.md](crates/execution/txpool/README.md): Base txpool validation/ordering, builder wire extensions, staged EIP-712 predicate authorization, wallet signing contract, rejection metrics, and rollout/security limits. +- [crates/execution/txpool/README.md](crates/execution/txpool/README.md): Base txpool validation/ordering, the separate validity pool, builder wire extensions, staged EIP-712 predicate authorization, wallet signing contract, rejection metrics, and rollout/security limits. ## Crates: infra @@ -231,7 +231,7 @@ Find a doc through the index below instead of grepping the tree. After adding, m - [crates/utilities/l1-head/README.md](crates/utilities/l1-head/README.md): One-line crate description: shared L1 head calculation helpers for proof services. No usage details in the README. - [crates/utilities/metrics/README.md](crates/utilities/metrics/README.md): Metrics helpers: define_metrics! for standardized Prometheus registration (optional custom struct), timed! for durations, inflight! for in-flight tracking. Use when instrumenting crates. - [crates/utilities/optimism-rpc/README.md](crates/utilities/optimism-rpc/README.md): One-line crate description: typed Optimism rollup JSON-RPC provider extensions and response types. No usage details in the README. -- [crates/utilities/reth-cli/README.md](crates/utilities/reth-cli/README.md): Reth-specific CLI utilities for Base EL binaries: init_reth! for version metadata, and the snapshot manifest archive generator shared by snapshot-manifest and the snapshotter sidecar. +- [crates/utilities/reth-cli/README.md](crates/utilities/reth-cli/README.md): Reth CLI utilities for Base EL binaries: init_reth! (version metadata), init_engine! (200ms-block persistence defaults), and the shared snapshot manifest archive generator. - [crates/utilities/retry/README.md](crates/utilities/retry/README.md): Shared RetryConfig wrapper over backon exponential retry builders, giving services common delay normalization, jitter, and bounded or unbounded retry limits. Use when adding retry logic. - [crates/utilities/ring-buffer/README.md](crates/utilities/ring-buffer/README.md): Generic fixed-capacity ring buffer (VecDeque of (I, V) entries) for position-keyed replay; the oldest entry is evicted on push when full. - [crates/utilities/runtime/README.md](crates/utilities/runtime/README.md): Async runtime abstraction (Clock, Spawner, Cancellation, Runtime traits) with TokioRuntime for production and a seeded deterministic executor with virtual time for reproducible tests. diff --git a/llms.txt b/llms.txt index 4b9376ab80..2c9ef66556 100644 --- a/llms.txt +++ b/llms.txt @@ -116,7 +116,7 @@ - [crates/execution/eip8130/README.md](crates/execution/eip8130/README.md): Native EIP-8130 account abstraction validation: enshrined authenticators (not precompiles) needing parity with contracts, validation layers, intrinsic gas formula, EIP-7623 floor, fee checks. - [crates/execution/evm/README.md](crates/execution/evm/README.md): BaseEvmConfig implementing reth ConfigureEvm, mapping timestamps to SpecId for upgrade-aware EVM environments, and re-exporting executor types. Use when changing Base EVM setup or fork specs. - [crates/execution/exex/README.md](crates/execution/exex/README.md): Proofs-history ExEx: versioned MDBX trie store serving historical eth_getProof without OOM, with pruner, debug_proofsSyncStatus, prune/unwind commands, metrics. Use to operate or debug proofs history. -- [crates/execution/flashblocks-node/README.md](crates/execution/flashblocks-node/README.md): FlashblocksExtension for the Base node: install_ext usage (install last), CLI flags (--flashblocks-url, --max-pending-blocks-depth, cached-execution), and the state, subscriber, RPC components. +- [crates/execution/flashblocks-node/README.md](crates/execution/flashblocks-node/README.md): FlashblocksExtension for the Base node: install_ext usage (install last), CLI flags (--flashblocks-url, --max-pending-blocks-depth, cached-execution), state, subscriber, RPC, and pool deadline expiry. - [crates/execution/flashblocks-node/tests/README.md](crates/execution/flashblocks-node/tests/README.md): Explains why flashblocks-node integration tests live in tests/ (public API only, single compilation unit with test utils) versus private-internals unit tests in src/. Read before adding tests. - [crates/execution/flashblocks/README.md](crates/execution/flashblocks/README.md): Flashblocks pending-state management: StateProcessor, subscriber, reconciler, the staleness invariant and depth limit, canonical reconciliation, and pending-aware RPC methods and subscriptions. - [crates/execution/metering/README.md](crates/execution/metering/README.md): Metering RPC: base_meterBundle, base_meterBlockByHash and base_meterBlockByNumber, plus the pseudo-opcode vocabulary for meteredOpcodes filters. Use to profile tx or block execution timing and gas. @@ -131,9 +131,9 @@ - [crates/execution/trie/src/db/schema.md](crates/execution/trie/src/db/schema.md): MDBX proof-history DB schema: versioned DupSort tables, tombstones, BlockChangeSet reverse index, proof window keys, cursor reads, append-only writes, block 0 backfill. For eth_getProof storage. - [crates/execution/tx-forwarding/README.md](crates/execution/tx-forwarding/README.md): base-tx-forwarding: forwards mempool txs to builder RPCs via per-destination queues fed by FIFO and tip-priority lanes; dedup, batching, resend; spawn_requests for custom ForwardRequest; CLI flags. - [crates/execution/txpool-rpc/README.md](crates/execution/txpool-rpc/README.md): Txpool RPC admin/status APIs and validity submissions; shared staged signature modes, signature rejection metrics, predicate forwarding and sequencer proxy behavior. -- [crates/execution/txpool-tracing/README.md](crates/execution/txpool-tracing/README.md): Txpool tracing extension: tracks mempool lifecycle events, txpool_transactionStatus RPC, enabling CLI flags, and reth_transaction_tracing_tx_event histogram. Use to debug tx latency or drops. +- [crates/execution/txpool-tracing/README.md](crates/execution/txpool-tracing/README.md): Txpool tracing extension: tracks mempool lifecycle events, txpool_transactionStatus RPC, enabling CLI flags, tx_event histogram and flow-labeled inclusion metrics. Use to debug tx latency or drops. - [crates/execution/txpool/PROPAGATION_PIPELINE.md](crates/execution/txpool/PROPAGATION_PIPELINE.md): Plan to forward mempool txs to builders via base_insertValidatedTransaction in 4 PRs (consumer, forwarder, builder endpoint, node integration). Only PR 1 is marked done. Names a crates/txpool/ path. -- [crates/execution/txpool/README.md](crates/execution/txpool/README.md): Base txpool validation/ordering, builder wire extensions, staged EIP-712 predicate authorization, wallet signing contract, rejection metrics, and rollout/security limits. +- [crates/execution/txpool/README.md](crates/execution/txpool/README.md): Base txpool validation/ordering, the separate validity pool, builder wire extensions, staged EIP-712 predicate authorization, wallet signing contract, rejection metrics, and rollout/security limits. ## Crates: infra @@ -190,7 +190,7 @@ - [crates/utilities/l1-head/README.md](crates/utilities/l1-head/README.md): One-line crate description: shared L1 head calculation helpers for proof services. No usage details in the README. - [crates/utilities/metrics/README.md](crates/utilities/metrics/README.md): Metrics helpers: define_metrics! for standardized Prometheus registration (optional custom struct), timed! for durations, inflight! for in-flight tracking. Use when instrumenting crates. - [crates/utilities/optimism-rpc/README.md](crates/utilities/optimism-rpc/README.md): One-line crate description: typed Optimism rollup JSON-RPC provider extensions and response types. No usage details in the README. -- [crates/utilities/reth-cli/README.md](crates/utilities/reth-cli/README.md): Reth-specific CLI utilities for Base EL binaries: init_reth! for version metadata, and the snapshot manifest archive generator shared by snapshot-manifest and the snapshotter sidecar. +- [crates/utilities/reth-cli/README.md](crates/utilities/reth-cli/README.md): Reth CLI utilities for Base EL binaries: init_reth! (version metadata), init_engine! (200ms-block persistence defaults), and the shared snapshot manifest archive generator. - [crates/utilities/retry/README.md](crates/utilities/retry/README.md): Shared RetryConfig wrapper over backon exponential retry builders, giving services common delay normalization, jitter, and bounded or unbounded retry limits. Use when adding retry logic. - [crates/utilities/ring-buffer/README.md](crates/utilities/ring-buffer/README.md): Generic fixed-capacity ring buffer (VecDeque of (I, V) entries) for position-keyed replay; the oldest entry is evicted on push when full. - [crates/utilities/runtime/README.md](crates/utilities/runtime/README.md): Async runtime abstraction (Clock, Spawner, Cancellation, Runtime traits) with TokioRuntime for production and a seeded deterministic executor with virtual time for reproducible tests.