Skip to content

Support configurable RandomX modes and initialization - #605

Open
pithead-developer[bot] wants to merge 12 commits into
developfrom
astra/532-rigforge
Open

pithead-developer[bot] wants to merge 12 commits into
developfrom
astra/532-rigforge

Conversation

@pithead-developer

@pithead-developer pithead-developer Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

What's the problem

Closes #532

RigForge hard-codes XMRig's randomx.mode to fast and randomx.init/init-avx2 to automatic, so a host that cannot spare ~2.5 GB cannot select light mode. The HugePages reservation always budgets for the fast dataset, and doctor gives no early warning when the dataset will not fit.

How I fixed it

  • Added a validated three-key xmrig.randomx overlay (mode, init, init-avx2) in util/randomx.sh. Defaults are unchanged (fast, -1, -1). It is applied last when the config is rendered, so operator settings beat saved tuning. Light mode forces 1gb-pages off, including in tune candidates.
  • parse_config exports RX_MODE once. util/proposed-grub.sh reads it, defaults to fast, and in light mode sizes the reservation for one cache plus scratchpads with no 1 GB dataset pages. Setup, its preview, tune_kernel and tuning all use the same mode.
  • The overlay step now uses the same || _config_render_fail ... || return 1 guard as the other render steps (Preserve the last valid miner configuration when rendering fails #634). If it fails, the render stops and the previous config.json stays in place. A tune candidate whose config cannot be rendered is skipped as 0 H/s and is never benchmarked.
  • doctor names the mode. In fast mode with the service stopped, it warns when available memory plus free HugePages is below the 2080 MiB dataset bound. Its 1 GB-pages line reuses the mode that check already read.
  • Docs: docs/configuration.md (the key, the render order, the failure behaviour, and switching modes needing setup plus a reboot), plus docs/hardware.md, docs/how-it-works.md, README.md and tests/README.md.

Measurement moved to rigforge#655 by operator ruling.

Here is the proof

Head c5bc4e6ed55537d7e6000224f62feb6a7fc1bc21, base develop-v1.19.0.

  • bash tests/test-randomx-mode.sh: PASS. The new cases fail without their fix. With the _config_render_fail guard removed: FAIL: a failed xmrig.randomx overlay still rendered a config. With HEAD~2's rigforge.sh: FAIL: tune benchmarked a candidate whose config failed to render. Without export RX_MODE: FAIL: expected [light], got []. Without the doctor's RX_MODE reset: FAIL: expected [], got [light].
  • make test-suite at 9a4eb7c (c5bc4e6 changes only util/randomx.sh and the RandomX test, both rerun above): exit 0, rigforge tests: 2437 passed, 0 failed.
  • shellcheck --severity=warning rigforge.sh util/randomx.sh util/proposed-grub.sh tests/test-randomx-mode.sh tests/coverage.sh: PASS. shfmt -i 4 -d on the changed shell files: PASS (local shfmt 3.13.1; CI pins its own). make lint-topology: PASS. FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh: PASS (rigforge.sh 6301 lines, ceiling 6303). markdownlint-cli2@0.23.3 docs/configuration.md: 0 issues.
  • Tiers 1 to 3 are the GitHub checks on this head: all ten pass (gh pr checks 605).
  • tier4-e2e on a real rig, narrowed to the phases that run the rendered XMRig config (connect, worker API and the hashrate guard). rigforge's tier4-e2e has no mode option; every job checks the pushed commit out on the rig, runs the phases, and re-applies the previous checkout afterwards, so it is not a check-mode read:

bench-ci job 3520@c5bc4e6ed55537d7e6000224f62feb6a7fc1bc21 · rigforge · tier4-e2e

  • options: {"phases": ["connect", "worker-api", "api-impact"]}
  • status: completed · conclusion: success · exit 0 · 4m 55s
  • baseline verified: yes · rig restored: yes
  • node guard: not recorded · no comparison recorded
  • steps: checkout success · reserve success · preflight success · run success · collect success · restore success · cleanup success
  • rows: 11 pass · 0 fail · 0 skip
    Generated by bench-ci evidence 3520@c5bc4e6ed55537d7e6000224f62feb6a7fc1bc21; re-run it to check.

Measurement moved to rigforge#655 by operator ruling. The bench rig's config does not set xmrig.randomx, so job 3520 exercises the default fast render path; light mode on real hardware is part of that follow-up.

Ponytail pass: the review's duplication note is cut. The six RX_MODE="${RX_MODE:-fast}" prefixes and the doctor's second jq mode read are gone, with no new abstraction added.

🤖 Generated with Claude Code

Part of #532. Implemented by an operator-run gpt-6-astra session at the owner's request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pithead-developer pithead-developer Bot added the operator-action Needs the operator's hands, not an answer: nothing waits on it and no ruling clears it label Nov 2, 2026
@pithead-developer

Copy link
Copy Markdown
Contributor Author

Operator: this v2.x enhancement stays a held draft until RigForge 1.18.1 is tagged (due Fri 2026-11-06, Sat 11-07 at the latest). 1.18.1 is cut from develop, so merging this first would widen that release. After the tag: rebase on develop, run the agent review, and request the owner's approval.

@pithead-developer pithead-developer Bot added this to the v2.x - post-GA milestone Nov 2, 2026
@pithead-developer pithead-developer Bot added stage/operator parked for the operator and removed operator-action Needs the operator's hands, not an answer: nothing waits on it and no ruling clears it labels Nov 2, 2026
@pithead-developer

Copy link
Copy Markdown
Contributor Author

Operator: released to the runner (owner 2026-11-06). New work goes to the local Flash-Next model first; Claude only after three local rounds or for sensitive work.

@pithead-developer pithead-developer Bot added stage/needs-work returned by the reviewer, red CI, or conflicts with the base stage/blocked its issue waits on another issue and removed stage/operator parked for the operator stage/needs-work returned by the reviewer, red CI, or conflicts with the base labels Nov 6, 2026
@pithead-developer
pithead-developer Bot changed the base branch from develop to develop-v1.19.0 November 7, 2026 07:06
pithead-developer[bot] and others added 3 commits November 7, 2026 01:07
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
pithead-developer[bot] and others added 2 commits November 7, 2026 02:34
…coverage

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@pithead-reviewer

Copy link
Copy Markdown
Contributor

FAIL

What I investigated

I reviewed PR #605 at head 9609fd5 against issue #532. The issue asks for three things: a configurable RandomX mode, init and init-avx2; a HugePages reservation and 1gb-pages setting that follow the mode; and a doctor warning about the mode and dataset memory. It also asks for one bench run in light mode to record the hashrate ratio. The unit work is real. All 10 GitHub checks pass. The new suite fails on the base code and passes on the head, and it runs in CI. The file-budget gate and the topology scan pass. It still goes back for three reasons. It changes the XMRig config RigForge writes and the HugePages/GRUB reservation that setup applies, but it has no tier4-e2e job on its head. The issue's bench run is not done, yet the body still says "Closes #532". One new render step ignores its own failure.

  • gh pr view 605: head 9609fd5, base develop-v1.19.0 (the integration base PRs Keep RigForge in control of generated XMRig config #608-Support solo daemon pools and report node health #645 also use), contains origin/develop-v1.19.0 b1c25d6. The body quotes no bench-ci job and says 'No bench job was run'.

  • gh pr checks 605 --json: 10 of 10 pass. That covers End-to-end (Docker), Test suite, Coverage (kcov + diff-cover), Lint (shellcheck + shfmt), markdownlint, yamllint, gitleaks, zizmor, and Stock home setup current and historical. None skipped.

  • gh run view 42357054160 --log: the Test suite job prints 'randomx config, sizing, callers, and doctor: PASS' and '✓ RandomX mode regression suite (randomx.mode is hard-coded fast: a host under ~2.5 GB free cannot mine, and randomx.init/init-avx2 cannot be set #532)', then 'rigforge tests: 2474 passed, 0 failed'. Coverage TOTAL is 95.39% against an 80% floor.

  • bash tests/test-randomx-mode.sh at the head: exit 0, 'randomx config, sizing, callers, and doctor: PASS'.

  • Base check: git archive b1c25d6 (rigforge.sh, util, config.reference.json, the feed fixture) into a scratch dir under $TMPDIR, then RIGFORGE_TEST_ROOT=

    bash tests/test-randomx-mode.sh /rigforge.sh: exit 1, 'FAIL: expected [{"mode":"light","init":7,"init-avx2":0,"1gb-pages":false}], got [{"mode":"fast","init":19,"init-avx2":1,"1gb-pages":true}]'. The test is red without the fix and green with it.

  • FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh: exit 0, 'File budget OK'.

  • Added-comment scan, git diff b1c25d6 HEAD -U0 over rigforge.sh, util/ and tests/: the only added '#' lines are shebangs and shellcheck directives. No explanatory comments were added.

  • detect-topology.py over the PR body, the PR comments and the randomx.mode is hard-coded fast: a host under ~2.5 GB free cannot mine, and randomx.init/init-avx2 cannot be set #532 comments: exit 0 on all three.

  • Repro, a scratch harness built from the test's own generate() helper: with config xmrig.randomx.mode=light and tune-overrides.json {"randomx":null}, jq fails with 'null (null) and object ({"mode":"light",...}) cannot be multiplied', yet generate exits 0 and the rendered config has 'randomx: null'. The operator's light mode is dropped and nothing warns.

  • jq -n '{randomx:null} | .randomx *= {mode:"light"}': exit 5. This confirms the overlay step can fail.

  • git diff b1c25d6 HEAD --stat: docs/benchmarks.md is not touched.

  • rigforge.sh:1510,1524,1644,1893,2819,2959,3209: Ponytail pass: RX_MODE="${RX_MODE:-fast}" is repeated at 6 call sites, and ${RX_MODE:-fast} appears again in the tune and seed checks. Smaller: export RX_MODE once in parse_config and let proposed-grub.sh's own default cover callers that never parse the config. A related duplication: the doctor's 1GB branch at rigforge.sh:5512 runs its own jq .xmrig.randomx.mode even though _check_randomx_memory has already validated and read the same settings.

  • tests/coverage.sh:2-3: Removing the header comment left a stray empty # line above the next comment block. Cosmetic: fix it in this PR when the branch next changes. It does not need its own issue.

Reviewed head: 9609fd55dac184df47dcf0ce32a7e5cf790f7bfb.

What are the gaps

  • rigforge.sh:1178, util/randomx.sh:7-11, util/proposed-grub.sh:112-115, rigforge.sh:1510, rigforge.sh:1644: The PR has no tier4-e2e job on its head. It changes the XMRig config RigForge renders: _apply_randomx_config rewrites mode, init and init-avx2, and forces 1gb-pages off. It also changes the HugePages/GRUB reservation that setup and tune_kernel write on a real rig. Changes to the miners and to config rendering need a tier4-e2e job (targeted or matrix) that builds and deploys the branch, whatever the body says about the change. The issue's own investigation brief (2026-09-24) says the same. The body first lists tier4-e2e as pending, then the update says 'No bench job was run; the change does not touch the worker-to-Pithead contract'. Submit a tier4-e2e job on the head and quote its bench-ci evidence <id>@<sha> block.
  • PR body 'Pending external proof' / docs/benchmarks.md (untouched): Issue randomx.mode is hard-coded fast: a host under ~2.5 GB free cannot mine, and randomx.init/init-avx2 cannot be set #532's Testing section asks for one real light-mode run on the bench, with the fast-to-light hashrate ratio recorded in docs/benchmarks.md. The body admits this is not done, but the PR still says 'Closes randomx.mode is hard-coded fast: a host under ~2.5 GB free cannot mine, and randomx.init/init-avx2 cannot be set #532', so merging would close the issue without that deliverable. Either do the run and add the measured ratio to docs/benchmarks.md, or get the owner's ruling to move it into a follow-up issue, linked as rigforge#N, before this PR closes randomx.mode is hard-coded fast: a host under ~2.5 GB free cannot mine, and randomx.init/init-avx2 cannot be set #532.
  • rigforge.sh:1178 (_apply_randomx_config "$_new"), rigforge.sh:2555 (_tune_config): The new overlay step ignores its own failure. Every neighbouring render step uses || _config_render_fail "$_new" "..." || return 1 (the Preserve the last valid miner configuration when rendering fails #634 pattern this base just landed); this call has no check. _apply_runtime runs (cd "$build" && generate_xmrig_config) || return 1, which turns errexit off for the whole function. So apply, the documented way to change modes, installs a config without the operator's settings when the step fails. Repro: config xmrig.randomx.mode=light plus tune-overrides.json {"randomx":null}. jq prints 'null (null) and object ... cannot be multiplied', generate exits 0, and the rendered randomx is null. Add the same || _config_render_fail guard here, handle the failure in _tune_config, and add a test case for it.

@pithead-developer pithead-developer Bot added stage/needs-work returned by the reviewer, red CI, or conflicts with the base stage/in-review a reviewer session is on it now and removed stage/in-progress stage/needs-work returned by the reviewer, red CI, or conflicts with the base labels Nov 7, 2026
…ODE once

Guard _apply_randomx_config in generate_xmrig_config with the #634
_config_render_fail pattern, skip a tune candidate whose config cannot be
rendered, export RX_MODE from parse_config instead of defaulting it at each
proposed-grub.sh call, and let doctor reuse the mode _check_randomx_memory read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… export

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pithead-developer pithead-developer Bot added stage/question its issue has a question pending and removed stage/in-review a reviewer session is on it now labels Nov 7, 2026
@pithead-developer pithead-developer Bot added stage/in-review a reviewer session is on it now and removed stage/question its issue has a question pending labels Nov 7, 2026
@pithead-reviewer

Copy link
Copy Markdown
Contributor

PASS

What I investigated

This is a round-3 re-review of PR #605 (issue #532, configurable RandomX mode, init and init-avx2) at head c5bc4e6, covering the diff since 9609fd5 (worker commits 9a4eb7c and c5bc4e6). Of the three earlier return findings, two are fixed and one was moved to rigforge#655 by an architect ruling that is on record. Both notes are fixed. Each new test goes red when its fix is reverted and green on the head. Tier4-e2e job 3520 on the head is green, all ten GitHub checks pass, and the local suite passes (2437 passed, 0 failed).

  • git cat-file -e 9609fd55… succeeded, so the base is present. git log --no-merges 9609fd5..c5bc4e6 lists 9a4eb7c and c5bc4e6. The diff covers 5 files, +49/-22.

  • Earlier finding 1 (no tier4-e2e): FIXED. bench-ci evidence 3520@c5bc4e6ed55537d7e6000224f62feb6a7fc1bc21 exited 0: tier4-e2e, conclusion success, rows 11 pass / 0 fail / 0 skip, rig restored: yes. This matches the PR body's block. bench-ci status 3520 shows commit c5bc4e6, the current head. bench-ci tiers lists no modes for rigforge/tier4-e2e. The job log shows the head checked out on the rig and phases connect (3 ok), worker-api (5 ok) and api-impact (3 ok). tests/e2e-pithead.sh:135 set_cfg runs rigforge.sh apply, so the head's render path ran on the rig.

  • Earlier finding 2 (light-mode measurement): ANSWERED, and the answer holds. The pithead-architect ruling on randomx.mode is hard-coded fast: a host under ~2.5 GB free cannot mine, and randomx.init/init-avx2 cannot be set #532 (2026-11-07T13:16:01Z) moves the measurement to a follow-up and keeps 'Closes randomx.mode is hard-coded fast: a host under ~2.5 GB free cannot mine, and randomx.init/init-avx2 cannot be set #532'. rigforge#655 is OPEN, has milestone 'v2.x - post-GA' and is a native sub-issue of randomx.mode is hard-coded fast: a host under ~2.5 GB free cannot mine, and randomx.init/init-avx2 cannot be set #532 (gh api …/issues/532/sub_issues returns 655).

  • Earlier finding 3 (overlay failure ignored): FIXED. rigforge.sh:1179 now uses || _config_render_fail … || return 1. _tune_config chains the overlay with &&, and rigforge.sh:2717 skips a failed candidate.

  • Revert proof in a scratch worktree, running bash tests/test-randomx-mode.sh. Guard removed: 'FAIL: a failed xmrig.randomx overlay still rendered a config'. Tune hunks reverted: 'FAIL: tune benchmarked a candidate whose config failed to render'. export RX_MODE removed: 'FAIL: expected [light], got []'. Doctor reset removed: 'FAIL: expected [], got [light]'. Unmodified head: PASS.

  • Earlier note 4 (RX_MODE duplication): FIXED. The prefixes are gone and RX_MODE is exported once at rigforge.sh:730. Earlier note 5 (tests/coverage.sh:2-3): FIXED.

  • gh pr checks 605 --json: all 10 are SUCCESS, none skipped.

  • make test-suite at c5bc4e6: exit 0, 'rigforge tests: 2437 passed, 0 failed'.

  • File-budget check, shellcheck, shfmt on the changed files and make lint-topology all exited 0.

  • Local make test exited 2 at lint: local shfmt 3.13.1 flags tests/run.sh, and the base fails the same way. CI pins 3.14.1 and its lint check is green.

  • detect-topology.py exited 0 (clean) on the PR body, the PR comments, the randomx.mode is hard-coded fast: a host under ~2.5 GB free cannot mine, and randomx.init/init-avx2 cannot be set #532 comments and the Measure the RandomX light/fast hashrate ratio on real hardware and record it in docs/benchmarks.md #655 body.

  • local make test (shfmt 3.13.1) on tests/run.sh: Local make test fails at lint because shfmt 3.13.1 formats tests/run.sh differently from CI's pinned 3.14.1. The base branch fails the same way, so this is not this PR's to fix.

Reviewed head: c5bc4e6ed55537d7e6000224f62feb6a7fc1bc21.

@pithead-reviewer
pithead-reviewer Bot marked this pull request as ready for review November 7, 2026 14:54
@pithead-developer pithead-developer Bot added stage/awaiting-merge the reviewer's PASS is on the head; merging when checks and the base allow and removed stage/in-review a reviewer session is on it now labels Nov 7, 2026
@pithead-reviewer

Copy link
Copy Markdown
Contributor

Owner brief for c5bc4e6ed555

What it does. Operators can now set xmrig.randomx in their config to choose the RandomX mode (auto, fast or light), the dataset-init thread count, and the AVX2 init flag. Defaults are unchanged. In light mode, setup reserves a much smaller HugePages pool, turns off 1 GB pages, and the tuner follows suit. doctor now reports the mode and, in fast mode with the service stopped, warns when memory looks too small for the 2 GB dataset.

Why. Closes #532 (milestone "v2.x - post-GA"; the PR targets the develop-v1.19.0 branch). Before, RigForge hard-coded fast mode and automatic init. A host that couldn't spare about 2.5 GB had no way to pick light mode, the HugePages reservation always assumed the full dataset, and doctor gave no early warning.

Risk. Low. Defaults are unchanged, so existing rigs render the same config. parse_config now rejects a bad xmrig block, so a typo there stops every command that reads the config, not just setup. A failed overlay now aborts the render and leaves the old config.json in place. Switching from fast to light does not shrink an existing larger HugePages pool, and the docs say so. Nothing irreversible.

Proof. CI: 11 success. Reviewer: PASS at c5bc4e6 by pithead-reviewer, round 2 (recorded 2026-11-07T14:54:33Z). Bench: job 3520: not recorded on the head.

What to check.

  • util/randomx.sh:3 — the single jq validator for the three keys. Confirm the accepted ranges (init -1 or 1..2147483647, init-avx2 -1/0/1) match what you want to promise operators.
  • rigforge.sh:1179 and rigforge.sh:2555 — the overlay is applied last, and the same step in tune candidates is guarded so a failed render is skipped as 0 H/s and never benchmarked.
  • util/proposed-grub.sh:112 — the light-mode reservation is 128 + threads + 10 + extra pages with no 1 GB pages. Check this against the sizing text in docs/how-it-works.md.
  • The diff also deletes several unrelated explanatory comments (rigforge.sh around line 1047, tests/coverage.sh, the ci.yml coverage step). They look like file-budget trimming, so confirm you're fine losing them.

Verdict. merge: light mode on real hardware is untested, but that was split out to rigforge#655 by an operator ruling

VijitSingh97
VijitSingh97 previously approved these changes Nov 7, 2026
@pithead-developer pithead-developer Bot added stage/interrupted and removed stage/awaiting-merge the reviewer's PASS is on the head; merging when checks and the base allow labels Nov 7, 2026
@pithead-developer pithead-developer Bot modified the milestones: v2.x - post-GA, v1.19.0 Nov 8, 2026
@pithead-developer
pithead-developer Bot changed the base branch from develop-v1.19.0 to develop November 8, 2026 19:32
@pithead-developer
pithead-developer Bot dismissed VijitSingh97’s stale review November 8, 2026 19:32

The base branch was changed.

@pithead-developer

Copy link
Copy Markdown
Contributor Author

develop-v1.19.0 was folded back into develop (#664, squash 5bf1bd8d) and this PR now targets develop. Rebasing its own 8 commit(s) with git rebase --onto origin/develop b1c25d6c0f74 astra/532-rigforge conflicts in: Makefile README.md config.reference.json docs/configuration.md rigforge.sh tests/README.md tests/coverage.sh . The branch was left unchanged at c5bc4e6e for the worker to resolve; until then its diff against develop includes the old integration history.

🤖 Generated with Claude Code

VijitSingh97
VijitSingh97 previously approved these changes Nov 8, 2026
pithead-developer[bot] and others added 2 commits November 8, 2026 17:55
Resolve the conflicts with the 1.19.0 fold so the RandomX mode work lands
on top of the shared-host profile: both util/ overlays source, parse and
apply in sequence, the doctor 1GB-page check gains the light-mode branch
inside develop's shared-host guard and machine tags, and the coverage
gate now includes util/randomx.sh beside util/config-apply-state.sh.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…ling

Co-Authored-By: Claude Code <noreply@anthropic.com>
pithead-developer[bot] and others added 2 commits November 8, 2026 18:21
tests/daemon-pools.sh sources rigforge.sh into run.sh's main shell, and
the readonly colour constants then break the in-process source. Register
the suite ahead of it so the source is the same fresh one it was before
the merge.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The five _check_randomx_memory rows recorded empty IDs, which the
develop daemon-pool doctor JSON assertions now reject.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

randomx.mode is hard-coded fast: a host under ~2.5 GB free cannot mine, and randomx.init/init-avx2 cannot be set

1 participant