Skip to content

Seed ensemble model sampling per iteration in worker processes - #559

Merged
codelion merged 1 commit into
algorithmicsuperintelligence:mainfrom
fixfon:fix/554-ensemble-seed-per-iteration
Oct 29, 2026
Merged

codelion merged 1 commit into
algorithmicsuperintelligence:mainfrom
fixfon:fix/554-ensemble-seed-per-iteration

Conversation

@fixfon

@fixfon fixfon commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Summary

When random_seed is set (it defaults to 42), every worker process samples the same sequence of ensemble models. The controller writes one derived llm_seed into every llm.models[*].random_seed, each worker builds its own LLMEnsemble in _lazy_init_worker_components, and LLMEnsemble.__init__ seeds its sampler once from models_cfg[0].random_seed. A pick therefore depends on how many picks that worker has already made, not on the iteration: workers running even and odd iterations replay identical picks, and a recycled worker (max_tasks_per_child) restarts the same stream.

This adds LLMEnsemble.reseed(iteration), which reseeds the sampler from the configured seed and the iteration number, and calls it at the start of _run_iteration_worker. A pick now depends on the iteration alone, so it is the same whichever worker runs it, and seeded runs stay reproducible. When no seed is configured nothing changes. The str seed goes through random.seed's SHA-512 path, so it does not depend on PYTHONHASHSEED. The evaluator ensemble only calls generate_all_with_context, which never samples, so it is left as is.

Tests

New tests/test_ensemble_worker_seed.py runs the real _serialize_config, _worker_init and _run_iteration_worker with the real LLMEnsemble, stubbing only OpenAILLM.generate_with_context to record the sampled model:

  • workers running different iterations do not replay the same picks
  • recycled workers (one iteration each) do not all pick the same model
  • the same iteration picks the same model on any worker, in any order
  • unseeded runs stay random

Three of the four fail without the fix. python -m unittest discover tests: 916 tests OK (2 skipped). black --check is clean.

Fixes #554

With random_seed set, every worker process built its LLMEnsemble from the same model seed, so each worker replayed the same sequence of model picks whatever iterations it ran, and a recycled worker restarted it. Reseed the worker ensemble at the start of each iteration from the configured seed and the iteration number, so a pick depends on the iteration alone. Unseeded runs are unchanged.

Fixes algorithmicsuperintelligence#554
@codelion
codelion merged commit e2bdddd into algorithmicsuperintelligence:main Oct 29, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ensemble model selection is identical across worker processes when random_seed is set

2 participants