Skip to content

fix(scheduler): initialize RabbitMQ IO thread before shutdown - #2532

Open
linsheng-li wants to merge 1 commit into
MemTensor:mainfrom
linsheng-li:fix/rabbitmq-shutdown-thread-init
Open

linsheng-li wants to merge 1 commit into
MemTensor:mainfrom
linsheng-li:fix/rabbitmq-shutdown-thread-init

Conversation

@linsheng-li

Copy link
Copy Markdown

Description

Stopping a scheduler that has never initialized RabbitMQ logs AttributeError: 'OptimizedScheduler' object has no attribute '_io_loop_thread' during API shutdown. This includes local-queue deployments and initialization skipped in CI. No restart is required to trigger it.

The constructor initializes _rabbitmq_io_loop_thread, but both initialize_rabbitmq() and rabbitmq_close() use _io_loop_thread. Initialize the same field in the constructor. The production change is one line; connection cleanup, thread joins, shutdown ordering, and exception handling are unchanged. No new dependencies or configuration are required.

Related issue: N/A; the reproduction and regression tests are included here.

Type of change

  • Bug fix (non-breaking)

How Has This Been Tested?

  • Unit tests: pytest tests/mem_scheduler/test_rabbitmq_shutdown.py -q — 5 passed. Without the fix, 3 failed, 2 passed. Covers uninitialized/repeated close, CI initialization skip, connection cleanup with a real IO thread, native optimized-scheduler lifecycle cleanup, and propagation of unexpected join errors.
  • Test steps: isolated API shutdown with RabbitMQ uninitialized, followed by restart and a second shutdown — both exits were clean; graph/vector/business snapshots were unchanged. No generation or embedding requests were made.
  • make format — all Ruff checks passed; 631 files unchanged.
  • Pre-commit checks on both changed files — passed.

Additional directory run: pytest tests/mem_scheduler/ -q — 70 passed, 1 skipped, 1 failed. The existing test_dynamic_cache_layers_access requires PyTorch, which is absent in the test image; the same failure was reproduced against the unmodified source. Pytest also warns that the image lacks the asyncio plugin. The full repository suite was not run.

Checklist

  • Self-reviewed the one-line fix and focused tests.
  • Added tests demonstrating the failure and preserving existing cleanup behavior.
  • No documentation/API/schema changes are needed for this internal field-name correction.

@Memtensor-AI Memtensor-AI added area:scheduler 调度模块 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Oct 21, 2026
@Memtensor-AI
Memtensor-AI requested a review from wustzdy October 21, 2026 15:05
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2532
Task: c435fbdfe59dd7b0
Base: main
Head: fix/rabbitmq-shutdown-thread-init

✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s).

Generated by cloud-assistant via Open Code Review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:scheduler 调度模块 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants