Skip to content

[fix] Use MS state for disaggregated BE decommission - #523

Merged
morningman merged 2 commits into
apache:masterfrom
Al-assad:fix-ddc-ms-decommission
Oct 20, 2026
Merged

morningman merged 2 commits into
apache:masterfrom
Al-assad:fix-ddc-ms-decommission

Conversation

@Al-assad

@Al-assad Al-assad commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

What problem does this PR solve?

DDC scale-down interprets SystemDecommissioned and TabletNum using a single kernel's semantics. On kernels that publish SystemDecommissioned only after completion, this can repeatedly submit decommission SQL. DROP errors could also be ignored before reducing pod replicas.

Use MS node states as the completion authority and persist a per-target task in a StatefulSet annotation:

  • Adopt manual SQL, including legacy FE-first decommission, without expanding the CR's scale-down range.
  • Persist submission intent before SQL; do not automatically resend an uncertain submission after restart or transport failure.
  • Wait two FE synchronization periods after MS completion, read effective FE configuration, and verify every registered FE, including observers, before DROP.
  • Confirm stable absence in MS and every FE before atomically reducing replicas and clearing the task. Transient MS remove/add updates, identity changes, unknown states, and DROP failures cannot advance scale-down.
  • Resolve legacy instance IDs through MS and bound administrative HTTP/SQL calls. No CRD changes are required.

Release note

Decommission scale-down now waits for MS completion and FE synchronization. MS administrative HTTP access, SQL access to every registered FE, and permission to inspect FE configuration are required. Unreachable FEs, uncertain submissions, and changes to an active replica target pause the task; recovery guidance is included in the scaling documentation.

Check List (For Author)

  • Unit tests, including manual SQL across legacy/current state semantics, restart recovery, partial completion, FE lag, transient removal, identity replacement, failed persistence, uncertain SQL, DROP failure, and HTTP errors.
  • Behavior changed: decommission scale-down uses the persistent MS-driven workflow described above.
  • Documentation: English and Chinese scaling guides updated in this PR.

Validation:

  • go test -race ./pkg/common/utils/metaservice ./pkg/common/utils/mysql ./pkg/controller/sub_controller/disaggregated_cluster/computegroups -count=1 passed.
  • go vet ./... and operator build passed.
  • make manifests generate passed without generated-file changes; git diff --check passed.
  • go test ./... was attempted. The existing FE Test_ClearResources blocked waiting for informer synchronization and was interrupted after approximately four minutes; other reported packages passed. Live SelectDB 3.x/4.x cluster validation has not been performed.

@morningman
morningman merged commit 7c6de21 into apache:master Oct 20, 2026
3 checks passed
morningman pushed a commit that referenced this pull request Oct 20, 2026
…rsistence (#523, #524) (#526)

* [feature](operator) Persist mutable FE BE and MS configuration (#524)

* [feature](operator) Persist mutable FE BE and MS configuration

Preserve mutable runtime configuration on existing safe PVCs while keeping user ConfigMaps unchanged. Record compatibility defaults for existing clusters and provide a verified opt-in migration tool.

* Retain runtime configuration PVCs during compute group scale-down

(cherry picked from commit d4ae3df)

* [fix] Use MS state for disaggregated BE decommission (#523)

* fix: use MS state for disaggregated BE decommission

* Fix MS instance query colon compatibility

(cherry picked from commit 7c6de21)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants