[andrijapanicsb]

Description

Fixes #14522.

The problem. The orphaned-network cleanup added in #12409 runs every 10 seconds. It moves a network from Implementing to Shutdown when the network is older than job.expire.minutes and no pending job of type Network refers to it. But when a VM start implements a network, that work runs inside the VM's own job. So a network that is being implemented right now looks orphaned, and the cleanup shuts it down under the implementing thread. The start then fails with "Network is not yet fully implemented". The start and deploy APIs retry once, which usually hides it. Instance boot group starts (#13907, on main) start only once, so they fail. HA restarts also start only once (not tested here).

The fix.

1. AsyncJobManagerImpl.cleanupOrphanedNetworks(): the thread that implements a network holds the network's lock the whole time. The cleanup now tries that lock once, without waiting. If the lock is held, the network is skipped. If it is free, the network is re-read under the lock and moved to Shutdown only if it is still Implementing (and the existing pending-job check still passes). A network that is really stuck, for example after a management server crash, is still cleaned up as before.
2. Merovingian2.acquire() (framework/db): with a timeout of 0, today's wait loop never runs, so the lock is never tried and a timeout WARN is logged. Now a timeout of 0 makes exactly one attempt and returns at once. Positive timeouts are unchanged. No current caller passes 0; I checked all call sites (only an administrator setting network.lock.timeout, storage.pool.max.waitseconds or vm.job.lock.timeout to 0 could, and today that makes every lock attempt fail).

No schema change and no new setting.

How to reproduce

The full description, a log trace and the impact are in #14522. The A/B results on an unpatched build are in this comment.

A quick way on a test system:

1. Create an isolated network on a non-persistent offering, and make it look older than job.expire.minutes (default one day). On a test database: UPDATE networks SET created = DATE_SUB(NOW(), INTERVAL 2 DAY) WHERE id = <network id>;. Or use any network created more than a day ago.
2. Deploy a VM into it. For a network that is already in use: stop its VMs, wait until the network GC shuts it down (about 10-20 minutes with the defaults), then start a VM.
3. Watch management-server.log. Within about 10 seconds of the start you see Found orphaned network <id> in Implementing state without async job, and the first start attempt fails with "Network is not yet fully implemented". The API's retry usually passes. A boot group start fails.

With this PR, the orphan WARN no longer appears, and the start passes on its first attempt. At DEBUG level the log shows Network <id> in Implementing state is locked (being implemented) or was removed, skipping cleanup.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI
• test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

Unit tests:

• AsyncJobManagerImplTest covers these cases:
  • lock held: skipped;
  • lock free and still Implementing: moved to Shutdown;
  • implemented meanwhile: skipped;
  • pending Network job: skipped;
  • young network: the lock is not tried.
• Merovingian2AcquireTest: a timeout of 0 makes one attempt; a positive timeout still waits and retries. framework/db excludes com/cloud/utils/db/*Test* from surefire, so run it with -Dtest=Merovingian2AcquireTest.
• Without the fix, 9 of the 11 new tests fail.

Before and after, on a single-host KVM test environment:

• Build: the main nightly 24.0.0-SNAPSHOT.20261027, with the two changed classes swapped in.
• The changed code is identical on 4.20.

Case	Before	After
Deploy a VM into a network older than job.expire.minutes	Cleanup fired 8 s into the implementation; the first attempt failed, the retry passed	Cleanup skipped the network; passed on the first attempt
Instance boot group start (no retry) on such a network	The boot group start failed	Passed
Start a VM after the network GC shut the network down	The first attempt failed, the retry passed	Passed on the first attempt
A really stuck network (Implementing, nobody holds the lock)	Moved to Shutdown within 10 s	Still moved to Shutdown within 10 s

After the fix, no "Timed out on acquiring lock" line was logged. The only orphan line was the one for the really stuck network.

Thank you @Pearl1594 for #12409. This keeps its cleanup as it is and only narrows what it treats as orphaned.

[codecov]

Codecov Report

❌ Patch coverage is 80.55556% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 16.44%. Comparing base (e742aad) to head (2903a04).
⚠️ Report is 1 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...src/main/java/com/cloud/utils/db/Merovingian2.java	0.00%	5 Missing ⚠️
...stack/framework/jobs/impl/AsyncJobManagerImpl.java	93.54%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.20   #14527      +/-   ##
============================================
+ Coverage     16.43%   16.44%   +0.01%     
- Complexity    13676    13685       +9     
============================================
  Files          5669     5669              
  Lines        501631   501651      +20     
  Branches      60939    60944       +5     
============================================
+ Hits          82420    82475      +55     
+ Misses       409975   409940      -35     
  Partials       9236     9236              

Flag	Coverage Δ
uitests	4.16% <ø> (ø)
unittests	17.30% <80.55%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Pearl1594]

@vladimirpetrov @sureshanaparti we may need to consider this for 4.22.2, as the bug which this PR fixes would exist there are it was forward-merged..

[Pearl1594]

code lgtm

[Pearl1594]

@blueorangutan package

[blueorangutan]

@Pearl1594 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19836

[andrijapanicsb]

This bug comes, if not mistaken, from 4.20 branch , and is also forward-merged to MAIN, just so that you have the info.

Thx

[Pearl1594]

@blueorangutan test

[blueorangutan]

@Pearl1594 a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests