Repository navigation
[#2275] Make MemoryUsage usage tracking lock-free to remove broker-wide lock contention - #2671
Closed
mwashburn2 wants to merge 1 commit into
Closed
mwashburn2 wants to merge 1 commit into
mwashburn2 wants to merge 1 commit into
Conversation
…convoy The broker's SystemUsage MemoryUsage is shared by every destination, and every message reference increment/decrement took its ReentrantReadWriteLock write lock, while isFull()/getPercentUsage() on the dispatch and send paths took the read lock. On many-core brokers this produces a lock convoy that caps aggregate enqueue throughput. - Track usage in an AtomicLong; increase/decrease are lock-free. - Make percentUsage, limit, percentUsageMinDelta, usagePortion and limiter volatile so getters and isFull() are lock-free. - Only take the write lock when the computed percent actually changes, so listener callbacks and waitForSpace signalling keep their semantics. - waitForSpace() returns immediately when not full and only blocks under the lock when full. - Recompute percent on setUsage, limit change and min-delta change.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
SystemUsage.getMemoryUsage()is oneMemoryUsageinstance shared by every destination and leads to contention on multi-core machines.Production evidence (5.18.x, 64 cores, ~600 transport threads, mostly non-persistent topics)
Five thread dumps were taken about 30s apart. The global
MemoryUsageRRWL was the dominant contention point in every one:Breakdown from one dump, by operation on the lock:
isFull←Topic.sendisFull←Queue.senddecreaseUsage←decrementReferenceCountincreaseUsage←incrementReferenceCountgetPercentUsage←isParentFullNo thread was ever RUNNABLE inside
setPercentUsageorfireEvent. The cost is purely lock hand-off, not work done under the lock.The memory limit was never close to being hit, so nobody was actually waiting for space.
Change
MemoryUsage.usageis anAtomicLong.increaseUsage/decreaseUsage/getUsageno longer lock.Usage.percentUsage,limiter,percentUsageMinDelta,usagePortionandDefaultUsageCapacity.limitarevolatile.getPercentUsage(),getLimit(),getUsagePortion(),getPercentUsageMinDelta()andMemoryUsage.isFull()/isFull(int)are lock-free.percentUsageis still written only under the write lock, and only when the computed value actually changes.setPercentUsage→fireEventand thewaitForSpacecondition signalling therefore keep their existing semantics. Most inc/dec calls do not move the percentage, so the lock is rarely taken.waitForSpace()/waitForSpace(long)return immediately when not full. They only lock and await when full, rechecking under the lock.percentUsageis recomputed onsetUsage,onLimitChangeandsetPercentUsageMinDelta, so it cannot go stale.StoreUsage,TempUsage,JobSchedulerUsageandPercentLimitUsageare unchanged. They still use the locked base paths.Tests
MemoryUsageLockFreeTest(9 tests) covers:waitForSpaceblocking and releasing, with and without timeoutsetUsage