Skip to content

[#2275] Make MemoryUsage usage tracking lock-free to remove broker-wide lock contention - #2671

Closed
mwashburn2 wants to merge 1 commit into
apache:mainfrom
mwashburn2:memory-usage-lock-free
Closed

mwashburn2 wants to merge 1 commit into
apache:mainfrom
mwashburn2:memory-usage-lock-free

Conversation

@mwashburn2

@mwashburn2 mwashburn2 commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem

SystemUsage.getMemoryUsage() is one MemoryUsage instance shared by every destination and leads to contention on multi-core machines.

Production evidence (5.18.x, 64 cores, ~600 transport threads, mostly non-persistent topics)

Five thread dumps were taken about 30s apart. The global MemoryUsage RRWL was the dominant contention point in every one:

Dump Threads parked on global usage RRWL Threads parked on queue cursor RRWLs (holders blocked on the global lock)
1 49 15
2 34 2
3 26 45
4 60 26
5 58 8

Breakdown from one dump, by operation on the lock:

Operation Waiters
isFull ← Topic.send 38
isFull ← Queue.send 9
decreaseUsage ← decrementReferenceCount 7
increaseUsage ← incrementReferenceCount 3
getPercentUsage ← isParentFull 3

No thread was ever RUNNABLE inside setPercentUsage or fireEvent. The cost is purely lock hand-off, not work done under the lock.

The memory limit was never close to being hit, so nobody was actually waiting for space.

Change

  • MemoryUsage.usage is an AtomicLong. increaseUsage / decreaseUsage / getUsage no longer lock.
  • Usage.percentUsage, limiter, percentUsageMinDelta, usagePortion and DefaultUsageCapacity.limit are volatile. getPercentUsage(), getLimit(), getUsagePortion(), getPercentUsageMinDelta() and MemoryUsage.isFull() / isFull(int) are lock-free.
  • percentUsage is still written only under the write lock, and only when the computed value actually changes. setPercentUsage → fireEvent and the waitForSpace condition signalling therefore keep their existing semantics. Most inc/dec calls do not move the percentage, so the lock is rarely taken.
  • waitForSpace() / waitForSpace(long) return immediately when not full. They only lock and await when full, rechecking under the lock.
  • percentUsage is recomputed on setUsage, onLimitChange and setPercentUsageMinDelta, so it cannot go stale.
  • StoreUsage, TempUsage, JobSchedulerUsage and PercentLimitUsage are unchanged. They still use the locked base paths.

Tests

  • New MemoryUsageLockFreeTest (9 tests) covers:
    • concurrent inc/dec exactness
    • percent consistency after concurrent updates
    • listener events on threshold crossing
    • waitForSpace blocking and releasing, with and without timeout
    • parent propagation
    • limit and min-delta changes
    • setUsage

…convoy

The broker's SystemUsage MemoryUsage is shared by every destination, and
every message reference increment/decrement took its ReentrantReadWriteLock
write lock, while isFull()/getPercentUsage() on the dispatch and send paths
took the read lock. On many-core brokers this produces a lock convoy that
caps aggregate enqueue throughput.

- Track usage in an AtomicLong; increase/decrease are lock-free.
- Make percentUsage, limit, percentUsageMinDelta, usagePortion and limiter
  volatile so getters and isFull() are lock-free.
- Only take the write lock when the computed percent actually changes, so
  listener callbacks and waitForSpace signalling keep their semantics.
- waitForSpace() returns immediately when not full and only blocks under
  the lock when full.
- Recompute percent on setUsage, limit change and min-delta change.
@mwashburn2 mwashburn2 changed the title Make MemoryUsage usage tracking lock-free to remove broker-wide lock contention [#2275] Make MemoryUsage usage tracking lock-free to remove broker-wide lock contention Oct 15, 2026
@mwashburn2

Copy link
Copy Markdown
Contributor Author

Closing in favour of #2275's maintainer PR #2276 (#2276), which takes the same AtomicLong approach plus percent-bucket bounds. We'll evaluate #2276 instead (including a backport to our 5.18.x build, where we've seen this MemoryUsage lock convoy in production). Thanks!

@mwashburn2 mwashburn2 closed this Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant