Skip to content

pool: count a swap-cached page once in the memory_limit anon - #541

Merged
CMGS merged 1 commit into
mainfrom
pool/anon-swapcached
Nov 9, 2026
Merged

CMGS merged 1 commit into
mainfrom
pool/anon-swapcached

Conversation

@CMGS

@CMGS CMGS commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

A page that faults back in from swap keeps its swap slot. That page is then counted twice: once in AnonPages and once in used swap (SwapTotal - SwapFree). Because readAnonBytes added the two, it over-counted by SwapCached.

Change

  • readAnonBytes now returns AnonPages + SwapTotal - SwapFree - SwapCached.
  • docs/deploy.md: the memory_limit row describes the new formula.
  • TestMemoryLimitCrossings has a new case, "swap-cached anon counts once": 250G anon, 51G swap, 40G swap-cached, limit 300G. The node must stay under the limit and the gauge must read 261G. The old formula fails this case.
  • The test fixtures now take SwapCached as a parameter instead of a fixed 99 kB.

Evidence (hardware, inside a guest with its own swapfile; neither testbed host has swap)

After 4.0 GiB of anon was pushed to swap and faulted back:

Swap device SwapCached readAnonBytes, old formula New formula
8G 3158304 kB 7.03 GiB 4208536 kB, equal to AnonPages
4G 2096920 kB 6.02 GiB

The effect is that a node refuses claims too early, never too late. A production host that has swapped and faulted pages back reads high by up to its used swap.

Gates

  • make go-lint: 10× 0 issues.
  • asl -forwarder=false ./...: clean on both GOOS.
  • go test -race ./pool/: ok.
  • Mutation check: dropping - SwapCached fails the new case.

A page faulted back from swap keeps its swap slot, so it is both in
AnonPages and in SwapTotal - SwapFree. On a guest with 4.0 GiB of anon,
the old sum read 7.03 GiB, the excess equal to SwapCached, so a node that
swapped and faulted back refused claims early.
@CMGS

CMGS commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Hardware verification: pass

Setup. Neither testbed host has swap, so the test ran in a guest with its own 8G swapfile. 4096 MiB of anon was written under memory.max 1024M (S1), then the limit was lifted and every page was re-read (S2). readAnonBytes from 707ef2c (old) and from e9f058a (new) ran back to back on the same /proc/meminfo.

Point SwapCached old new AnonPages
S2, case 1 3201052 kB 7.073 GiB 4.021 GiB 4215616 kB
S2, case 2 3155492 kB 7.030 GiB 4.021 GiB 4215628 kB
S1, case 1 46080 kB 4.021 GiB 3.977 GiB
S1, case 2 592 kB 4.021 GiB 4.020 GiB
  • After swap-in, new minus AnonPages is +254 KiB and +284 KiB. Old reads high by exactly SwapCached, +75%, and that persists while swap slots stay cached.
  • While pages are still swapped out, swap-cache pages that are not mapped (readahead) are counted zero times by the new formula: about 45 MiB here, 1.1% of 4 GiB. That error is transient and bounded by the unmapped swap cache. /proc/meminfo exposes no exact correction (it would need mapped ∩ swap cache), so the formula is kept.

@CMGS

CMGS commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Ground truth, 100 ms sampling

Setup.

  • One guest with an 8G swapfile. 4096 MiB of anon runs in its own cgroup.
  • Truth is the hog's RssAnon + VmSwap from /proc/<pid>/status.
  • Each formula is compared against truth after subtracting a pre-hog baseline.
  • Two cases:
    • A: the hog writes under memory.max 1024M.
    • B: the hog writes with no limit, then memory.max is clamped to 1024M, which pushes resident pages out in a burst.
  • Both cases then lift the limit and fault every page back.

Error against truth, in MiB:

Phase old (main) new (this PR) lru*
A push-out −4.7 … −1.0 −34.5 … −1.9 −4.7 … −1.1
A swapped-out steady −1.1 … −0.1 −34.7 … −33.7 −1.1 … −0.2
A faulted-back steady +3113.9 … +3114.2 −0.9 … −0.6 −0.6 … −0.2
B clamp push-out +0.9 … +2.9 −47.5 … +2.3 +1.0 … +2.9
B faulted-back steady +3231.2 … +3231.5 +3.8 … +4.1 +4.0 … +4.1

* lru = Active(anon) + Inactive(anon) − Shmem + (SwapTotal − SwapFree) − SwapCached

  • This PR undercounts by the unmapped swap cache while pages are swapped out: at most 47.5 MiB on a 4 GiB hog, about 1%, which persisted at 34 MiB in case A. That is the bounded approximation Codex accepted.
  • Main overcounts by everything faulted back, +76–79%, and the overcount persists.
  • The lru form tracks truth in every phase, because unmapped swap-cache pages stay on the anon LRU. It would miss mlocked anon, which was 0 here.
  • This PR keeps the simpler form. Switching to lru is a separate decision.

@CMGS
CMGS merged commit a28458b into main Nov 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant