console: show how a replica's workers share its CPU on the redesigned cluster page

[leedqin]

Motivation

Sixth PR in the cluster details redesign stack, on top of #39607. The CPU chart shows a replica's CPU as a share of its allocation, averaged over its workers. The CPU troubleshooting guide points out that a replica whose work sits on one worker can peg a core while reporting a modest cpu_percent, and that high CPU with evenly spread work points elsewhere: an expensive object, ad-hoc queries, or an undersized cluster. The Console had no way to tell these apart.

This adds a Workers button to the CPU chart, next to #39508's Processes. It opens a drawer that reads the replica's per-worker scheduling counters live and leads with a finding in the guide's terms.

Description

What it reads. The drawer reads only while open.

• Every 5s, it reads each worker's total time in running dataflows. It sums the root operators of mz_scheduling_elapsed_per_worker (list_length(address) = 1), because a root's time includes its children.
• Every 30s, it reads the same counters per dataflow, named through mz_compute_exports. Transient (t) exports pool into one "Ad-hoc queries and subscribes" row.
• Both reads run on the replica being diagnosed, through cluster and cluster_replica on the session.
• They go over one WebSocket session (WebsocketQueryClient), held only while reading. An HTTP read would open a session each time, and a session start costs mz_catalog_server about 35 ms of CPU.
• Guards on the replica's load:
  • statement_timeout of 10s;
  • each read waits at least 10× as long as the last one took;
  • reads pause after 10 minutes and stop when the drawer closes.
• The replica's own CPU comes from the processes drawer's per-process metrics SUBSCRIBE (useReplicaProcessSamples), an index lookup on replica_id over the last 20 minutes. The card's series can't be reused: past a 1h window they are per-bucket peaks, and past 24h they are polled.
• Readings are kept for 30 minutes in per-environment TanStack DB collections, so a reopened drawer continues from them.

What it shows.

• A verdict over the last 5 minutes:

  • a saturated worker: at least 90% of a core and twice the other workers' average;
  • uneven work: the busiest worker at half a core or more and twice the others;
  • light load: uneven, but no worker near its limit, so no warning;
  • every worker busy: under 10% idle;
  • high replica CPU (75% or more), split into time running dataflows and time outside them, with advice that depends on which dominates;
  • idle;
  • balanced.

  The busiest worker is compared with the others' average, not with an average that includes it, which can't reach 2× on two workers. Only a saturated worker outranks high CPU. Uneven work under high CPU becomes a note on that verdict. On a multi-process replica, a skewed worker's verdict also says whether its process used the most CPU over the same window, by the Processes drawer's rule (1.25× the others). Below that, it says the processes were even and the skew is between workers.

  The skew verdicts name the object that took most of the busiest worker's time. They also give a copyable EXPLAIN ANALYZE CPU WITH SKEW for that object, pinned to the replica. Until 30s of readings exist, the drawer shows a first look from the lifetime counters instead.

• Processes and workers link to each other. On a multi-process replica, the Workers drawer links to the Processes drawer and back, opening on the same replica. Both drawers draw the replica's line and its busiest process or worker in the replica's color, and the rest muted.

• Two charts on one clock axis:

  • replica CPU against time running dataflows, as % of the allocation;
  • the busiest worker against the average worker, as % of a core over a trailing 30s.

• The busiest workers. The busiest worker always shows, with what it spent its dataflow time on: its top three objects, each with its share of the worker's time, the time itself, and how that compares with the other workers' average on the same object. When work is skewed, the next four workers follow, each with its top object. The rest are one range.

• Where the CPU went: a UniversalTable with one row per object, measured since the drawer opened or since each dataflow started. Each row has:

  • a bar per worker against a dashed even-share line, colored by that worker's time against the other workers' average;
  • the hottest worker, as a multiple of the other workers' average, the one meaning "×" has across the drawer;
  • its extra time over the other workers' average;
  • its share and its CPU time.

  Since the drawer opened, the share is % of the replica's CPU allocation, the CPU chart's unit. Objects, an "Other dataflows" footer and a line for work outside running dataflows add up to the replica's CPU. Over each dataflow's lifetime there is no common span, so the share is of all dataflow CPU instead. Objects under 1% of the dataflow CPU aren't flagged as skewed. Header tooltips define each column.

Counter semantics. A dataflow's time leaves the counters when it shuts down (handle_shutdown in src/compute/src/logging/timely.rs). So the counters fall when a dataflow is dropped, an ad-hoc query's dataflow finishes, or the replica restarts.

• Only the two ends of a span are compared, so a fall in between, as when an ad-hoc query's dataflow finishes, doesn't break the verdict window or the chart line. A span whose end is below its start, as after a restart, isn't measured, and the chart's line breaks there.
• A per-dataflow read that fell rebases the table's baseline.
• Work in dataflows that start and finish between two readings is never seen. The high-CPU verdict counts it as outside running dataflows and says so.

Supporting changes.

• useObjectLink is shared with the memory by object card.
• TimeAxis takes a tickFormat.
• SubscribeCollection gains getRows.
• Size allocations include workers.
• SessionVariables includes statement_timeout.
• The Processes drawer takes an initial replica and a link to the Workers drawer. Its findings are an Alert with a headline and the numbers behind it, and its 1.25× evenness rule (standsOut, EVEN_RATIO) moves to processReadingsModel.ts, shared with the worker verdict.
• buildExtendedRequestQuery is exported.

The drawer ships with the redesign, so it is behind the same UI preview flag and off in self-managed.

Cost.

• On mz_catalog_server: one session per stretch of reading, and one metrics SUBSCRIBE per open drawer.
• On the diagnosed replica: one join and reduce over introspection every 5s, plus a per-dataflow read every 30s. That read returns one row per dataflow and worker.

The read time on a large replica isn't measured yet. The backoff and the 10-minute cap bound it.

Verification

• workerCpuModel.test.ts:
  • busy shares between readings and a 30s trailing average, with no line across a pause or a counter reset;
  • spans compare only their ends: ad-hoc queries finishing between readings keep the chart line and the verdict window, while a restart breaks both;
  • verdicts for saturated, uneven, light, balanced, every worker busy, idle and high CPU, including high CPU with idle dataflows and high CPU over uneven work;
  • a pegged worker is caught on a two-worker replica, and with peers each well behind it;
  • dataflow time is measured against cores;
  • per-object rows over a window, with ad-hoc dataflows pooled, and a hottest worker on two workers isn't capped at twice an even share;
  • a worker's top objects, with the other workers' average time on each;
  • replica CPU per scrape, averaging processes.
• workerCpuSampler.test.ts:
  • reads every 5s and per-dataflow reads at most every 30s;
  • backs off to ten times a slow read;
  • pauses itself and closes its session at the 10-minute limit;
  • pausing and resuming during a read keeps one cadence and draws no false gap;
  • resuming after a failed read reads again on a new session;
  • a fallen per-dataflow counter rebases the baseline;
  • a new sampler continues from earlier readings, marked after a gap;
  • a read that lands after stop is ignored.
• WebsocketQueryClient.test.ts:
  • queued queries run one at a time and resolve with their own rows;
  • an error rejects only its own query;
  • a close rejects the pending query and every later one;
  • numeric parameters are sent as strings.
• ResourceUsage.test.tsx: the Workers and Processes drawers switch to each other on the same replica.
• BusiestWorkers.test.tsx: the busiest worker's breakdown shows even when work is balanced, with each object's time against the other workers'. An object the busiest worker spent less on reads as a multiple below 1×, naming the object's hottest worker when the object is skewed onto another one, for example "3s · 0.4× the others, hottest w2".
• ObjectCpuTable.test.tsx:
  • a window is measured against the replica's allocation, with what ran outside dataflows;
  • a lifetime is measured against all dataflow time;
  • an object too small to matter isn't flagged as skewed.
• WorkerCpuVerdict.test.tsx:
  • a saturated worker's verdict offers a copyable EXPLAIN ANALYZE CPU WITH SKEW, pinned to the replica;
  • high CPU outside dataflows links to ad-hoc query troubleshooting;
  • high CPU in dataflows doesn't suggest splitting clusters;
  • low replica CPU is only called out when it is low;
  • a saturated worker on a multi-process replica is placed on the busiest process, or names the process that was, and with even processes says the skew is between workers;
  • the first look names a lifetime-skewed worker and leaves out stale CPU metrics.

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. MEDIUM -- Skew verdicts can't fire on 2-worker replicas, which get "spread evenly" instead

console/src/platform/clusters/ClusterPage/WorkerCpu/workerCpuModel.ts:213

isSkewed compares the busiest worker with an average that includes the busiest worker, so the ratio can never exceed the worker count. On a 2-worker replica (M.1-xsmall, the smallest size that gets the Workers button), saturated and uneven can only fire if the other worker is completely idle. A worker pegged at 100% next to one at 20% is therefore reported as "Work is spread evenly", or as "this isn't worker skew" once replica CPU reaches 75%.

Details

With two workers, busiest >= 2 * (busiest + other) / 2 reduces to other <= 0. The introspection logging dataflows run on every worker, so the other worker never reads exactly zero. For [1.0, 0.2], the average is 0.6 and 1.0 < 1.2, so workerVerdict returns balanced. If the replica is at 75% or more, cpuVerdict takes the highCpu branch, and WorkerCpuVerdict.tsx:202 prints "the busiest worker 100% of its core, so this isn't worker skew." Either way, the EXPLAIN ANALYZE ... WITH SKEW suggestion never appears, and BusiestWorkers collapses into a single range. The lifetime check in the first look (WorkerCpuVerdict.tsx:137) has the same ceiling.

This is the case the PR description motivates the drawer with: one worker pegged while cpu_percent looks modest. Four workers are affected too. In [1.0, 0.4, 0.4, 0.4] one worker is pegged at 2.5× each peer with the replica near 55%, and the verdict is still balanced, because the busiest worker has to outweigh all the others combined. The tests only use four-worker inputs with near-idle peers, such as [0.97, 0.14, 0.12, 0.13].

A fix is to compare the busiest worker against the mean of the other workers. That gives 5× for [1.0, 0.2] and 2.5× for the four-worker case. The existing test cases keep their verdicts: [0.21, 0.17, 0.16, 0.18] comes out at 1.24× and stays balanced.

[def-]

QA LLM Review

1. MEDIUM -- High-CPU verdict headline says "spread evenly" while its own body reports uneven work

console/src/platform/clusters/ClusterPage/WorkerCpu/WorkerCpuVerdict.tsx:249

This push turns uneven work under high replica CPU into a note on the highCpu verdict (workerCpuModel.ts:293). The dataflow-dominated branch still uses the headline "mostly running dataflows spread evenly across its workers", so the drawer's top-line finding contradicts the "Work is uneven, w0 at 85% … against 42% for the other workers" sentence directly below it.

Details

This branch is easy to reach. Take a 2-worker replica at one worker per core, with workers [0.85, 0.42] and the replica at 80% CPU. workerVerdict returns uneven, since 0.85 ≥ 2 × 0.42 and 0.85 < 0.9. cpuVerdict then returns highCpu with dataflowFraction 0.635 and outsideFraction 0.165. Because outside ≤ dataflow, Verdict takes the branch at line 246 and renders "r1 is using 80% of its CPU, mostly running dataflows spread evenly across its workers." Four workers at [0.85, 0.42, 0.42, 0.42] hit the same branch.

Before this push the headline was accurate, because an uneven verdict never reached highCpu. No test renders highCpu with uneven set. The model test at workerCpuModel.test.ts:272 uses [0.7, 0.2, 0.2, 0.2] at 90% CPU, which lands in the "mostly outside" branch instead.

One fix is to drop "spread evenly across its workers" from the label when uneven is set.

[def-]

QA LLM Review

1. MEDIUM -- Finished ad-hoc queries keep resetting the verdict window, so replicas loaded by ad-hoc queries never get a verdict

console/src/platform/clusters/ClusterPage/WorkerCpu/workerCpuSampler.ts:262

Any reading where a worker's counter fell is marked afterReset, and verdictWindowStart won't reach back past it. When a slow-path SELECT is running at one reading and has finished by the next, all of its accumulated time leaves the counters. On a replica whose CPU goes mostly to queries that take a few seconds, that happens every few readings. The window never reaches 30s, so the drawer shows "Measuring…" until the 10-minute pause and never gets to the "mostly outside its running dataflows" verdict that links to ad-hoc query troubleshooting.

Details

A worker's counter sums only the dataflows that are still running. A slow-path peek runs in a transient dataflow, which src/adapter/src/coord/peek.rs:1000 drops once the peek is queued, and handle_shutdown then retracts its time. Between two readings 5s apart, a counter moves by the new work minus whatever finished queries had already accumulated at the earlier reading. When maintained dataflows add little, that is often negative, and hasCounterReset has no tolerance for it.

I ran synthetic counters through the PR's hasCounterReset and verdictWindowStart, with the drawer's 30s gate. The load was 10% maintained work plus 70% from back-to-back queries, so the replica sits at 80%. With 2, 3, 4, 6 or 8s queries, 30 to 100 of 121 readings were resets, and no reading in 10 minutes had a verdict. With random 1.5 to 4.5s durations, 45 of 120 readings were resets and none had a verdict. At 20 to 30% maintained work, a verdict showed on roughly 55 to 60% of readings. At 40% or more, there were no resets. The worker chart breaks its line at the same readings, so most of its points fall below the 10s minimum.

workerBusyBetween only compares the window's first and last readings, so falls in between don't distort the verdict. One option is to break the window only when the last reading is below the first. A replica restart, or a dropped dataflow with more time than the window added, still trips that. A query that was running when the window started is undercounted by a few seconds, against minutes of new work.

[leedqin]

Both QA reviews are addressed in 87f9e42:

• Skew on two-worker replicas: the busiest worker is now compared with the other workers' average, in the live verdict and the first look. [1.0, 0.2] and [1.0, 0.4, 0.4, 0.4] both read as saturated, and the earlier cases keep their verdicts.
• Finished ad-hoc queries resetting the window: the verdict window and the chart line now compare only a span's two ends, so a fall in between doesn't break them. A restart, where every earlier reading is higher, still does. The model tests cover an ad-hoc load where every other reading falls.

[def-]

QA LLM Review

1. MEDIUM -- The saturated-worker verdict names a busiest process even when the processes are even

console/src/platform/clusters/ClusterPage/WorkerCpu/WorkerCpuVerdict.tsx:303

The new process sentence names whichever process has the highest mean CPU, with no evenness threshold. A process with many workers barely moves when one of them saturates, so on large multi-process replicas the sentence often credits another process with "the most CPU" by about a point. It then links to the processes drawer, which applies EVEN_RATIO (1.25) and calls the same processes even.

Details

busiestProcess (processReadingsModel.ts:152) is a bare argmax. ProcessesDrawer only calls a process busiest when standsOut holds (ProcessesDrawer.tsx:158). WorkerCpuDrawer.tsx:245 passes the raw result to Verdict, which always prints a sentence for it.

Take 2 processes of 62 workers, with w3 at 100% and every other worker at 20%. w3 adds (1 − 0.2)/62 ≈ 1.3 points to process 0, so process 0 reads about 21% and process 1 about 20%, a ratio of 1.06. Any larger difference between the processes, for example in work outside dataflows, decides which one comes out ahead. If process 1 wins, the verdict for w3 says "Process 1 used the most CPU, 21% against 20% for the other 1." That points away from the worker the verdict is about. "CPU and memory by process" then opens a drawer that says "CPU and memory are spread evenly across the processes." If process 0 wins, "Its process, process 0, used the most CPU" claims a lead the processes drawer doesn't count. The tests only use 61 vs 34 and 70 vs 40, both well past 1.25.

One fix is to share standsOut with the verdict, name a process only when it stands out, and otherwise say the processes used CPU evenly.

[leedqin]

Busiest process named when the processes are even is addressed in 80fef06. standsOut now lives in processReadingsModel.ts, and both the processes drawer and the worker verdict use it. The verdict names a process only when it passes 1.25× the others. Otherwise it says the processes used CPU about evenly and the skew is between workers, not processes, so it agrees with the drawer it links to.

[def-]

QA LLM Review

1. MEDIUM -- The busiest worker's breakdown says "about as much as the other workers" when it spent far less

console/src/platform/clusters/ClusterPage/WorkerCpu/BusiestWorkers.tsx:113

comparedWithOthers only checks the upper side of EVEN_RATIO, so any ratio below 1.25 reads "about as much as the other workers", including 0.2×. The breakdown is new in this push and shows on every verdict. When another worker carries one of the busiest worker's top objects, it calls that object even on this worker, while "Where the CPU went" flags the same object as skewed.

Details

This needs one object hot on the busiest worker and another hot on a different worker, for example two keyed objects whose hot keys hash to different workers, or a single-key reduce. Take 4 workers, a_idx at [20, 2, 2, 2] s and b_mv at [3, 3, 15, 3] s, with w0 the busiest. I rendered BusiestWorkers at the PR head with these rows, and w0's breakdown read:

• "a_idx 87% · 20s, 10.0× the other workers' average"
• "b_mv 13% · 3s, about as much as the other workers"

The other workers averaged 7s on b_mv, so w0 spent 0.43× their average. The table's hottest-worker cell for the same row reads "w2 · 5.0×" in red. Tests cover only ratios of 6× and exactly 1×.

One fix is to treat the band from 1/EVEN_RATIO to EVEN_RATIO as even, and below it say "less than the other workers' average", or give the multiple.

[leedqin]

Breakdown calling a much smaller share even is addressed in bbbd2b7. Even is now the band from 1/1.25 to 1.25. Below it, the line says "less than the other workers" and gives the multiple. When the object is skewed onto another worker, it also names that worker, so it agrees with the table's hottest-worker cell. Your a_idx / b_mv example is the new test: b_mv on w0 now reads "3s, less than the other workers, 0.4× their average, hottest on w2".

[def-]

QA LLM Review

1. MEDIUM -- The new "less than the other workers" note is always clipped, so "hottest on wN" never shows

console/src/platform/clusters/ClusterPage/WorkerCpu/BusiestWorkers.tsx:129

The string this push adds for objects below the even band is wider than the breakdown's fixed 280px note column (templateColumns at line 83), and the column truncates with an ellipsis and no tooltip. Every such row is cut off in the middle of "their average", so the hot worker this branch names, the stated purpose of the comment at line 115, never shows in the drawer.

Details

I rendered the notes in Chromium using the console's Inter font and the text-small style (12px, the same font features), in a 280px nowrap/ellipsis box:

• "3s, less than the other workers, 0.4× their average, hottest on w2" is 386px wide and shows as "3s, less than the other workers, 0.4× their av…"
• "12m 34s, less than the other workers, 0.43× their average, hottest on w127" is 447px wide and shows as "12m 34s, less than the other workers, 0.43× …"
• Without the hottest-worker suffix, "450ms, less than the other workers, 0.7× their average" is still 327px wide.

Both pre-existing strings fit: "20s, 10.0× the other workers' average" and "3s, about as much as the other workers" each measure 280px or less. The new test at BusiestWorkers.test.tsx asserts toBeVisible() on the full string, and that passes because jsdom does no layout. The table below still shows the hottest worker, so nothing is lost from the drawer as a whole, but the breakdown row cannot show it. Shortening the phrase (for example "0.4× the others, hottest on w2") or letting the note wrap would fix it.

[leedqin]

Clipped "less than the other workers" note is addressed in 0e88dc6. The note is shorter and matches the table's wording: "3s · 0.4× the others, hottest w2". It also wraps to a second line before it truncates, and the full note shows on hover, so a long duration or a three-digit worker id still shows the hottest worker.