Every student in the machine-learning course got a workspace with a 12-core CPU limit, some with 24. Together those limits add up to more than four times the cluster's CPU, yet the jobs almost never waited for a core. These charts show why, and where the CPU actually went.
Overcommitment1 / 9
We promised 4.6× the CPU we have
CPU limits of the ML workspaces placed on each node, against the node's allocatable cores and the most the ML workspaces ever used together in one minute.
Allocatable coresML workspace CPU limitsPeak ML usage (1-min)
Data table
Usage over the semester2 / 9
The whole semester fit inside the cluster
Busiest minute of each day across the three worker nodes. ML workspaces peaked at 42 cores, a third of the cluster, and everything on the nodes together never came close to the 128 allocatable cores.
All workloads on the nodesML workspaces
Concurrency3 / 9
Students don't all press Run at once
Share of busy minutes by how many ML workspaces were running a job (above 1 core) in the same minute, across the cluster. We sized for a lab of 25. We never saw more than 4.
Data table
What held jobs back4 / 9
The node held back an ML job for one minute all semester
Every minute an ML workspace used more than 1 core, classified by what capped it. Almost always the job simply used what it needed. When it was capped, its own CPU limit did it, not a crowded node.
Data table
Headroom5 / 9
When ML jobs ran, the node had cores to spare
Free cores on the node during minutes when an ML workspace was running a job. Red marks minutes with less than 4 cores left: one minute on kube-w2, none on the other nodes.
Data table
Where the CPU went6 / 9
The biggest CPU consumer was forgotten processes, not ML jobs
CPU-hours per day used by ML workspaces. Between mid-March and late May, three workspaces kept a process spinning at a steady 2.5–3 cores, day and night, for up to six weeks each. They used of all ML workspace CPU. The real ML jobs are the short spikes.
Runaway processes (3 workspaces)ML jobs (all other workspaces)
Data table
The worst overlap7 / 9
Two jobs at their full limit on one node, half the node still free
kube-w3, Sunday 29 March 2026: the heaviest overlap of the semester, with two ML jobs side by side on one node. Both held their 12-core limit for about 12 minutes, while the node never used more than half of its 64 cores.
Workspace AWorkspace BNode CPU in use (all workloads)
Memory8 / 9
Memory was never overcommitted
Unlike CPU, the 12 GiB memory limits of all ML workspaces on a node fit inside its allocatable memory. The busiest node used 45% of its memory at peak, counting every workload, and no node ever reported memory pressure.
Allocatable memoryML workspace memory limitsPeak memory in use, all workloadsPeak ML workspace memory
Data table
Memory per workspace9 / 9
Most workspaces used a fraction of their 12 GiB
Peak memory, ML workspacePeak memory, runaway workspaceTypical (median) memory
Data table
How these numbers were measured
Source: Prometheus fed by the OpenTelemetry collector (kubeletstats and k8s_cluster receivers), 60-second resolution, 12 January – 31 July 2026.
ML workspaces are the 38 workspaces whose main container runs the DDW (machine learning) image, each with a 12- or 24-core CPU limit and a 0.1-core request, on three worker nodes with 32, 32 and 64 allocatable cores.
A workspace counts as running a job in a minute when it averaged more than 1 core. "Own limit" means it ran at 90% of its limit or more. "Node short of CPU" means fewer than 4 allocatable cores were free on its node.
Three workspaces ran a constant 2.5–3 core background process for weeks. They are shown on slide 6 and left out of the job analysis on slides 3–5 and 7, because they were not jobs a student was waiting for. Slides 1–2 include them.
Memory is the pod working set (the figure the kernel and kubelet act on), per minute. Every ML workspace had a 12 GiB memory limit and a 0.125 GiB request; the three runaway workspaces are included in the memory slides. A process that exceeds the limit is killed by the kernel without restarting the workspace, so likely kills are counted from a drop of 3 GiB or more within 3 minutes after reaching 95% of the limit.
Values are 1-minute averages, so spikes shorter than a minute are smoothed out. CFS throttling counters were not collected, so limit-capping is inferred from usage reaching the limit.