CoreWeave's SUNK runs Slurm inside Kubernetes on one shared compute pool

The problem: two schedulers, two pools of compute

CoreWeave describes Slurm and Kubernetes as the two leading tools in the HPC and AI world. Slurm, developed by SchedMD, is the de facto scheduler for many HPC workloads and is designed for batch jobs with a finite lifespan. Kubernetes was built for long-running workloads such as inference.

According to CoreWeave, the Kubernetes scheduler for batch workloads is less popular than Slurm and lacks some functionality. Other Kubernetes-based schedulers, such as Volcano and YuniKorn, aim to offer Slurm-like scheduling but face an uphill battle against the large body of knowledge built around Slurm.

Some companies run both systems side by side. CoreWeave says this means each cluster has its own pool of compute to manage, sometimes in different clouds with separate storage. Resource management is completely separate, so any collaboration between the two requires manual work. The company argues this makes it harder to maximize utilization of GPU resources.

What SUNK does and why CoreWeave built it

SUNK brings Kubernetes containerized deployments and GitOps to Slurm, and integrates a Slurm scheduler plugin into Kubernetes. In practice, Slurm acts as a Kubernetes scheduler and Slurm jobs run inside Kubernetes. CoreWeave says this supports both burst and batch workloads on the same central platform, with compute usable from either the Kubernetes or the Slurm side.

The stated motivation is client efficiency. On very large and expensive HPC clusters, getting as close to 100% utilization as possible matters, because idle compute that is still being paid for is costly. CoreWeave says its cloud is built entirely on Kubernetes, and that many clients who preferred Slurm either managed it separately or asked for an integration. Since no such solution existed, the company built one.

Features listed by CoreWeave include:

• A Slurm cluster deployed as Kubernetes resources through a highly customizable helm chart, opening up Kubernetes-based GitOps workflows.
• Easy tracking and configuration of prolog and epilog scripts, quick deployment of staging clusters, and support for s6 scripts and services.
• Configurable authentication, including LDAP through a companion OpenLdap helm chart or third-party options such as Authentik and GAuth.
• Kubernetes benefits such as fast scheduling, containerization, high availability of control plane services, dynamic node scaling, requests and limits, and shared filesystems via PersistentVolumeClaim resources.
• State management with two-way syncing between Kubernetes and Slurm, automatic topology generation, Pyxis container execution support, and GRES support with auto-identification.

How SUNK is built

All SUNK services are containerized in Kubernetes. The typical Slurm components run inside pods, each with configurable resource requests. Login nodes are included, and once users connect to them, Kubernetes is abstracted away and the experience is that of a normal Slurm cluster. Slurm configuration, topology, prolog and epilog scripts, and sensitive data such as database passwords are deployed as Kubernetes ConfigMaps and Secrets, so they can be managed in one place.

Compute nodes run as bare-metal Kubernetes nodes. The slurmd daemons run within compute pods in a 1:1 mapping. A Slurm syncer acts as a middleman between the two sides, exchanging information through Slurm's REST API. Cluster-wide operators monitor resources and make changes whether they originate in Kubernetes or Slurm, and metrics can be exposed through Prometheus.

CoreWeave says its team custom-built three components to make this work: the Nodeset, the Syncer and the Scheduler. The Nodeset is a CRD that defines a Slurm node in the Kubernetes environment. It is defined similarly to a statefulset or deployment but maps one to one to nodes, more like a daemonset. It keeps status fields reflecting Slurm state, which enables protected updates and scaling of Slurm nodes based on state in both systems. Nodeset pods run both slurmd and munged and mount shared configmaps and shared filesystem volumes as PVCs.

The custom Kubernetes scheduler uses the synced Slurm state to place native Kubernetes workloads onto compute nodes, which lets users shift one pool of compute between Slurm jobs (bursty workloads) and Kubernetes workloads (serverless workloads).

What it means for HPC and AI teams

For organizations that rely on Slurm but want Kubernetes for production services such as inference, SUNK offers a way to keep familiar Slurm workflows without running a second cluster. According to CoreWeave, the result is a single platform and a single pool of compute, with greater flexibility in choosing which workloads run on the hardware and better odds of keeping expensive GPUs busy. The claims come from CoreWeave's own description, and the source text provided does not include performance measurements.