By Adam Wolf & Damian Erangey
Almost every infrastructure team running modern AI is wrestling with the same question. The orchestrator their AI workloads want, Kubernetes, and the orchestrator their HPC environment was built on, Slurm, pull in different directions.
This piece, drawn from the HPCKP 2026 session of the same name, looks at why that tension exists, what the workloads actually look like, how teams are bridging the two today, and where the pattern is heading. It closes with five concrete actions and how ClearML, an AI infrastructure platform with roots in HPC, fits the meta-scheduler approach.
Same GPUs, two operating models, one workforce in the middle
HPC centers run on Slurm, PBS Pro, and LSF: battle-tested, fair-share, and tuned over years for tightly coupled MPI. AI teams want, and increasingly need, Kubernetes: containers, elasticity, GitOps, and a deep cloud-native ecosystem. The catch is that both often target the same expensive GPUs, with one workforce caught in between. The question is no longer which scheduler wins. It is how the two coexist.
Two orchestrators, two design centers
Before talking about bridging them, it helps to be precise about what each system was built to do, because it was not the same job.
What HPC schedulers were built for
Slurm, PBS Pro, and LSF exist to squeeze maximum throughput out of a shared, expensive cluster running tightly coupled scientific code. They bring fair-share queues tuned over decades, gang scheduling (a 100-node MPI job gets all of its nodes at once or it waits), and topology awareness (the scheduler knows which nodes share an InfiniBand switch and places processes accordingly). They run on bare metal with very low overhead. Climate simulation, molecular dynamics, and distributed training over InfiniBand are where they shine.
What Kubernetes was built for
Kubernetes comes from a different design center: long-running containerized services that self-heal, scale out, and survive node failures. It is declarative (you describe the desired state and Kubernetes makes it true), elastic by default (so it works well with cloud and spot capacity), and it carries the cloud-native ecosystem of operators, service mesh, GitOps, and monitoring. For inference serving, CI/CD, and the stateful pieces around an AI workflow, this is the right tool.
Neither is wrong. The mistake is forcing one to do the other’s job. Slurm is poor at long-running services, and Kubernetes, until recently, was poor at gang-scheduled, tightly coupled batch. The productive answer is to use both, deliberately.
Workload gravity: where each job naturally wants to run
Every workload has a center of mass. Fight that gravity and you pay for it in operational pain. Tightly coupled distributed training (multi-node, NCCL over InfiniBand, hundreds of GPUs communicating every step) has strong HPC gravity: it wants gang scheduling, topology-aware placement, and bare-metal latency. Push it onto vanilla Kubernetes and you can spend months on networking and pod-affinity rules. Inference serving sits squarely in Kubernetes territory, with autoscaling that is traffic-driven and can scale to zero, and the CI/CD pipelines around training belong there too. Hyperparameter sweeps are a softer case (hundreds of single-GPU trials can go either way), but Kubernetes with a batch operator on top is usually simpler. Then there is a flexible middle band, single-node training, data preprocessing, and short fine-tuning runs, that genuinely does not care where it lands. That middle band is where a smart control plane earns its keep, routing flexible work to whichever backend has free GPUs right now.
| Workload | Best fit | Why |
|---|---|---|
| Large-scale MPI training | HPC scheduler | Gang scheduling, topology aware, InfiniBand |
| Multi-node distributed training | Either | Kubernetes is closing the gap with Volcano, Kueue, KAI, and ClearML |
| Single-node fine-tuning | Kubernetes | Containerized, fast iteration, ephemeral |
| Hyperparameter sweeps | Kubernetes | Embarrassingly parallel, elastic, queue-friendly |
| Model serving and inference | Kubernetes | Autoscaling, blue/green, service mesh |
| Classical HPC simulation | HPC scheduler | Tightly coupled, long-running, fair-share policy |
The pattern is simple. HPC keeps the tightly coupled jobs, Kubernetes absorbs the containerized AI workflows and serving, and a smart middle layer routes everything else.
The three hard parts of going hybrid
The hard part is not the technology. It is getting the two systems to cooperate. Three places are worth naming.
Data and storage
Data often lives on a parallel filesystem (BeeGFS, VAST, WEKA, DDN etc) and Kubernetes expects CSI volumes and object storage. Moving terabytes between them is slow and costly, and the common failure mode is teams duplicating data across both worlds. The short answer is a shared namespace underneath both, so the same data is reachable by the same path whether a job lands on Slurm or Kubernetes. Solve it once, at the storage layer.
Security and identity
HPC clusters authenticate with Kerberos, LDAP, and host UIDs. Kubernetes uses service accounts, RBAC, and network policies. Reconciling identity, secrets, and audit logs across both is genuinely hard, and getting it wrong invites a security incident, a user revolt, or both.
Operational overhead
This is the one that actually kills projects. Two submission paths, two queues, two monitoring stacks, two on-call rotations. Users deal with it every day, and admins carry it forever. Do nothing and the cost compounds.
The target architecture: one control plane, two backends, shared resources
The architecture that teams are moving toward, whether they are evaluating it, piloting it, or already running it in production, has three layers. At the top sit the users: AI builders, researchers, CI/CD pipelines, and notebooks, all submitting work through one interface, whether their code is classic HPC or standard Python ML. In the middle sits a unified control plane, the brain that handles job queuing, applies policy (quotas, priorities, project budgets), and provides observability across everything. This is where the meta-scheduling decision happens: the control plane reads the submitted job (its GPU type, resource requests, coupling, and whether it arrived as a container spec or an sbatch-style batch job) and routes it to the right backend. Below that sit the two backends: the HPC scheduler (Slurm, PBS Pro, LSF) doing what it does well, and Kubernetes with batch primitives such as Volcano or Kueue layered on for gang scheduling and queue semantics.
ClearML can serve as that control plane, because schedulers, queues, and jobs are its core primitives across bare metal, Slurm, and Kubernetes. At the bottom, the layer people underestimate, is a shared physical fabric: the same GPUs, the same parallel filesystem, the same InfiniBand or RoCE network. Two hardware silos break the economics the moment you double-buy GPUs.
Users submit once. The control plane routes to the right backend based on workload shape. The hardware stays shared.
Three ways teams are closing the gap today
These approaches are not mutually exclusive.
Approach 1: Kubernetes on Slurm
Treat Kubernetes itself as a Slurm job: request nodes for a fixed window and stand up an ephemeral Kubernetes cluster on them. You reuse all of your HPC fair-share policy, and Slurm stays the source of truth. The tradeoff is that Kubernetes prefers to be long-lived, so making it short-lived and elastic can be cumbersome. You see this in research labs and prototypes, such as Usernetes and various K8s-in-Slurm projects.
Approach 2: Slurm on Kubernetes
The mirror image: containerize Slurm, run the controller and worker daemons as pods, and let Kubernetes own the underlying nodes while Slurm keeps the scheduling logic. This is far easier for new clusters built from scratch and harder to retrofit onto a production Slurm install you are not allowed to touch. The Slinky project from SchedMD and various Slurm operators are the things to look at.
Approach 3: A meta-scheduler (Winner)
Put a job router above both schedulers. Slurm stays as it is, Kubernetes stays as it is, and the meta-scheduler becomes the single submission interface: it applies policy, routes jobs by workload shape, and gives you one observability pane. Examples include ClearML and Volcano-plus-Kueue federations. This approach is the winner. It does not force you to rebuild existing infrastructure; it respects what each scheduler is good at, and it puts the integration burden in the layer that is cheapest to change, your platform code, not your physical cluster.
Where the industry is heading
Four trends are pushing meta-scheduling from option to default.
- Kubernetes is absorbing workflow orchestration. Batch primitives, gang scheduling, and queue semantics that once meant building it yourself are now available from multiple projects, including Volcano, Kueue, and ClearML. Workloads that used to require Slurm increasingly run well on Kubernetes.
- HPC sites are adopting containers. Pyxis and Enroot on the Slurm side, and Apptainer and Singularity more broadly, turn a training image into a portable artifact. The same container can run under Slurm one day and Kubernetes the next. The artifact is unified even when the schedulers are not.
- Fractional and elastic GPUs are going mainstream. NVIDIA MIG, AMD partitioning, and software-level fractional GPUs let a single accelerator host multiple isolated workloads, which changes capacity planning and makes a smart meta-scheduler more valuable, because it routes on finer-grained resources than whole GPUs.
- Meta-scheduling is becoming the norm. Platform teams no longer accept the framing of Slurm or Kubernetes as the right question. They expect a single submission surface above both.
That last point is backed by the numbers. According to ClearML’s State of AI Infrastructure at Scale 2025-2026 report, almost half (49.2 percent) of IT leaders at F1000 companies named maximizing GPU efficiency across existing hardware, including shared compute and fractional GPUs, as their top priority for the next 12 to 18 months.
Five tasks for Monday morning
- Inventory your workloads. Classify each by three dimensions: how tightly coupled it is, how long it runs, and what its data path looks like. In the talk’s experience, a common split is roughly one fifth that truly needs HPC primitives, about a third that are unambiguously Kubernetes, and a remainder that can live wherever capacity is free. Most teams skip this step and then architect in the dark.
- Pick one primary scheduler per cluster. Per cluster, not per slide deck. HPC-first means Slurm wins and Kubernetes is the bridge target; K8s-first means the reverse. Avoid a 50/50 split on a single cluster, because it doubles operational cost and never gets cheaper.
- Standardize the artifact. Make a container the unit of work, runnable under Slurm (Pyxis or Apptainer) and under Kubernetes natively. Same image, two backends. This is the cheapest, highest-leverage move on the list, and the one teams most often under-invest in.
- Unify identity and data. Solve authentication once (OIDC across both worlds), solve accounting once, and solve storage paths once, so the same dataset is reachable by the same path from a Slurm job and a Kubernetes pod. The seams are where projects die, and most of that cost lives here. Fix it early and everything else gets cheaper.
- Put a meta-scheduler on top. Give users one queue, one CLI, and one observability pane, and let policy (quota, project, priority) decide where a job lands, rather than platform constraints or whichever scheduler someone happens to know.
How ClearML bridges both worlds
ClearML is a concrete example of the meta-scheduler approach, and its roots are in HPC. Schedulers, queues, and jobs are its core primitives, applied uniformly across bare metal, Kubernetes, and Slurm.
The screenshot below highlights this approach. Here we have configured two distinct GPU resource pools backed by a single physical pool of GPUs.
- A SLURM resource pool to accept and run SLURM jobs
- A Kubernetes resource pool to accept and run container-shaped operations (GPUaaS Operations, MLOperations, Model Serving, LLM serving, etc.)
From a user’s point of view, the backend does not matter: they assign tasks to the correct queue, and ClearML handles the scheduling onto the correct pool/cluster.
Option A: Manage Slurm through the ClearML Glue Agent
This pattern suits sites with a healthy Slurm cluster they are not going to rebuild. You install the ClearML agent on a node that can run sbatch and map each ClearML queue to an sbatch template. Agents can also utilize Singularity or Pyxis containers in Linux clusters managed with Slurm.
When a user pushes a job to that queue, the agent converts it to an sbatch submission automatically, with environment, packages, and data prewired, and submits it to the existing cluster. Containers under Slurm (Singularity, Pyxis, Enroot) work transparently.
The screenshot below shows ClearML working as a meta scheduler, with a user submitting a fine-tuning script to the SLURM queue. Note that there are also other GPU queues available for non-SLURM-specific tasks.
Users get live queue visibility and can reprioritize or abort jobs without admin access to Slurm. ClearML manages the monitoring at this stage while Slurm keeps running the HPC infrastructure, and nothing changes for an existing user’s code.
Best fit: mature HPC sites that want AI and ML lineage without disrupting Slurm.
Option B: Run Slurm inside Kubernetes
This is the other direction, for organizations without a dedicated Slurm cluster. ClearML can provision a Slurm cluster on demand inside Kubernetes (a login pod and a set of worker pods, with MPI and multi-node networking wired automatically) and tears it down when the queue empties. Users still submit sbatch as if it were a normal cluster, while the platform sees elastic Kubernetes capacity. Best fit: Kubernetes-first organizations that need HPC-style workflows without standing up a static cluster.
Whichever pattern fits your current workload gravity, the layer above stays the same: one control plane for experiments, lineage, fractional GPUs, quotas, and observability across every backend you run.
Learn more
Read ClearML’s State of AI Infrastructure at Scale 2025-2026 report for the full survey data, or explore the ClearML Infrastructure Control Plane to see how a single control plane manages Slurm, Kubernetes, and bare metal together.