By Adam Wolf
Enterprise AI compute does not live in one place. It lives on Kubernetes clusters, on Slurm and PBS schedulers inherited from two decades of HPC investment, on virtual machines and bare metal kept for isolation or licensing reasons, and on burst capacity in the cloud. A queue-based scheduling model abstracts all of it behind one interface, so a user submits work the same way regardless of which fabric executes it. By fabric we mean the execution substrate a job actually lands on: a Kubernetes cluster, a Slurm or PBS cluster, a pool of VMs, bare metal, or cloud capacity that comes and goes. This post explains why the multi-fabric reality is permanent, what Kubernetes does and does not solve, and how a unified queue actually works.
Why isn’t Kubernetes enough to run all of your AI compute?
Kubernetes can’t run all of your AI compute because a large share of AI-relevant compute was never on Kubernetes, is not moving to Kubernetes, and has good reasons for both.
Start with the HPC estate. Research labs, national facilities, semiconductor firms, and pharma R&D run Slurm and Portable Batch System (PBS) clusters that represent decades of tuning and institutional knowledge. Those schedulers were built for the exact shape that large-scale training work has:
- Gang scheduling. A 512-process MPI job has to start every one of its processes at once or not start at all.
- Backfill. Small jobs drop into the gaps ahead of a large reservation without pushing it back.
- Topology-aware placement. Related processes land on nodes that sit close to each other across low-latency interconnects.
Kubernetes, by contrast, grew up scheduling stateless microservices one pod at a time. Its default scheduler has no native concept of “all 64 pods or none,” which is why the ecosystem has spent years building batch add-ons (Kueue, Volcano, and their predecessors) to graft batch semantics onto it. The grafts work. But a team that already runs a tuned Slurm cluster would be giving up a scheduler that natively understands gang-scheduled batch work in exchange for one that has been taught to imitate it. Very few make that trade, and it is hard to argue they should.
The best approach is hybrid and integrates with the HPC schedulers instead of replacing them: Klique’s HPC integration describes Slurm and PBS workloads being submitted, scheduled, and tracked through the same queues and resource controls as the rest of the platform, without changing how the jobs are written.
Then there is the other end of the spectrum: workloads Kubernetes was not designed for, arriving from the agentic side. Google’s Agent Substrate is a scheduling layer built alongside Kubernetes because agent sessions look like sleeping OS processes rather than long-running services. Google multiplexes mostly-idle sessions onto limited compute by snapshotting them and oversubscribing capacity 30x or more, and its Agent Sandbox allocates 300 sandboxes per second per cluster. As a result, GKE sandbox use has grown 16x in five months.
When the company that invented Kubernetes builds a second scheduler beside it for a new class of workload, that says less about Kubernetes than about how wide the range of scheduled work has become. No one fabric covers all of it.
Meanwhile, regulated industries keep VMs and bare metal for isolation, air-gapped networks, and appliance-style deployments. Add cloud burst capacity, and the realistic enterprise picture is four or five permanent fabrics. Standardizing on one means abandoning capacity you already own. On the other hand, letting each fabric keep its own front door means every user learns several submission systems, or more realistically, files a ticket and waits.
What does a unified queue model actually do?
A queue is a policy boundary with a stable interface on the user side and pluggable execution on the infrastructure side.
On the user side, work enters a queue with a resource request: this container or task, this many GPUs or this fraction of one, this memory, this priority. The user’s mental model ends there. A data scientist launching a notebook, a researcher submitting a 256-GPU training run, and an agent requesting compute through an API all speak to the same abstraction. That is the point we argued in The Three Languages of Enterprise AI: the translation cost between tickets, APIs, and manifests is where enterprise AI programs stall.
On the infrastructure side, the Klique Orchestrator monitors queues and places each job on whatever fabric backs that queue. The same pattern exists against three different fabrics, and the last column is the one that matters:
| Fabric | How a queued task becomes a running job | What the user changes |
|---|---|---|
| Kubernetes | A Kubernetes-Glue controller pod pulls from the queue and renders a Kubernetes job from a pod template. | Nothing |
| Slurm | The queue carries a batch script template, and a queued task is converted into an sbatch job. | Nothing |
| VMs and bare metal | An agent services one or more queues and runs the tasks it finds there. It also rebuilds the environment, so the code, the container or package environment, and the configuration all arrive on the target machine. | Nothing |
Which fabric ran the job is recorded, but from the submitter’s perspective it is an implementation detail, in the same way nobody asks which physical core ran their thread.
Between those two sides sits policy: the queue-to-fabric mapping, plus the quotas, reservations, and limits that decide placement. Klique’s resource policy is explicit about how capacity reaches people, stating that administrators expose allocated compute to users through execution queues, and the queues themselves are created and monitored in the orchestration UI. That policy layer is what the whole exercise produces. You already had the fabrics. What you did not have was one place to decide who gets them.
Users and agents submit to a queue. The queue carries the policy, and the orchestrator places the work on whatever fabric backs it. Nothing has to move for this to be true.
How do quotas and fair share keep the queue honest?
Without policy, shared queues degrade into whoever submits first or whoever is willing to submit at 2 a.m. Three mechanisms keep shared capacity civilized.
A reservation is the floor. Klique’s resource policy defines a reservation as the amount of compute guaranteed to be available to the group, so the fraud-model team always has access to at least N GPUs’ worth of capacity, no matter what the rest of the company is doing. Reservations are what make a shared pool politically survivable, because critical teams cannot be starved.
Limits work in the other direction, capping the top. The limit is the maximum amount of compute the group’s jobs can use concurrently, so no team, however enthusiastic, can consume past its cap while others wait. Limits are what prevent the tragedy of the commons that shared clusters are famous for.
In between the two, prioritization decides the order. The documented controls are workload prioritization across available resources and, on the worker side, queue ordering: a single agent can service multiple queues, with priority set by their order. The principle those controls are there to enforce is fair share, meaning a team running over its quota does not get to push other teams’ jobs out of the queue.
The operational consequence is that queue wait time becomes a meaningful, per-team service-level indicator rather than random suffering. Our AI Workload SLOs post proposed treating 95th-percentile queue wait as a platform SLO, under an hour for standard jobs, minutes for high-priority work. That SLO is only writable because the queue is the single choke point where wait is measured and policy is enforced. Five fabrics with five front doors produce five incomparable wait-time distributions and no accountability.
What about the humans who used to file tickets for this?
IT does not go away. The job moves up a level: instead of provisioning individual requests, platform teams define the queues themselves, map each one to a fabric, and set the policy behind it, meaning who may use which queue, with what reservations, limits, and priorities.
The ticket path is what that replaces. So what dies with it?
In the ticket world, a data scientist who needs a GPU VM files a request; IT translates the request into infrastructure actions (find capacity, provision, configure access); days pass; the credentials arrive; and six weeks later nobody remembers to reclaim the VM, which joins the stranded-capacity statistics from our fractional GPU discussion. Every step is a translation between the human’s language and the infrastructure’s, performed by hand, by the most expensive translators available.
In the queue world, the same data scientist submits to a queue they are entitled to use. Policy checks entitlement in milliseconds. Capacity is granted from the shared pool, used, and returned when the workload ends. No human translated anything; the queue is the same self-service interface for people and for agents, with identity, policy, and lifecycle handled underneath by one control plane instead of three.
That is a better job, and it is the only version of the job that scales to a workforce where, as we covered in Half Your Team Is Already Vibe Coding With Personal API Keys, the population of people who want compute has expanded far beyond the teams IT used to know by name.
What changes operationally?
Three numbers move.
Utilization rises because pools that were stranded per-fabric and per-team become one schedulable supply, and fractional allocation can pack the interactive tail.
It also becomes observable: the orchestration dashboard reports usage by resource group and counts a worker idle when its GPUs and CPUs are idle, with GPUs treated as idle below 80% average utilization. Queue wait becomes measurable and improvable per team, because there is finally one place to measure it.
And capacity planning gets one demand signal instead of five, because every fabric’s demand flows through the same queues, so the “do we buy more GPUs or burst to cloud” conversation starts from data rather than from whichever team complained loudest. The burst path is queue-driven too: autoscalers attach to queues and provision instances when work is pending, then tear them down when it is not.
And one number stops mattering: the count of fabrics. Nothing in this model requires migrating workloads off Slurm, onto Kubernetes, or anywhere else. The fabrics stay where they are and only the front doors consolidate.
FAQ
Can one scheduler really span Kubernetes and Slurm?
Yes. Klique’s queues abstract Kubernetes, Slurm, PBS, VMs, and bare metal behind one submission interface, with the orchestrator running work on whichever fabric backs the queue. The HPC documentation covers the Slurm and PBS side, and the orchestrator reference covers placement across all of them.
Do users need to know which fabric runs their job?
No. Users submit to a queue; reservations, limits, and fair-share policy determine where the workload executes, and the placement is recorded but invisible to the workflow.
Why not just migrate everything to Kubernetes?
Because Slurm and PBS are better at gang-scheduled batch work, the migration cost is enormous, and even Kubernetes’ own ecosystem is adding new scheduling layers for agent workloads. The multi-fabric estate is permanent; the fix is one policy layer above it.
What is fair-share scheduling?
An arbitration principle: when demand exceeds supply, no team’s backlog should starve another team’s access. In Klique the documented controls that produce it are reservations and limits, which set each group’s floor and ceiling, plus workload prioritization and queue ordering between them.
Does the queue model handle agents as well as people?
Yes. Agents request compute through APIs against the same queues, under their own identities and quotas, which is the architecture we describe in Agents Need Their Own Tenants Too.
To see one queue model spanning your fabrics, request a Klique demo.
