Two levels of scheduling
Klique is a meta scheduler. Users choose a compute resource by what it gives, for example multi-node training, large tasks or a shared GPU for a session, through whichever access channel suits them. IT decides, through policies and pool order, which pools serve each compute resource. Klique then picks the pool for each task, from the group's policy, the task size and the order of the profile's pools. A Klique agent then hands the work to that environment, where its own scheduler places it. Klique works alongside these schedulers rather than replacing them: a Slurm cluster keeps its partitions and accounting, and a Kubernetes cluster keeps its scheduler.
The resource manager in the control plane does logical scheduling and accounting: it counts GPUs against pools and policies. The Klique agent in each pool then does the environment-specific work. On Kubernetes the worker daemon renders the pod from a template, with GPU limits or a MIG slice, node selectors and volumes. On an HPC cluster the HPC submitter renders the batch script and submits it as the user's own Linux account. On bare metal the worker daemon picks free GPUs and starts a container or virtual environment. Inside the running workload, the worker daemon rebuilds the task's environment, runs it and streams logs, metrics and results back.
Pools, profiles and policies
Administrators describe capacity once, in the Resource Configuration and Resource Policies settings. Users then submit to compute resources and receive what their group's policy allows.
| Object | What it is | Set by |
|---|---|---|
| Pool | Capacity in one environment, counted in whole GPUs: a Kubernetes cluster, an HPC cluster or partition, a group of bare-metal hosts, a cloud autoscaler or a neocloud. | Administrator |
| Resource profile | A task size, fixed or read from the task, and an ordered list of pools. Within a pool, profiles have an execution priority. | Administrator |
| Resource policy | A user group's reservation and limit, and the profiles it may use. Within a profile, policies have a queuing priority. | Administrator |
| Compute resource | The queue a group submits to, one per policy and profile, named for what it gives, such as multi-node training or a shared GPU. Users pick it; IT decides where it runs. | Created when a profile is connected to a policy |
| Worker daemon or HPC submitter | The agent that pulls tasks for a pool and runs them in its environment. | Installed in each environment |
Pools are sized in whole GPUs whatever the work. A task on a share of a GPU, or on a MIG slice, counts as that share, so the accounting stays aligned with the hardware.
In the Klique documentation: Resource Management covers policies, profiles and pools; Resource Governance covers reservations, limits and quotas; Governed Compute covers how queues reach each environment; and Klique Orchestrator covers the agent installed in each environment, called the worker daemon here.
Every environment, one model
Each environment runs a Klique agent that pulls work for its pool. Clusters as a service sit beside them: Klique can stand up a Slurm cluster on Kubernetes, or a K3k virtual Kubernetes cluster, for a team that wants its own scheduler or API. The agent connects outbound to the control plane, so environments need no inbound ports. On an HPC cluster the compute nodes need outbound HTTPS to the control plane too, not only the login node, because the task's process on each node reports results, logs and metrics home. Adding an environment is the same each time: install the agent, define a pool and connect it to profiles.
| Environment | Klique component | Places the work | How a task runs |
|---|---|---|---|
| Kubernetes, on-prem | A worker daemon per cluster or namespace, installed with Helm | The Kubernetes scheduler | A pod from the compute resource's template, with GPU limits, node selectors and volumes |
| Slurm, PBS or LSF | The HPC submitter, one per cluster, on the login node | The cluster's own scheduler, with its own partitions and accounting | A batch job from a template, submitted with sbatch, qsub or bsub; the task runs natively or in a Singularity or Pyxis container |
| Bare metal and VMs | A worker daemon on each host | The worker daemon, on the host's free GPUs | A container in Docker mode, or a virtual environment on the host |
| Public cloud VMs | An autoscaler, run by the apps worker daemon, in the customer's account | One instance per task, of the instance type the autoscaler instance sets | An instance started on demand, up to the autoscaler's maximum instances, and stopped after an idle timeout |
| Managed Kubernetes, neoclouds | A worker daemon per cluster | The Kubernetes scheduler, with the cluster autoscaler adding nodes | Pods, as on any Kubernetes cluster |
| Slurm on Kubernetes | A Klique clustering service, launched on a Kubernetes cluster | Slurm, running in pods | Existing sbatch scripts, with worker pods that scale with demand |
| K3k | A Klique clustering service: Kubernetes in Kubernetes, launched inside a host cluster | The virtual cluster's own Kubernetes API, on the host cluster's nodes | A team gets its own Kubernetes cluster and namespaces without its own hardware |
How public cloud and managed Kubernetes capacity is added and removed, including VM autoscaling, managed Kubernetes node autoscaling and pool order across on-prem and cloud, is covered in the Cloud Autoscaler reference architecture.
Usage patterns
AI work now spans two kinds of scheduler. HPC schedulers were built for tightly coupled work on a shared cluster, with fair-share queues, gang scheduling and placement that follows the network topology. Kubernetes was built for containerised, elastic, long-lived services. Most estates run both, often on the same kind of GPUs, and each kind of work has a natural home.
The middle band is where a meta scheduler does the most good. Single-node training, fine-tuning and data preparation go to whichever environment has free GPUs, and multi-node training can run on either side as Kubernetes gains the batch features HPC schedulers have long had. On HPC a task runs natively or in a Singularity or Pyxis container, so the same image can run under Slurm and on Kubernetes.
Three ways to bring HPC and Kubernetes together
Where mixed estates break, and how Klique closes the gaps
Data and storage
Data on a parallel filesystem, while Kubernetes expects volumes and object storage, often ends in copies in both worlds. Present the same shared filesystem to every pool; Klique attaches volumes to tasks through templates, so a task finds its data at the same path wherever it runs.
Identity
HPC clusters use Linux accounts and directory services; Kubernetes uses service accounts and RBAC. Klique gives one sign-on and one set of roles for every environment, and on Slurm each batch job can run as the submitting user's own Linux account.
Operations
Two submission paths, two queues and two monitoring stacks double the work for users and administrators. Klique gives one way in, one view of every queue, one dashboard and one ledger across environments.
Placement patterns
Pool order is set per profile, not per user. A task goes to the first pool in the order with free capacity for its size.
The patterns below are common starting points; in each line the first item is the compute resource and the rest is the pool order.
Owned capacity first, rented capacity for peaks
Work fills the owned GPUs first. When they are full it goes to the neocloud, then to on-demand cloud instances, up to the autoscaler's maximum instances.
Committed capacity before on-demand
Committed hours are billed whether they are used or not, so they come first. On-demand instances start only when the reserved capacity is full.
Multi-node training on HPC or Kubernetes
Large distributed tasks go to the HPC cluster first, where its scheduler places all the nodes together, and to multi-node scheduling on Kubernetes when the cluster is busy.
Interactive sessions near the data
Sessions stay on the pool next to the data and the shared storage, on a memory-capped share of a GPU.
A compute resource per GPU class
Each GPU class gets its own compute resource, so a task asks for the class it needs and lands on any pool that has it.
Data that stays in a region
The project's profiles list only pools in the permitted region, so its work cannot be placed elsewhere. Its output destination is set to storage in that region; by default outputs go to the control plane's file server. Code comes from git or the image, secrets come from vaults and are injected at start, and logs and metrics stream home. The Cloud Autoscaler reference architecture covers where the work lives.
GPU sharing on any hardware
A GPU can be shared between tasks in several ways, depending on the hardware and the environment. Shares are counted in the pool like whole GPUs, so a task on a share of a GPU uses that share of the group's reservation.
| Method | Isolation | Where |
|---|---|---|
| Static MIG | Hardware slices in a fixed layout; tasks request a slice | NVIDIA GPUs with MIG, on Kubernetes |
| Dynamic MIG | Hardware slices created per task; the GPU is repartitioned as demand changes | NVIDIA GPUs with MIG, on Kubernetes |
| Memory-capped containers | GPU memory capped per container; compute shared by time slicing | Any NVIDIA GPU: on Kubernetes, and on bare metal and VMs in Docker mode |
| AMD partitions | Compute and memory partitions set on the GPU | AMD Instinct GPUs |
| CPU offload for model serving | Idle models move to CPU memory, so one GPU serves more models | Model endpoints on vLLM, SGLang and llama.cpp |
Multi-node and HPC work
On HPC clusters
The batch template can take the node count and other directives from the task. The HPC submitter submits it as a batch job with the cluster's own command, and the task runs natively or in a container on the nodes the scheduler allocates.
As the user's own account
On Slurm, the HPC submitter's user map (agent.slurm.user_map) runs each batch job as the Linux account mapped to the Klique user who submitted it. A user with no mapping is refused unless a default account is set. PBS and LSF follow the same submitter pattern, an HPC submitter on the login node with a batch template. The GPU as a Service reference architecture covers HPC user mapping and sessions.
On Kubernetes
Multi-node scheduling runs one task across pods on several nodes. The task states the GPUs per node and the number of nodes.
Slurm on Kubernetes, a clustering service
Klique can run a Slurm cluster as pods on Kubernetes, with a login pod and worker pods that scale with demand. Existing sbatch scripts work unchanged.
Kubernetes in Kubernetes (K3k), a clustering service
Klique can run a virtual Kubernetes cluster inside a host cluster, so a team or tenant gets its own Kubernetes API and namespaces. Its pods run on the host cluster's nodes.
What Klique adds to an HPC cluster
Queue visibility and reordering, environment provisioning, experiment tracking and GPU sharing. Users submit through Klique instead of logging in to the cluster.
Layers above the scheduler
Scheduling GPUs on a Kubernetes cluster is one layer. Klique covers that layer in every environment and adds governance and services above it, on the same platform and with the same identity.
Scheduling across environments
Kubernetes, Slurm, PBS, LSF, bare metal, VMs, public cloud providers and neoclouds behind the same compute resources, with ordered pools and GPU sharing on NVIDIA and AMD hardware.
Identity and access
Single sign-on with the company's identity provider, RBAC on projects, queues and apps, service accounts for automation, and vaults for secrets injected into tasks.
Tenants
One control plane can serve many isolated tenants, such as business units or the customers of a provider, each with its own users, projects, compute resources, policies and worker daemons. The provider sets each tenant's share, and the Platform Management Center shows compute, usage and cost across tenants. The Multi-Tenant reference architecture covers this in detail.
Self-serve services
An app catalogue with remote IDEs, model serving and custom apps, and clustering services for teams that want their own Slurm or Kubernetes cluster, all on governed compute and reached through internal routes.
Tokens and models
An LLM gateway in front of self-hosted and public models, with routing rules, token budgets and wallets, and guardrails. Its self-hosted models run on the same GPU pools this scheduler manages. See the LLM Gateway and Tokenomics reference architecture.
Development lifecycle
Experiment tracking, data versioning, pipelines and a model registry, recorded for every task whichever environment ran it.
Metering and billing
Klique meters GPU-hours, tokens and storage per tenant, project and user, and exports the usage to the billing or chargeback system.
Agent access and secure sandboxes
An MCP server, so AI agents can submit and manage work with the identity, quota and metering of the user they act for. Coding agents run in sessions: credentials come from the platform's vaults, and model calls go through the LLM gateway, where a user's gateway key is their own Klique token. Compute and tokens are governed in one place, under the same identity and budgets.
Visibility and metering
Orchestration dashboard
Available and used GPUs, CPUs and workers in every environment, grouped by category and group, with utilization over time and an event log.
Policy view for users
Users see the policies that apply to them: current usage, reservation, limit and the compute resources available to them.
Across tenants
The Platform Management Center shows orchestration, usage and cost for every tenant in the deployment.
Metering
GPU-hours are recorded per user, project and pool, by GPU class. Klique meters usage per tenant and exports it to the billing or chargeback system.
Compared with a cluster scheduler
A GPU scheduler for one Kubernetes cluster and a meta scheduler solve different problems. The left column describes a typical Kubernetes GPU scheduler; check a specific product's documentation for its details.
| Capability | A GPU scheduler on one Kubernetes cluster | Klique |
|---|---|---|
| Where work runs | Pods on one Kubernetes cluster | Kubernetes clusters, Slurm, PBS and LSF clusters, bare metal, VMs, public cloud providers and neoclouds |
| When a cluster is full | Work waits in the cluster | Work goes to the next pool in the profile's order, including rented capacity |
| GPU sharing | Fractions on the cluster's GPUs | Static and dynamic MIG, memory-capped containers and AMD partitions |
| Quotas | Per project, in the cluster | Reservation and limit per group, held across every pool |
| Identity | Cluster accounts and roles | Single sign-on, RBAC, service accounts and identities for agents |
| Model tokens | Not covered | LLM gateway with routing rules, budgets and guardrails |
| Billing | Not covered | GPU-hours, tokens and storage metered per tenant, project and user, and exported to the billing or chargeback system |
| Services for users | Workloads on the cluster | App catalogue, remote IDEs, coding agents, model serving and experiment tracking |
Design considerations
Match each agent to its profiles
The resource manager counts GPUs; the agent applies them. Set GPU limits in the pod or batch template to match each profile, or tasks are counted correctly but run with the wrong allocation.
Size pools to the hardware
Count each pool in whole GPUs, whatever the mix of whole, fractional and MIG work it runs. The scheduler treats a pool as full when its declared capacity is used, so keep the declared capacity equal to the real GPUs.
Keep reservations within capacity
The sum of reservations on a pool should not exceed what the pool holds, or a reservation cannot always be met.
Order pools by cost and locality
Owned capacity first, committed capacity next, on-demand last. Keep interactive sessions near the data.
Plan for preemption
Borrowed work can be preempted when a group below its reservation needs capacity. Preempted work goes back to the front of its queue. Long tasks should save checkpoints.
Map HPC users
On Slurm, map Klique users to Linux accounts in the user map so batch jobs run as the person who submitted them, and set a default account if unmapped users should not be refused.
Set cloud limits
Give each autoscaler instance an instance type, a maximum instance count and an idle timeout. A cloud pool with no running instances still takes work, because the autoscaler registers as its listener and scales from zero. The Cloud Autoscaler reference architecture covers these settings.
Allow outbound only
Agents connect out to the control plane over HTTPS; no environment needs inbound ports from it. On HPC clusters, allow outbound HTTPS from the compute nodes as well as the login node.
Glossary
- Meta scheduler
- A scheduler that decides which environment runs a task, above the schedulers that place it on nodes.
- Compute resource
- What a user submits work to, named for the allocation it gives, such as multi-node training or a shared GPU: a queue, one per policy and profile. IT decides which pools serve it.
- Access channel
- A way to submit work: the UI and its apps and workflows, the SDK and CLI, the REST API, AI agents over MCP, or CI/CD pipelines.
- Pool
- Capacity in one environment, counted in whole GPUs.
- Resource profile
- A task size and an ordered list of pools.
- Resource policy
- A user group's reservation and limit, and the profiles it may use.
- Reservation
- The GPUs a group is guaranteed.
- Limit
- The most a group can use at once, including idle capacity it borrows.
- Klique worker daemon
- The agent that pulls tasks for a pool and runs them on Kubernetes, bare metal, VMs or a neocloud. The Klique documentation calls it the Klique Orchestrator.
- HPC submitter
- The agent on an HPC cluster's login node that turns tasks into batch jobs for Slurm, PBS or LSF.
- Apps worker daemon
- The agent that runs app controllers, including cloud autoscalers.
- Autoscaler
- Starts cloud instances for queued work in the customer's account, of the instance type it sets and up to its maximum instances, and stops them after an idle timeout.
- Clustering services
- Slurm on Kubernetes, and Kubernetes in Kubernetes (K3k): clusters Klique runs on a Kubernetes cluster for teams that want their own scheduler or Kubernetes API.
- Neocloud
- A provider that rents GPU capacity, usually as Kubernetes clusters or VMs.