Many ways to provision, one compute resource
Compute is provisioned in many ways: from the app catalogue, in an attended session, from a script through the SDK or CLI, by a pipeline or schedule, or by an agent over MCP. Each one ends at the same place, a compute resource. It's the one choice made at provisioning time, and it carries no information about where the work will run. The GPU as a Service Reference Architecture covers each way to provision in more detail.
Launch an app
From the UI, for consumers and builders: pick an approved app (vLLM, SGLang or NVIDIA NIM for serving, JupyterLab or VS Code for development, Claude Code for agentic work) and choose a compute resource from a dropdown. The form never mentions a cluster, a node or a container.
Interactive sessions
Interactive, for builders: VS Code, Jupyter, SSH or a remote desktop, opened on platform GPUs rather than a laptop. A fraction of a GPU is often enough, and the session stays on it until the user closes it.
Clone & enqueue
Non-interactive, for builders: take any past run, change a parameter in the UI and send it to a compute resource. It's the same code with a different config. It runs unattended and reports back as it goes.
From an existing script
From the SDK: two lines to track a script, one more to hand it to the pool. The run stops locally and resumes on the GPU the compute resource provides.
from clearml import Task task = Task.init(project_name="fraud-models", task_name="train") task.execute_remotely(queue_name="compute resource") # continues on the pool
Any script, no code change
From the CLI: wrap an existing repo or script and send it to a compute resource straight from the terminal or a CI pipeline.
clearml-task --project fraud-models --name nightly \ --repo git@git.corp:ml/fraud.git --script train.py \ --queue "compute resource"
Pipelines, HPO & schedules
Automation: each pipeline step, each hyperparameter trial and each scheduled or triggered run names its own compute resource. Prep on CPU, train on several GPUs, evaluate on a slice.
@PipelineDecorator.component(execution_queue="compute resource") def train(dataset_id): ...
An agent, over MCP
Agentic: the platform is also a tool an agent can call. From Claude Code, a desktop assistant or a team's own agent, it can launch an app, enqueue a run, or read back a run's results. It works under the same identity, policy and attribution as the person who owns it. The agent never gets its own route to the hardware.
A cloud burst, step by step
Follow one training task. The on-prem GPUs are all busy, so the task overflows to the cloud region. The timeline shows each component's part and its typical duration.
Scheduler
Behind every compute resource sits a small scheduler that runs about once a second. It works in three passes, and the order drives hybrid placement:
- Fill every team's reservation first. Try each profile's first pool, then its second, then the rest.
- Make room if a guarantee is being missed. If a team under its reservation is waiting and the pools are full, work from teams running over theirs is stopped and put back at the front of its queue.
- Then hand out the spare, up to each team's limit, again first pool before second.
“Full” means the pool's declared capacity, the number IT enters, is used up. When the scheduler picks a pool, it moves the task into a hidden queue that only that pool's worker daemons listen on. A cloud pool with no nodes yet still counts, because its autoscaler registers as the listener on that queue. This lets a pool scale from zero: the work is admitted first, and the node that runs it arrives next.
The same scheduler places work across Kubernetes, Slurm, bare-metal and cloud pools, covered in the Meta-Scheduler Reference Architecture.
When it runs in the cloud, where does the work live?
Every security and data team asks this question. The control plane holds the record of the work, not the work itself. Heavy bytes (data, checkpoints, weights) stay beside the GPUs that use them. The data that comes home is small: parameters, logs, metrics, lineage, and pointers to where the outputs were written.
| Asset | Location during a cloud run | Crosses back? |
|---|---|---|
| Task record | Created on the control plane when the task is defined. It holds the repo, commit and uncommitted diff, parameters, packages, and who ran it. | Lives on-prem |
| Code | Cloned onto the node from the git server at start, and any uncommitted changes recorded on the task are applied. Or it's already baked into the container image. The node needs a route and credentials to the git server. | No, stays in the region |
| Environment | Rebuilt on the node from the recorded packages, or taken from the image, then cached for the next task on that node. | No, stays in the region |
| Container image | Pulled from a registry. Point cloud pools at an in-region mirror to cut cold starts and egress. | No, stays in the region |
| Input data | A versioned dataset: metadata on the control plane, files in storage. It's fetched and cached on the node. For heavy data, store the dataset in an in-region bucket, or tell cloud workers to read the registered location from a local mirror. | No, stays in the region |
| Secrets | Held in vaults on the control plane and injected into the task at start. They're never baked into images or code. | No, flows out only |
| Outputs | Checkpoints, artifacts and models go to the task's output destination. The default is the control plane's file server, back across the WAN. Set it per project or per pool to an in-region bucket to keep bytes in the cloud, or to on-prem storage for anything that must come home. The control plane records where they went. | Depends on the output destination |
| Model weights | For serving, downloaded at start from the model registry or Hugging Face. Put the cache on a persistent volume, or every cold node downloads them again. | No, stays in the region |
| Logs & metrics | Streamed to the control plane while the task runs, so a cloud task can be followed exactly like an on-prem one. Anything a task prints comes home too, so a task should never print secrets or raw records. | Yes, streamed home |
| A model endpoint | Runs on the cloud node, on a static route behind that site's internal routes. The token traffic it serves is metered centrally (section 07). | Usage records only |
| Scratch disk | Local to the node. Anything not written to an output destination is lost when the node is removed. | No, removed at scale-down |
Tokens: one gateway for every model
GPUs are one way to consume AI. Tokens are the other. Every model call, to a self-hosted model or to a public API, goes through one endpoint, the LLM gateway. The gateway checks who is calling, whether they're allowed, and whether their team has budget left. It then chooses a model from rules IT has set, and records the cost against the user and project. In this scenario the gateway runs on-prem, so every call enters through the customer's data centre, including calls to extra model capacity in the cloud region or to a public API.
chat, not a vendor model. IT can move that alias from a public API to a model self-hosted on-prem, or to cloud capacity, with no change on the client side.Where a token request goes, case by case
| The model runs… | Path of the request | Metered | Good for |
|---|---|---|---|
| On-prem GPUs | Client → gateway → a self-hosted model server on owned pools. The request stays in the building. | On-prem, at the internal rate | Routine and sensitive traffic, the default |
| Cloud GPUs | Client → LLM gateway → WAN → the cloud site's internal routes → the model server's static route in-region | On-prem, same ledger | Extra serving capacity, users near that region |
| Public API | Client → gateway → provider. The gateway swaps the user's identity for the provider key it holds. | On-prem, at the provider's price | Frontier quality when a rule or a team's entitlement calls for it |
Putting a self-hosted model in front of callers follows the same order on-prem or in the cloud region. The platform team creates a static route, a fixed address that outlives any one endpoint. The serving app is launched against that route, so the endpoint it stands up answers there. The route is then registered with the LLM gateway under an alias, and the alias is what applications call.
Coding agents and desktop tools that talk to a provider directly can be pointed at the same endpoint, so their tokens land on the same ledger instead of on someone's personal provider account.
Routing rules, guardrails and budgets sit behind that alias. The LLM Gateway and Tokenomics Reference Architecture covers them.
Compute policy engine
The compute policy engine keeps owned GPUs busy first and spends cloud capacity on purpose. It has two sets of controls, one for GPUs and one for tokens, and both write to the same ledger.
GPU controls
Policies give each team a reserved amount and a limit, over the compute resources they're allowed to use. When several teams want the same GPUs, execution priority decides who goes first. Profiles decide the size of the allocation: a GPU fraction, a whole GPU, or several GPUs on one node. Give each compute resource a usage weight, so a fractional GPU counts as that fraction of a GPU-hour.
Token controls
A budget per project, daily, weekly or monthly, ties spend to an outcome with an owner. Per-user limits stop one person running away with it. Rate limits protect shared models. Guardrails are set in the global policy or per AI resource: deny and allow patterns at the gateway's edge, and the NIM Guardrails app for a model-based check of input and output.
Placement patterns
The scenario here is the first pattern below: on-prem first, cloud for overflow. The same building blocks support other strategies. Each one is a profile's ordered pool links, so changing strategy means reordering links, not moving workloads.
On-prem first, cloud for overflow
The default for most customers. Owned GPUs are always used first, and the cloud absorbs peaks such as quarter-end retrains or a new model launch.
Burn down a cloud commitment
For a committed-spend agreement that has to be used up, or for moving off the cloud gradually as on-prem capacity arrives. Flip the order back later.
Specify capability, not a GPU model
A profile can specify GPU memory rather than a GPU model. When the newest GPUs are busy, an older one with enough memory still qualifies, and cloud is the last resort.
Train on HPC, serve on Kubernetes
Training runs that belong on the cluster don't have to wait for it. The endpoint that serves the result launches from the same catalogue onto Kubernetes.
Sovereign by policy
For EU projects only. A regulated project can only reach in-country pools, and its outputs are written to in-country storage. Only metadata reaches the control plane. A second site in another jurisdiction is just another pool with its own worker daemons: its models and data stay there, and only reports come home.
Design considerations for cloud burst
Decide these on purpose before the first burst, rather than finding them out in production.
Cold start is minutes, not seconds
Node boot plus image pull dominates, then model weights. For latency-sensitive work, keep a small warm minimum, pre-bake images, mirror the registry in each region, and put weight and package caches on persistent volumes.
Move compute to data, not data to compute
Egress is billed per gigabyte and large datasets are slow to move. Keep a versioned copy in each burst region, and let the dataset version pin exactly which bytes were used.
Set the output destination per pool
By default every checkpoint travels home to the file server, and each one is billed as egress. Cloud scratch disk disappears at scale-down. Point cloud pools at an in-region bucket, and checkpoint long runs so preemption or spot reclaim costs minutes, not hours.
Tune the idle window
Short windows save money but cause more cold starts. Long windows do the reverse. Set it per pool: short for bursty training, longer for serving and interactive pools.
Reachability runs one way
Cloud worker daemons need outbound HTTPS to the control plane's API and file server (over VPN, private link or a published endpoint), and to the git server, registry and package mirrors. Batch work needs no inbound connection into the cloud. A site that serves endpoints or sessions needs its own internal routes and ingress. Those routes are the only inbound path to a site.
Mind the WAN hop for tokens
Calls to a cloud-hosted model go through the one LLM gateway. Put that gateway near the heaviest callers, and keep chatty, latency-sensitive models close to them.
Keep location out of the name
Users only ever see a compute resource. Its name describes the allocation, never the location. That way IT can reorder pools, add a region or change clouds without anyone changing their code or habits. Offer users only these compute resources, so every task goes through policy.
Keep declared capacity accurate
The scheduler overflows once a pool's declared capacity is used, not by probing the hardware. Keep those numbers true, and retire any compute resource no node serves, or work will wait on it.
Secrets come from vaults, not images
Inject credentials at run time from group vaults. Never bake keys into an image or pass them as a launch parameter, where they would be recorded on the task for anyone who can read it.
Budget at two levels
GPU policies (reserved, limit, priority) and token budgets (project, user, rate) cover different risks. Cloud spend also has hard ceilings: the autoscaler instance's maximum instances, and the resource policy's limit on the cloud compute resource. Set both levels, and review them against the same per-project ledger.
When something fails
When spot capacity is unavailable, the autoscaler falls back to on-demand. When the cloud pool has no VM free, work waits on the compute resource until the autoscaler adds one, up to its maximum instances. If the apps worker daemon that runs the autoscaler is down, no new VMs are added until it is back. Long runs on spot should save checkpoints, so they can resume.
Connections
This table gathers the connections a cloud burst needs. Users sign in through SSO, and agents over MCP through OAuth. Worker daemons authenticate with the credentials they were registered with.
| From | To | Direction | Purpose |
|---|---|---|---|
| Users, apps and agents | Control plane: UI, API, MCP | Into the control plane, HTTPS | Send work to compute resources, manage it, read results |
| Worker daemons, on-prem and cloud | Control plane API and file server | Outbound HTTPS from the site | Pull work, stream logs and metrics, upload outputs when the output destination is the file server |
| Autoscaler | Cloud API | Outbound, from the VPC or from on-prem | Run, describe and terminate instances, tag them, request spot |
| Users | The site's internal routes | Into the site: the only inbound path to a site | Reach sessions and endpoints running there |
| LLM gateway | Cloud-site models and public model APIs | Outbound from on-prem | Route model calls. Provider keys stay in the gateway |
| GPU nodes | Git server, registry and package mirrors | Outbound from the node | Clone code, pull images and packages |
Glossary
- Control plane
- The Klique server: UI, API and MCP, compute resources and scheduler, policies, identity, vaults, and the metadata and usage ledger. It holds intent and records; the data stays where the work runs.
- Klique worker daemon
- The small worker process on every GPU node, whether VM, bare metal, Kubernetes or Slurm. It polls the control plane for work over outbound HTTPS, runs it, and reports back.
- Compute resource
- The target a user sends work to, from any provisioning method. Under the hood it's a queue. It's the only piece of this architecture a user ever sees.
- Resource pool
- A set of nodes running worker daemons in one place: an on-prem cluster, a Slurm partition, or an autoscaled cloud account.
- Profile
- The allocation a task receives (a GPU fraction, a whole GPU or several GPUs), linked to one or more pools in priority order.
- Resource policy
- Which teams may use which compute resources, with a reserved amount, a limit, and an execution priority.
- Autoscaler instance
- The autoscaler's definition, stored in the control plane: the compute resources to poll, instance types, limits, idle timeout and credentials.
- Autoscaler
- The running process that creates GPU VMs when work waits on the compute resource it polls, and removes them when they go idle. On Kubernetes, the cloud's own node autoscaler takes this role.
- Apps worker daemon
- The component that runs autoscalers and app controllers. By default it sits beside the control plane. It can also run on a small instance in the customer's VPC with an IAM role.
- Internal routes
- The routes to sessions and endpoints at each serving site, with access control. A self-hosted model's fixed address on them is a static route.
- LLM gateway
- The single endpoint for every model call. It handles identity, access, guardrails, budgets, routing rules, and metering.
- Alias
- A model name that callers use, such as
chatorembed, that IT can point at any model without clients changing. - Output destination
- Where a task writes checkpoints, artifacts and models: an in-region bucket or on-prem storage. The control plane stores the pointer.