GPU as a service reference architecture

Interactive and non-interactive GPU requests

Everything a GPU does here is one of two kinds. Non-interactive work runs to completion and gives the GPU back: training, fine-tuning, batch inference, optimisation trials, pipeline steps. Interactive work holds a GPU while someone, or something, uses it: notebooks, IDEs, coding agents, SSH sessions and model endpoints. Every provisioning path creates one or the other; the estate underneath decides how it is stood up.

Kubernetes architecture

On Kubernetes the worker daemon runs as a controller in its own namespace, scoped to it by default. It never runs the work itself: for every task it renders a pod spec from a base template plus the compute resource's overrides, and the Kubernetes API does the rest. Whole GPUs, MIG slices and memory-capped fractions are all just different specs.

Bare-metal architecture

On bare metal, or VMs with GPU passthrough, the worker daemon runs on each GPU host, and how a task runs depends on the mode it was started in. In Docker mode it runs the task's container and executes the task inside it. In virtual environment mode it builds a Python environment on the host, installs the task's packages, clones the code, applies any uncommitted changes and runs the task natively, reusing cached environments between tasks. Either way the work lands on the GPUs the worker daemon assigns. One worker daemon can own all of a host's GPUs and hand each task the number its compute resource asks for, or one worker daemon can own each GPU set. Fractions need Docker mode: memory-capped containers, down to an eighth of a GPU.

HPC architecture: Slurm, PBS and LSF

On an HPC cluster, Klique works through the cluster's own scheduler rather than around it. A submitter on the cluster's login node turns each task into a batch job, using a batch template tied to the compute resource, and submits it with sbatch on Slurm. On Slurm, a user map (agent.slurm.user_map) submits each batch job as the user's own Linux account. PBS and LSF follow the same submitter pattern: a batch template, submitted with qsub or bsub. The scheduler places it; inside the job, a worker daemon builds the environment and runs the task, natively or in a Singularity, Apptainer or Pyxis/enroot container.

A mixed estate

Most estates are a mix: Kubernetes for sessions and endpoints, dedicated hosts for heavy training, an HPC cluster (Slurm, PBS or LSF) for multi-node runs, and the cloud for peaks. One control plane serves them all. Each compute resource lists its pools in order, so placement is a decision the platform team makes once. The Meta-Scheduler reference architecture shows how the scheduler places work across estates.

The app catalogue, by how it uses a GPU

The catalogue is the fastest path: a form in the UI, and the same call in the API or over MCP. Grouped by how each app consumes a GPU, which is what matters for planning pools and policies:

Every launch form exports and imports as JSON, so a team's standard session or endpoint can be kept in git and relaunched the same way.

GPU provisioning paths

The catalogue suits point and click. The other paths suit code, portals, agents and pipelines. All of them create the same objects, so each lands on the same compute resources, policies and ledger.

SDK and CLI

Two lines track a script; one more hands it to a GPU. clearml-task wraps any repository without changing its code.

from clearml import Task
task = Task.init(project_name='AI Engineering', task_name='train')
task.execute_remotely(queue_name='compute resource')

REST API

Every other path is built on the API. Internal portals, platforms and automation call it directly with a service user's keys: to create and enqueue tasks, launch catalogue apps, and read usage.

MCP

The platform ships an MCP server that fronts the API. Agents such as Claude Code connect to it over HTTPS with OAuth, and every tool call runs under the user's own identity and permissions: the same compute resources, policy and ledger as the UI. Registering it takes one command, claude mcp add --transport http klique https://<host>/mcp.

App launches can be refused, by an admin lock or the company's running-apps quota, and the agent gets a readable reason rather than a raw failure.

GitOps

Training belongs in the same loop as the code. A push or a pull request starts a CI job in GitHub Actions or GitLab CI; the job uses clearml-task or the SDK, with API keys held as CI secrets, to run the work on a compute resource. ClearML's own GitHub actions cover the common steps: train a model on a comment, post a run's metrics to the pull request, and flag a model that got worse.

Experiments and pipelines

Whichever path starts a run, the platform records it the same way: the code and uncommitted changes, packages, parameters, console output, metrics, models and artifacts. That record makes any run reproducible and comparable, and it is the record an agent reads when it compares runs over MCP.

A pipeline chains runs. Each step names its own compute resource, so preparation lands on CPUs, training on many GPUs and evaluation on a slice, and outputs pass between steps as artifacts.

@PipelineDecorator.component(execution_queue='CPU')
def prepare(url): ...

@PipelineDecorator.component(execution_queue='multi-node GPUs')
def train(data): ...

@PipelineDecorator.component(execution_queue='GPU slice')
def evaluate(model): ...

Where the work lives

Model serving

For most application teams, GPU as a service means a model endpoint. The serving apps (vLLM, SGLang and NVIDIA NIM) take a model from the registry or from Hugging Face by name, request the compute resource, and set the number of GPUs for tensor parallelism. The endpoint is OpenAI-compatible and sits behind an authenticated internal static route.

Putting that endpoint in front of callers follows the same order each time. The platform team creates a static route, a fixed address that outlives any one endpoint. The serving app is launched against that route, so the endpoint it stands up answers there. The route is then registered with the LLM gateway under an alias, and the alias is what applications call. The model behind the route can be swapped, and a second instance added for capacity, with client code left as it is.

Routing rules, guardrails, budgets and token metering sit behind that alias; the LLM Gateway and Tokenomics reference architecture covers them.

One governed path underneath

Whatever the path, the same controls apply, because every path ends at a compute resource:

1. Compute resources are what people pick. Name them by what they give (a whole GPU, half a GPU, a node), never by where they run.
2. Resource policies decide which teams may use which compute resources, and give each team a reservation, a limit and a priority. The Meta-Scheduler reference architecture covers how the scheduler applies them.
3. Pools behind each compute resource are tried in order: on-prem first, cloud burst when full. The Cloud Autoscaler reference architecture shows how cloud capacity scales up from zero and back.
4. Metering records every run with an owner and a project. Sessions and endpoints also report their runtime, and model calls report tokens.

Identity comes from the company's single sign-on: OIDC, SAML or LDAP. Agents over MCP act under their user's identity, and CI acts as its own service user, with the same policy and attribution as anyone else. Every task carries its owner.

Design considerations

Decide these before opening up GPU access, rather than finding them out in production.

Glossary

Compute resource

What a user sends work to, from any path. It names a size, never a location. Under the hood it is a queue.

Profile

The allocation behind a compute resource (a fraction, a GPU or several GPUs, which can be specified by GPU memory) and its ordered pool links. Reordering the links changes placement for every path at once.

Resource pool

A set of nodes running worker daemons in one place: an on-prem Kubernetes cluster, bare-metal hosts, a Slurm partition or an autoscaled cloud account.

Resource policy

Which teams may use which compute resources, with a reservation, a limit and an execution priority.

Task

One tracked run: its code, environment, parameters, outputs and owner. Sessions and endpoints are tasks too.

App instance

A launched catalogue app, stored as a task of type application.

Apps worker daemon

Runs each app's controller, which creates the app's workload on the chosen compute resource.

Klique worker daemon

The worker in each pool that pulls work and stands it up: a pod on Kubernetes, a container on bare metal, a batch job on HPC.

Pod template

On Kubernetes, the base pod spec plus each compute resource's overrides, from which the worker daemon renders every pod.

Worker daemon on bare metal

The worker daemon process on a GPU host, running each task as a container in Docker mode or in a Python environment it builds on the host, on the GPUs it assigns.

HPC submitter

On an HPC cluster, the process on the login node that turns tasks into batch jobs for Slurm, PBS or LSF, one batch template per compute resource.

Internal routes

Authenticated routes to sessions and endpoints.

LLM gateway

One endpoint for model calls: aliases, routing, budgets and metering.

MCP server

The platform's Model Context Protocol server: the API as tools for agents, under the user's identity.

Pipeline

A chain of tasks, each on its own compute resource, run by a controller task.

Model registry

Registered models with their versions and the runs that produced them.