By Adam Wolf
Fractional GPU allocation is the practice of splitting one physical GPU among multiple workloads, each with its own allocated share of compute and memory. It exists because of an uncomfortable arithmetic: most AI workloads do not need a whole accelerator, most accelerators sit idle most of the time, and the accelerators in question are the most expensive line item in the data center. This post explains how fractional GPUs work at the hardware and scheduler level, the real differences between static and dynamic approaches, where each one wins, and how to think about the utilization math that justifies the whole exercise.
Why fractionalize a GPU at all?
You fractionalize a GPU because whole-GPU allocation wastes most of the card, and the waste is now measured rather than suspected. Cast AI telemetry across tens of thousands of Kubernetes clusters places average GPU utilization at roughly 5%, meaning organizations pay for something like twenty times the capacity they consume.
The waste has a specific shape. It is not that nobody wants GPUs; it is that GPU demand is granular and allocation is not.
- A data scientist's notebook session needs 8 GB of memory and occasional bursts of compute.
- A development environment needs the CUDA stack present but spends most of its life waiting on a human.
- A low-traffic internal inference endpoint needs milliseconds of compute per request, thousands of times a day.
Park any of these on an 80 GB card with exclusive access and the arithmetic is brutal: the notebook strands 90% of the memory and nearly all of the compute, and it does so for the entire lifetime of the session, including nights and weekends.
The classical answer was queueing discipline: make people share by taking turns. But turn-taking is exactly wrong for interactive workloads, where the human needs the environment now and idles it later. Sharing GPU capacity inverts the model: instead of one workload at a time using a little of the card, several workloads at a time each use the slice they asked for.
What is static GPU partitioning?
Static partitioning divides a GPU into fixed slices before workloads arrive, with the isolation enforced by the hardware itself.
The reference implementation is NVIDIA’s Multi-Instance GPU (MIG), available on the A100 and later data center generations, which partitions a single physical GPU into as many as seven isolated instances. Each instance gets dedicated streaming multiprocessors, dedicated memory with its own bandwidth allocation, and fault isolation, so a crash or an out-of-memory event in one instance does not touch its neighbors. From the workload’s point of view, a MIG instance simply is a GPU: smaller, but private and predictable. AMD offers the same capability on its CDNA 3 and CDNA 4 data center accelerators, the MI300X and MI355X generations, where compute partitioning splits the accelerator dies into 1, 2, 4, or 8 partitions through the SPX, DPX, QPX, and CPX modes, and memory can be presented either as one uniform space or as four NUMA domains. Supported modes differ by generation on both vendors.
Static partitioning has two properties that matter operationally. First, predictability: because the virtual boundaries are hardware-enforced, a workload in one instance cannot degrade the latency of a workload in another, which is precisely what production inference needs. Second, rigidity: the partition layout is set ahead of time, and changing it generally requires draining the workloads on the card first. The constraint is specific rather than a matter of caution, because a GPU’s MIG geometry can only be changed while the device is free, so repartitioning means evicting whatever is running, reconfiguring the partitions, and then rescheduling the work. If your demand arrives in shapes that do not match the configured profile, static partitions strand capacity in smaller pieces instead of large ones, an improvement, but not a solution.
What is dynamic partitioning and memory-based sharing?
Dynamic approaches make the slicing decision at scheduling time instead of configuration time, sized to what the workload requested rather than to a fixed hardware profile.
The most broadly useful form is container-level memory limits: memory is hard-capped per container, several workloads co-reside on the card within their caps, and the GPU’s compute is shared across the containers through time slicing. This matches the actual demand profile of interactive work almost perfectly. Ten notebook sessions, each capped at 8 GB on an 80 GB card, fit simultaneously, and because human-driven sessions burst rather than sustain, the time-shared compute is usually there when any one of them needs it.
The honest trade-off is that time-shared compute is a statistical guarantee, not a hardware one. If three of those ten notebooks all launch heavy training cells at the same moment, they contend. That is why dynamic sharing belongs to workload classes that tolerate contention (development, experimentation, spiky low-volume endpoints) and why sustained heavy jobs belong on dedicated fractions or whole cards.
A second practical consideration is failure behavior: memory limits are enforceable, but co-resident workloads share more of the card’s fate than MIG instances do. This is another reason production serving tends toward hardware isolation.
The two approaches are complements, not competitors. Static partitioning turns one card into several predictable small ones. Dynamic sharing packs opportunistic work into whatever capacity demand leaves behind.
Whole-GPU allocation strands most of the card for the life of the session. Static partitioning turns one card into several predictable small ones. Container-level memory limits pack capped workloads onto the card and time-share the compute between them.
What does the utilization math actually look like?
GPU fractioning can return a large share of a cluster to the schedulable pool without slowing any workload down, because it eliminates idle reservations, not idle compute. Take a concrete, deliberately conservative example. A 100-GPU cluster of 80 GB cards runs a mixed estate: 30 cards on training jobs that genuinely saturate them, 20 on production inference, and 50 effectively assigned to interactive work, notebooks, development environments, and experiments, each holding a whole card.
If those 50 interactive cards follow the industry pattern, they run at single-digit utilization. Suppose fractioning GPUs lets you consolidate that interactive demand at an average of five workloads per card, a modest packing factor given 8 to 16 GB requests on 80 GB cards. The same interactive demand now occupies 10 cards instead of 50. Forty cards return to the schedulable pool: more training throughput, more inference headroom, or deferred procurement, whichever the business needs. No workload got slower; the change is that idle reservations stopped masquerading as consumption.
That is the general shape of the fractioning payoff: it does not make any single job faster; it makes the cluster’s stranded capacity visible and schedulable again. The Cast AI number says the stranded fraction is the majority of the fleet. Whether your own fleet looks like the example is a measurement question rather than an argument, and the orchestration resource dashboard is where it gets answered: it reports total and idle GPUs, CPUs, and workers for a tenant, alongside utilization over time and a resource event log.
The orchestration resource dashboard reports total and idle GPUs, CPUs, and workers for a tenant, with utilization over time and a resource event log. Idle capacity you can see is capacity you can reclaim.
When should you use which?
A practical placement rule by workload class:
- Whole GPUs for large training and fine-tuning jobs that saturate the card. Fractioning has nothing to offer a job that already uses everything.
- Static partitions for production inference and any workload where latency isolation is a requirement rather than a preference. Hardware-enforced boundaries are what let you write a service-level objective (SLO) you can keep, a subject we treated in depth in AI Workload SLOs.
- Dynamic partitions and memory-based sharing for notebooks, development environments, experiments, CI jobs, and spiky low-volume endpoints, the workloads whose duty cycle is mostly idle.
The critical operational point: the placement decision should belong to the scheduler and its policy, not to the end user. Users ask for what a job needs; quotas, reservations, and workload class determine what form of allocation satisfies it. The moment users hand-pick whole GPUs “to be safe,” you are back to stranded capacity with extra steps.
How does Klique implement fractional GPUs?
Klique’s Orchestration Engine supports static and dynamic GPU partitioning as well as memory-based sharing, across both NVIDIA and AMD accelerators. They are assigned through the same queue and quota model that governs whole-GPU workloads. A fraction is not a special case; it is one more thing a queue can grant. Resource policies set a reservation for each user group, the amount of compute guaranteed to be available to it, and a limit, the maximum its jobs can use concurrently. Administrators expose that compute through execution queues, each tied to a resource profile that defines how much compute a job receives and which resource pools it draws from. The same machinery applies whether the unit being scheduled is a whole card, a MIG instance, or an 8 GB memory-capped share.
A resource policy sets the compute guaranteed to a group and the maximum its jobs can use concurrently. A fraction is granted by the same policy that grants a whole card, which is what keeps utilization a managed number rather than a periodic archaeology project.
Two aspects are worth underlining for platform teams. First, you do not need to use Kubernetes to schedule a fractional GPU partition. The same queues schedule across Kubernetes, Slurm, PBS, VMs, and bare metal, so an organization with HPC-style clusters is not excluded from the utilization gains. The scheduler integration is documented for Slurm and PBS, where Klique maps its queues to the scheduler’s batch script templates and submits tasks as scheduler jobs, so jobs do not have to be rewritten. Second, because every fraction is requested through governed queues under a real identity, consumption is attributable. The platform can tell you which team’s notebook and IDE sessions occupy which slices, which is what makes utilization a managed number rather than a periodic archaeology project.
Fractioning is not the only lever on models-per-GPU, and the two are worth knowing apart. Fractioning divides a card between workloads that are running. Unified Memory Technology addresses the case where more models are deployed than VRAM can hold at once: active models live in GPU VRAM, models not currently serving requests are offloaded to system RAM, which is typically 10 to 100 times larger, or to disk when configured, and they load back in on demand. Swapping is dynamic, so there is no need to predefine which models stay resident. For an inference estate serving many low-traffic models, that is often the larger win, and it composes with fractional allocation rather than competing with it.
One caveat belongs in every GPU fractioning conversation: supported partitioning modes vary by GPU vendor, generation, and driver stack. The Fractional GPUs documentation lists the modes the platform supports, and you should validate the specific hardware matrix for your fleet with your Klique team before standardizing on a configuration.
If you would like to see how fractional GPU allocation could raise utilization on your own clusters, request a demo.
FAQ
What is a fractional GPU?
A fractional GPU is a slice of one physical accelerator, with its own memory allocation and compute share, scheduled to a workload as if it were a smaller dedicated GPU.
What is the difference between static and dynamic partitioning?
Static partitioning fixes hardware-isolated slice sizes ahead of time, as NVIDIA MIG does with up to seven instances per card. Dynamic partitioning sizes fractions at scheduling time to match each workload’s request, typically enforcing memory limits while time-sharing compute.
Which workloads benefit most from GPU fractioning?
Interactive and bursty workloads: notebooks, development environments, experiments, and low-traffic inference endpoints, since they rarely need a full card and idle most of the time.
Does fractioning make individual jobs faster?
No. It recovers stranded capacity, letting the same fleet serve more workloads. Jobs that saturate a whole GPU should keep a whole GPU.
Does fractioning work outside Kubernetes?
On Klique, yes. Fractional allocation runs through the same queues that schedule Slurm, PBS, VM, and bare-metal compute, and the Slurm and PBS integrations are documented separately.
