AI Workload SLOs: How to Define Reliability for Training, Fine-Tuning, and Inference

By Adam Wolf

Site reliability engineering settled the question of how to define reliability for web services a decade ago. Service Level Indicators (SLIs), Service Level Objectives (SLOs), error budgets, the whole framework. The question that has not been settled, and that platform teams are increasingly asked to answer, is what that framework looks like when the workload is a multi-week training job, a six-hour fine-tune, or a continuous inference fleet serving reasoning-mode token streams to people and agents alike. The answers are not the same as for a microservice. This piece looks at the SLOs that actually matter for each AI workload phase, the failure modes they exist to catch, and how Klique treats SLO enforcement as a first-class platform concern.

The SLO playbook needs an update

The classical SRE framework is built around three primitives. Service Level Indicators (SLIs) measure something specific. Service Level Objectives (SLOs) are the targets those indicators should hit. An SLO is an internal commitment; an SLA is a contractual one with penalties attached. Error budgets quantify how much SLO breach is acceptable in a given window, and they govern the trade-off between reliability and feature velocity. The model assumes a service that takes requests and returns responses, with availability and latency as the dominant metrics.

AI workloads break a lot of that. The most obvious break is at training time, where the workload is a multi-day or multi-week batch job with no requests and no responses, and where the dominant failure mode is hardware. Meta’s analysis of over 150 million A100 GPU hours across 4 million jobs found a mean time to failure of about 47.7 days for 8-GPU jobs and roughly 7.9 hours at 1,024 GPUs. The classical SLO machinery does not know what to do with a workload that, in practice, fails by definition every few hours and needs to recover gracefully without losing days of progress.

Inference workloads break the framework in a different way. They look more like traditional services, but the underlying compute is GPU-bound, the resource footprint is stochastic per-request, and latency has two components with entirely different bottlenecks: prefill is compute-bound and scales with input length, decode is memory-bandwidth-bound and scales with output length. Under continuous batching, decode latency for any one request depends on the composition of the batch it happens to share the GPU with. The shape of a classic latency SLO, “99% of calls finish within N milliseconds,” carries over to AI inference intact. The number and the metric do not. A web service has one meaningful latency; a streaming LLM endpoint has at least three, and time-to-first-token, inter-token latency, and total completion time behave differently and matter differently to the person or agent on the other end.

The right move is not to abandon the SRE framework. It is to translate it. AI workloads need SLOs. They just need different SLIs than a microservice does, and the platform underneath needs to expose those SLIs as first-class metrics. This is where Klique’s view is that the platform layer is not optional. Without it, SLOs become folklore.

SLOs for training

Training is a batch workload that runs on shared GPU capacity, contends with other jobs for scheduling, and fails for reasons that have nothing to do with the code being trained. A serviceable SLO surface for training has five SLIs.

Queue wait time at admission. How long does a training job sit in queue before it starts? This is the SLI that captures whether the cluster is operating in a reasonable resource regime or starving teams. A reasonable SLO might target the 95th percentile queue wait at under one hour for standard jobs and under 15 minutes for high-priority work. Klique’s Orchestration Engine exposes per-queue wait time, and its quota and fair-share machinery enforces the policies that keep the SLO achievable. A team running over its quota does not get to push other teams’ jobs out of the queue.

Job success rate. What percentage of submitted training jobs complete successfully? Two metrics hide inside this one, and they need separating. Infrastructure-attributable job failure rate (node failures, NCCL timeouts, filesystem faults, unrecoverable hardware errors) is the platform’s SLO, because the platform team can engineer against it. Total job success rate includes OOMs, NaN gradients, and configuration errors; it is a valuable health metric for the organization, but it belongs to the teams writing the jobs, not the team running the cluster. Collapsing them means a team shipping bad configs burns the platform’s error budget, and nobody can act on the number. An SLO of 99% on infrastructure-attributable failures and a tracked-but-unbudgeted total success rate is the more workable split. Klique’s job telemetry and lineage metadata are what make the attribution possible: you can see whether a failure originated in the job or under it, which GPUs cluster the failures, and which queues are affected.

Effective Training Time Ratio (ETTR). What fraction of wall-clock time on the cluster goes to productive training versus checkpointing, recovery, and idle waiting? This is the metric Meta publishes as its job-run productivity indicator, and Google Cloud tracks a similar measure it calls runtime goodput. Meta’s reliability study makes the sensitivity to scale concrete. Its two research clusters run at 6.50 and 2.34 failures per 1,000 node-days, and at those rates its largest high-priority runs (2,048 to 4,096 GPUs) average an ETTR of roughly 0.9 with optimally spaced checkpoints and five-minute restart and checkpoint overheads. Push to the largest feasible run the paper models, about 12,000 GPUs, two-thirds of the cluster, and holding ETTR at 0.9 requires either cutting the failure rate from 6.50 to about 1 failure per 1,000 node-days or getting checkpoint write overhead down to the order of 10 seconds. The lesson is the shape of the curve, not any single number: ETTR degrades superlinearly with job size, and the only levers are failure rate and checkpoint cadence. Klique’s monitoring stack exposes the per-job and per-cluster metrics that go into the ETTR calculation, which is the prerequisite for treating it as an SLO at all.

Checkpoint recovery time. When a training job fails, how long does it take to restart from the most recent checkpoint? Two costs live here and they trade against each other. The classic Young-Daly result puts the optimal checkpoint interval at roughly the square root of twice the checkpoint write cost multiplied by the mean time to failure, which sets how much progress you expect to lose when a job dies; that part is tunable. Restart time is the other half: requeue latency plus reading the checkpoint back and re-establishing collectives. Meta observed 5 to 20 minutes of restart overhead on its clusters. A recovery SLO of under 10 minutes for production training jobs therefore targets the good end of what a well-run cluster achieves today, and it is mostly won through fast requeue and checkpoint read bandwidth rather than through the interval math. Klique’s artifact management and its orchestration of distributed checkpoint storage are what make this measurable.

Mean Time To Failure (MTTF) at job scale. How often does a job of this size experience an infrastructure failure that requires intervention? For a 1,024-GPU job, the answer is roughly every 8 hours by Meta’s data, which is uncomfortably frequent. Setting an SLO target for MTTF gives the platform team an explicit reliability commitment they can engineer toward, through hardware burn-in, health gating, and proactive replacement. Klique’s job-level telemetry captures the failure events that feed into MTTF tracking. One caveat on framing: MTTF is a target rather than a budgeted objective. It does not decompose into an error budget the way a ratio-based SLI does, so treat it as the input that determines whether your ETTR and recovery-time budgets are achievable at a given job size, not as a budget in its own right.

Notice that none of these are availability metrics in the traditional SRE sense. Training does not have “99.9% uptime.” It has job-level success characteristics that the SRE framework can be adapted to, once the right SLIs are in place.

SLOs for fine-tuning

Fine-tuning sits between training and inference in almost every dimension. Jobs are shorter (hours to days, not weeks). Failure profiles are similar to training, but the recovery cost is lower because there is less wall-clock to lose. The fleet shape is different because fine-tuning runs are more frequent and more parallel: a single team might run dozens of fine-tunes per week against the same base model.

The SLO set for fine-tuning overlaps with training but emphasizes different things.

Time-to-first-result variance. A fine-tuning job that completes in 2 hours one day and 6 hours the next is harder to work with than one that consistently takes 3 hours, even if the absolute throughput is the same. SLOs for fine-tuning should constrain not just the median completion time but the spread. Klique’s run-level telemetry lets teams see this variance across runs and trace it back to scheduler decisions, data loading delays, or specific node health.

Cost-per-experiment variance. Fine-tuning fleets are where FinOps and SRE meet. If the cost per fine-tune varies by 3x across runs of similar configuration, the platform is not behaving deterministically and the team’s budget is harder to forecast. Klique’s resource consumption tracking, tied to project and team identity through the Governance Engine, makes the variance visible and the cause traceable.

Iteration cadence. How many fine-tunes can a team get through in a week? This is not a traditional SLI, but for an AI engineering organization it is the SLI that actually correlates with output. A platform that supports 50 fine-tunes per team per week is fundamentally different from one that supports 5. Quotas, fair-share scheduling, and queue depth metrics are the SLO machinery that protects iteration cadence.

The platform that runs fine-tuning is usually the platform that runs training. Klique’s position is that the SLO surface should be the same, with different targets per workload class. The control plane stays consistent. The targets reflect the workload.

SLOs for inference

Inference SLOs are the closest analog to traditional SRE practice, but with twists that catch teams that try to apply web-service patterns directly.

Request success rate. Before latency, the basic question: what fraction of inference requests return a valid response? For AI fleets this is less trivial than for a web service, because the failure modes are different: capacity rejections when the fleet is saturated, admission timeouts under queue pressure, context-length overflows, upstream model-provider errors on routed traffic, and truncated streams that return a 200 and then die mid-generation. That last category is the one traditional monitoring misses entirely: a stream that terminates after forty tokens is a failed request that looks like a success in the access log. A reasonable SLO is 99.9% request success rate with truncated streams counted as failures. Klique’s AI Gateway classifies these at the endpoint level, per tenant and per model.

P95 time-to-first-token (TTFT). The user-perceived latency of a streaming response is dominated by how long it takes to produce the first token. A reasonable SLO for interactive applications is P95 TTFT under 500ms, with stricter targets for voice and conversational agents. Klique’s AI Gateway captures TTFT at the endpoint level, broken out by tenant and model.

P95 inter-token latency. Once tokens are streaming, how long between them? This metric drives the perceived “smoothness” of an LLM response. A reasonable SLO is P95 inter-token latency under 100ms for chat experiences and under 50ms for voice. Reasoning models break TTFT as a proxy for perceived latency. A model that emits thirty seconds of hidden chain-of-thought before its first user-visible token has an excellent TTFT and a terrible user experience, and the two numbers no longer track each other. Reasoning fleets need time to first visible token as a distinct SLI alongside raw TTFT, and they need inter-token latency measured separately across hidden and visible spans; the hidden tokens are invisible to the user but consume identical GPU budget, so they belong in the capacity model even when they sit outside the latency SLO.

Throughput at SLO. How many concurrent requests can the fleet serve while staying within the latency SLO? This is the metric that ties capacity planning to SLO enforcement. Raw throughput is meaningless if it requires the fleet to operate outside its latency commitments. Klique’s autoscaling targets throughput at SLO directly, scaling out before latency degrades rather than after.

Autoscaling response time. When demand spikes, how long does it take to add capacity to the inference pool? This is a meta-SLO that determines whether the latency SLOs are achievable under realistic load patterns. An autoscaler that takes 5 minutes to add a GPU pod is too slow for most modern AI traffic patterns. The constraint here is physical: most of the time to add an inference pod is spent moving model weights from storage into GPU memory, which is why naive autoscaling on a large model lands in the multi-minute range no matter how fast the control plane is. Sub-minute scale-out is achievable, but only through pre-warmed pools, cached or snapshotted weights, and admission control that degrades gracefully while capacity arrives. A useful SLO here is P95 time-to-serving-capacity, measured from scale-out trigger to first request served within latency SLO. Klique’s Orchestration Engine treats this as an explicit operational concern rather than a side effect of pod scheduling.

Per-tenant fairness. In multi-tenant inference fleets, the right SLO question is not just “is the median user getting their TTFT,” it is “is the worst tenant getting their TTFT.” And in 2026, many of those tenants are not people. Agent workloads consume inference around the clock, and a reasoning-mode agent that burns 10x the tokens of a standard chatbot can starve other tenants of GPU time if the scheduler does not enforce fairness. Klique’s Governance Engine is built around exactly this case: every consumer, human or agent, is a governed tenant with its own quota, its own budget, and its own audit trail, so one tenant’s appetite cannot become every other tenant’s latency problem.

The SLO category that has no web-service analog

Everything above is a performance SLO. AI workloads have a second category that traditional SRE has no equivalent for: the response arrived on time and was wrong.

For agent traffic, this is not a philosophical concern; it is an operational one. An agent calling a structured-output endpoint needs schema-valid JSON; a malformed response that returns in 200ms is a failure that retries, doubling load and blowing the capacity model. Three SLIs are tractable enough to budget against today:

Schema validity rate for structured-output endpoints. Measurable, unambiguous, and directly consequential for agent workloads. A target of 99.5% or higher is reasonable.

Unintended refusal rate on traffic that should be served. Refusals spike after model version changes and prompt-template edits, and they are invisible in latency dashboards.

Eval regression on deploy. Not a continuous SLI but a gate: a model or prompt change that degrades a held-out eval set past a threshold is a reliability event, and it should burn error budget the same way a latency regression does.

This is where AI SLOs genuinely depart from the SRE playbook rather than merely translating it. A microservice that returns 200 has succeeded. An inference endpoint that returns 200 has only started to be evaluated.

A note on GPU saturation

GPU saturation gets cited often as an AI infrastructure SLI. The instinct is sound: idle GPUs are expensive. The problem is that saturation by itself is not an SLO target. A fleet at 100% GPU saturation with hour-long queue waits and breached latency SLOs is failing, even though the saturation number looks great. A fleet at 65% saturation with short queues and met SLOs is succeeding, even though there is theoretical capacity available.

The right framing is that saturation is an efficiency indicator paired with the SLO-compliance indicators above. The target is high utilization at SLO compliance, not high utilization alone. Klique’s Orchestration Engine is designed around this distinction. The fair-share machinery and the queue-depth metrics give the platform team the levers to push utilization up without pushing SLOs down.

Error budgets in an AI context

Error budgets are the SRE primitive that turns SLOs from a measurement exercise into a governance mechanism. When the budget is intact, teams ship. When it runs out, teams stabilize before they ship more. The mechanism only works if the budget is connected to actions that reduce risk.

Two definitions have to be pinned down before any of this is enforceable. The window: a rolling 28 days is the common default, and it matters because a monthly calendar window resets your budget on the first of the month regardless of what happened on the 31st. The measurement point: TTFT measured at the model server excludes admission queueing and gateway overhead, and will read materially better than the same metric measured at the gateway ingress, which is closer to what the caller experiences. Gateway-side measurement is the honest default. Any SLO stated without both of these is a number without a claim attached.

In an AI context, the actions look different. When an inference fleet burns through its TTFT error budget, the right response is rarely “freeze deployments.” It is more likely “throttle reasoning mode for low-priority traffic,” “cap context length on the affected endpoint,” or “scale out the decode pool ahead of the next demand peak.” When a training cluster burns through its ETTR error budget, the right response is to triage the failure modes, replace the GPUs that are crashing repeatedly, and tighten the health-gating policy.

Error budgets in AI need to be tied to capacity decisions, model behavior decisions, and tenant policy decisions, not just deployment freezes. Klique’s control plane gives platform teams the policy primitives to make those decisions structurally, rather than negotiating them ticket by ticket.

How Klique makes AI workload SLOs enforceable

Five capabilities sit at the center of how Klique approaches SLO enforcement across the training, fine-tuning, and inference lifecycle.

One control plane for the full lifecycle. Training, fine-tuning, inference, and agent workloads all run on the same platform, with the same identity model, the same scheduling policies, the same observability. The SLO surface for each workload class can be different, but the machinery that exposes and enforces them is shared. Teams that built their training stack on one platform and their inference stack on another end up reconciling two SLO regimes that should have been one.

Quotas, fair-share scheduling, and queue policies. SLOs are enforced through scheduling, not through dashboards. The Orchestration Engine’s quota and fair-share machinery is what turns SLO commitments into actual platform behavior. Queue depth, queue wait time, and tenant fairness are first-class concerns of the scheduler, not after-the-fact reporting metrics.

Continuous observability with SLO-relevant SLIs. Klique’s monitoring stack captures the metrics that AI workload SLOs actually need: queue wait time, job success rate, checkpoint operations, TTFT, inter-token latency, throughput, per-tenant compute and token consumption, and GPU utilization. The metrics are exposed at the granularity that lets platform teams set and track SLOs at the team, project, and tenant level, not just the cluster level.

Per-tenant isolation that survives high load. The hardest SLOs to maintain are the per-tenant fairness ones. When a noisy neighbor starts consuming disproportionate resources, the SLOs of every other tenant tend to degrade together. Klique’s tenancy model extends to every actor on the platform, teams, users, and agents alike, with isolation primitives the scheduler enforces, including fractional GPU allocation across both NVIDIA and AMD accelerators. SLO compliance holds for tenants because the platform makes it impossible for them to starve each other.

Operational primitives tied to error budget actions. When SLOs breach, the platform should make remediation easy. Klique’s policy machinery supports the actions that error budget breaches actually call for: throttling specific tenants, capping context length on specific endpoints, requeuing low-priority training jobs to free capacity for production inference, and routing traffic to alternate model variants through the AI Gateway. The actions are first-class platform operations, not bespoke incident-response scripts.

None of this is a turnkey SLO product. Klique is the platform layer that makes AI workload SLOs measurable, enforceable, and actionable. The SLO definitions themselves still belong to the platform team that owns the cluster. The platform makes them possible.

The right question to ask about your AI platform

Most AI platform conversations focus on what the platform can do. The SLO conversation is about what the platform commits to. If you cannot answer the question “what is your P95 TTFT SLO on the production inference fleet and how often do you breach it,” your AI platform does not have a reliability story. It has a launch story. The right answer includes a nonzero breach rate. A platform that claims it never breaches has either set the target too loose to mean anything or is not measuring anything.

The enterprises that get this right are the ones whose AI services scale into production without becoming a permanent incident response burden. The ones that get it wrong are the ones whose AI capabilities ship faster than the reliability machinery can keep up with, and who then spend the next year rebuilding the platform underneath. Klique’s view is that the platform should be SLO-aware from day one, with the metrics, policies, and enforcement primitives that the SRE framework calls for, applied to AI workloads from the start.

If you would like to discuss how Klique fits into your AI reliability and SLO strategy, get in touch.