LLM Deployment on Kubernetes: Where Production Breaks Under Load
A cluster handles demo load just fine and falls apart at the first spike: calls pile up in the model request queue, response time grows nonlinearly, and some requests fail due to timeout.
The cause is usually not the model itself — when deploying an LLM on Kubernetes, a lack of resources is exposed, either absent in the environment or already consumed by other workloads.
Accelerators are a scarce and indivisible resource.
If devices are concentrated on a single node, training, fine-tuning, and inference compete for them: the scheduler keeps service replicas waiting, the queue grows, and response time degradation appears even under moderate traffic. Scaling does not help here — you cannot add replicas without free accelerators.
A separate failure point is environment update conflict. Rolling out a new version of the model and related services happens in the same cluster as production traffic, and during warm-up throughput drops.
Without environment isolation, test and production workloads share the same devices, so one engineer's experiment becomes a customer incident.
All these failures are predictable, and each is resolved through configuration rather than on-call heroics: separate node pools, queue priorities, and a rollout strategy with warm-up and fast rollback. Below we look at how this is handled in practice — from capacity planning to the update policy.
What Managed Inference Delivers Under Peak Load
Managed inference is when model replicas live on GPU nodes managed by an orchestrator, and traffic comes through a single entry point.
The scheduler finds free accelerators itself, brings up a new replica next to the one already serving responses, and removes it when the queue empties. Deployment stops being a manual operation on a specific machine and becomes a declarative description of the desired state.
Stable response time is maintained not by the average but by the tail of the distribution. Scaling is driven by queue depth and accelerator utilization, not CPU load: a replica is added before requests start waiting in the buffer.
That is why the 95th percentile stays within the target range both during a spike and during quiet hours.
The share of successful requests grows because failure stops being total.
Readiness probes keep traffic away from a replica that has not warmed up yet, unhealthy instances are removed from load balancing, and timeouts and retries are configured so the client receives a response rather than an error.
Automated rollout of a model version follows a canary scenario: the new build receives a small share of traffic, quality and latency metrics are compared with the current one, and only then does that share grow. Rollback is a return to the previous image, without rebuilding the environment or manual edits.
The flip side of the same mechanism is a reduction in manual operations work. Nightly restarts, manual node scaling under load, and config edits are moved into declarative policies, and the engineer focuses on the model rather than on-call duties.
Model Deployment Options: Comparing Modes
The deployment mode determines how the cluster shares accelerators between models and how the system behaves at peak.
The choice depends on the number of models, workload isolation requirements, and traffic predictability: migrating between options after launch is more painful than choosing the right one up front.
Comparison of Modes
| Option | When It Fits | Limitations |
|---|---|---|
| Single node with accelerators | One or two models, internal environment, hypothesis validation | No workload isolation: a heavy request to one model degrades the neighboring one; scaling only upward, with the node resource as the ceiling |
| Shared accelerator pool | Multiple models with different load profiles and a common peak time | Requires quotas, preemption priorities, and per-model limits; noisy neighbor remains a risk, and failure of one node affects everyone |
| Dedicated group per model | Production models with strict response requirements and sensitive data | Under low load accelerators sit idle; more expensive to maintain, but failure is localized to one model |
| Model spread across multiple nodes | The model weights do not fit in one node or headroom is needed for concurrent requests | Inter-node communication adds latency; fault tolerance and updates are harder to plan |
In practice, teams start with a shared pool: it provides flexibility and allows measuring the real request profile.
A dedicated group is created for one or two models where the cost of downtime is higher than the cost of idle accelerators, and workload isolation is reinforced with separate namespaces, quotas, and priorities.
Scaling limits should be fixed in advance — the criterion by which a workload moves to a dedicated group and what happens when a node fails.
Implementation Stages: From Environment Audit to Handover
Deploying a model into a cluster is not installing a single service but a sequence where each step builds on the previous one.
If you skip the audit and go straight to configuration, the design will fall apart at the first change in the load profile: queues grow, accelerators sit idle, and rollback is assembled under pressure. Below is the order of work that we repeat on every project.
- Audit of the model inference environment — inventory: which models and weights are used, where they run now, the request profile, accelerator memory limits, and the list of consumers. The output is a map of the current state and a list of bottlenecks.
- Deployment design — topology of node pools, replica distribution, quotas, autoscaling, and environment isolation. The result is a configuration reproducible from a repository, not assembled by hand.
- Configuration and basic validation — manifests, weight cache, resource limits, network policies, metrics and log collection. We make sure the service responds and correctly survives a restart.
- Load testing — a profile close to production: concurrent requests, long contexts, peak spikes, and failure of an individual node. We look at time to first token and throughput, and record the degradation threshold.
- Version rollback plan — how to return the previous model image and configuration in one operation, tested on a staging environment. We do not release without a proven rollback.
- Handover to operations — access, policies, alerts, documentation, and on-call shift training. From there, the support team runs the system.
The stages proceed strictly in order: a skipped audit comes back as a redesign, and an untested rollback comes back as downtime at the worst possible moment.
What You Keep: Configurations, Designs, Policies
Everything we configure during deployment stays in your repository and cluster — without dependency on our team.
We hand over not a “folder with files” but a working environment: your engineer can stand up the staging environment from scratch using the documentation without asking us questions.
- Deployment manifests. Deployments, services, entry points, quotas for GPU nodes, autoscaling by request queue length. Applying them on a clean cluster reproduces the environment exactly.
- Fault tolerance design. Replica placement across zones, resource limits, readiness probe thresholds, and behavior when a node goes down. It shows what degradation is acceptable and what happens to the queue.
- Model update policy. Canary traffic share, metrics for comparing answer quality, rollback criteria, and the point of no return. An update stops being a service outage.
- Team access policies. Cluster roles, namespace separation, access to secret stores with keys, and per-team resource limits.
- Operations documentation. Architecture diagram, failure points, typical incidents and response procedures, metrics and logs, and degradation guidelines.
- Image build scripts. Reproducible builds, tagging, and pre-rollout checks.
After that, you run the service yourself: change the model, add teams, scale the node pool. We stay in touch for new tasks, but not as the only ones who can touch the configuration.
Case Study: Stable Inference Under Peak Load
The customer is a text generation service handling a stream of requests.
Average load was steady, but during morning hours the number of requests per minute tripled, and that exact picture broke: inference replicas did not finish warming up before being added to load balancing, some requests closed with an error, and rolling out a new model version stopped traffic intake for the entire environment rebuild time.
Metrics were captured before the intervention and one month after:
| Metric | Before | After |
|---|---|---|
| Share of failed requests during peak windows | 11% | 0.4% |
| Response time at the 95th percentile | 4.8 s | 1.3 s |
| Time to roll out a new model version | 40 min with traffic intake stopped | 5 min without stopping |
| Replica readiness after start | 6–9 min | 25–40 s |
The changes involved three things: warm-up, distribution, and rollout.
The warmed model weights now live in the image, a new replica does not enter load balancing until it is ready to accept requests, and the version rolls out to a share of traffic with a gradual expansion of that share.
When the queue grows, the scheduler brings up additional replicas; when it shrinks, it removes them, so no permanent reserve for peak is held, and the NVIDIA accelerator fleet did not grow.
Rollback to the previous version is one command, traffic returns within minutes, and this was separately verified in a training rollback.
How Does Inference Deployment Differ from Model Fine-Tuning?
These are two different environments with opposite load profiles, and mixing them in one environment is a source of constant failures. Inference deployment serves requests: traffic arrives unevenly, a response is needed instantly, and downtime is immediately visible to the user.
The model fine-tuning environment works in batches: long runs, full accelerator utilization, and stopping halfway costs a lost day of computation.
Resource separation dictates the topology. Inference needs a guaranteed reserve of accelerators and memory headroom for the request queue — otherwise failures will occur at peak. Fine-tuning is better off taking all free hardware and releasing it when finished.
That is why we separate the environments into dedicated node pools and by schedule: training starts in a low-load window and does not take performance away from serving.
Update frequency also differs. Weights change every few weeks, while the surrounding layer — routing, limits, cache, metrics — is updated constantly; the fine-tuning environment has its own cycle: experiment, quality check, and release of a new version.
Failure requirements are mirrored. Inference needs graceful degradation and fast rollback to the previous version: multiple copies, liveness probes, and draining traffic from an unhealthy node.
Fine-tuning cares more about preserving progress — regular checkpoints so that a node failure costs a restart of the last segment rather than the entire run.
We run both environments on Kubernetes as a common platform, but with different policies: separate pools, their own scaling rules, and different scheduler priorities.
This separation removes the main class of failures — when training displaces serving and the product goes down entirely.
Technical Doubts Before Launching Inference
Before launching a model into a cluster, teams ask almost the same questions — and rightly so: the answers determine how the service will behave during an update and in an incident. Below are short, to-the-point answers without generalities.
How do you know the bottleneck is not the accelerator?
We look at the entire chain: request queue, batch size, input data preparation, and transfer between services. Often the accelerator sits idle while time is spent on preprocessing and waiting in the queue. If the idle share is consistently high, it is not compute that needs optimizing.
How do you update a model without downtime?
The new version comes up next to the old one and first receives a small share of traffic — we compare answer quality and latency on live requests. Weights are stored as a separate versioned artifact, so rollback is switching traffic back, not rebuilding. Logs and responses are always tied to the model version.
How do you give teams access to the environment?
Through namespaces and roles: a developer sees the logs and metrics of their service but does not manage the cluster or other teams' workloads. Credentials live only in the standard secret store, not in configurations or chat.
Access is granted to a role, not a person — when team membership changes, nothing needs to be revoked manually.
What happens when an accelerator fails?
The node is removed from load balancing, and its workload moves to free ones. If there is no reserve, requests queue with a time limit — the service slows down but does not return an error.
At the same time, the degradation plan kicks in: we reduce parallelism and input length, and critical scenarios are served by a compact model.
We describe degradation levels, their thresholds, and the switching order in advance and verify them in drills — so during an incident, automation makes the decision, not the on-call engineer.
Let's Discuss the Task: What to Send for an Environment Assessment
To come back with an assessment and work plan rather than general thoughts, we need a short snapshot of your environment: it shows where the bottlenecks are, how many accelerators will really be needed, and what decisions must be made before the first rollout.
Send what you have — some inputs are often missing, and that is fine, we will fill them in together on a call.
- Model and weight format: open or fine-tuned, size, acceptable precision (full or quantized), whether access to private weights is needed and where to pull them from.
- Load profile: requests per second at peak and on average, context length, share of streaming responses, whether there are long batch jobs separate from interactive traffic.
- Current deployment environment: cluster and its network, availability of nodes with accelerators, what you use to manage resources, where weights are stored, and how images are built.
- Availability requirements: target share of successful requests, acceptable downtime during updates, and whether warm-up is required before putting a new instance into service.
- Access and boundaries: who owns the cluster, which external services must be available, and whether there are requirements for request isolation between teams.
- Observability: what is already collected — accelerator utilization, request queue, latency, logs — and where it is stored.
- Growth plan: load growth over the horizon, the emergence of new models, and the need for fine-tuning on your own data.
That is enough to propose a placement design, an update and rollback policy, and a work plan with stages and acceptance criteria. Describe the task in whatever form is convenient — we will come back with an assessment and a plan.
