Production-Ready LLM Serving on Dedicated GPU Hardware: Turnkey Solution

A three-person startup deployed Llama-3-70B on a single A100 80GB — and the model crashed with CUDA OOM on every second request. The issue wasn't the GPU but the configuration: vLLM allocated 0.95 of GPU memory to the KV-cache, leaving no headroom for context length variations. We switched gpu-memory-utilization to 0.85 and enabled CPU swapping — the crashes stopped. Such situations are our daily work. Over the last few years, we've deployed more than 30 production inference systems for Large language model on dedicated GPU servers (on-premise or bare metal) and guarantee stable operation even under peak loads. Our team has 5+ years of ML infrastructure experience and has successfully deployed 30+ production inference systems with 99.9% uptime.

A dedicated server delivers predictable performance, no cold starts, and full data control. It is the optimal choice for high-load scenarios with data residency requirements or custom pipelines.

Problems we solve

VRAM miscalculation. For example, Llama-3-70B in BF16 requires 140 GB — not every server can handle that. We use quantization (AWQ/GPTQ) and tensor parallelism to optimally utilize resources. 4-bit quantization reduces VRAM requirement by 75% compared to BF16, allowing a 70B model to fit on 2×A100 and saving up to $15,000 in hardware costs. Mixtral-8x7B (MoE) activates only 13B parameters but needs 90 GB VRAM — easily fits on 2×A100 80GB after 4-bit quantization.

Service crash on OOM. On long contexts (>16k tokens), vLLM may crash. The solution is configuring max-model-len and gpu-memory-utilization paired with a systemd service that automatically restarts the process on failure. Additionally, we set up a watchdog script that checks the health endpoint.

Downtime during model updates. Without blue-green deployment, every update means downtime. We start a new version on a different port, test it, then switch the nginx upstream — no downtime.

How to select the right GPU?

Choosing a GPU depends on model size and required latency. Here's a recommendation table for popular models, with cost considerations: using 4-bit quantization can cut hardware expenses by 75%.

Quantization to INT4/INT8 reduces VRAM requirements 3–4 times, allowing a 70B model to fit on 2×A100. For a typical 70B deployment, this saves $15,000 in GPU hardware.

vLLM vs TGI: performance comparison

vLLM uses PagedAttention — efficient KV-cache management, giving 50% higher throughput at small batch sizes compared to TGI. In our benchmark of Llama-3-8B with AWQ quantization: vLLM delivered 1200 tokens/sec vs 800 for TGI (batch=32). vLLM outperforms TGI by 1.5x in throughput for batch sizes up to 32. However, TGI handles long context (≥32k tokens) better due to more aggressive tensor parallelism.

If your scenario is an online chat with short queries, choose vLLM. If you need to process multi-page documents, go with TGI.

How to prevent OOM errors?

On OOM, first check gpu-memory-utilization — it is often set too high. The recommended value is 0.85–0.90, leaving the rest for CPU swap. Also configure max-model-len proportional to average context length. In systemd, add Restart=always and a health-check script — the service will restart automatically.

Deployment and update process

Stages

1. Analytics — evaluate the model, expected RPS, and latency SLA. Select GPU and quantization.
2. Design — configure tensor parallel, batch size, quantization, and framework choice.
3. Implementation — install CUDA, drivers, deploy vLLM/TGI as a systemd service with watchdog.
4. Testing — load testing (locust/vegeta), measure p99 latency, check for long-tail issues.
5. Deployment — nginx reverse proxy with rate limiting, SSL, Prometheus + Grafana monitoring.
6. Documentation — describe endpoints, configurations, and update procedures.

Zero-downtime update

Classic blue-green: launch the new version on port 8001, test it, switch nginx upstream from port 8000 to 8001, then stop the old service. The entire process takes ~30 seconds; with proper keepalive settings, users don't notice the switch.

What's included in the work

• GPU selection and server configuration.
• Installation of CUDA and drivers (Docker if needed).
• Deployment of vLLM/TGI with systemd + watchdog.
• Nginx reverse proxy with rate limiting and SSL.
• Monitoring (Prometheus + Grafana — dashboards for GPU, latency, throughput).
• Zero-downtime model updates.
• Integration with MLOps pipelines.
• Documentation and team training.
• 30 days of post-deployment support.

Timeline: 2 to 7 days depending on complexity (GPU availability, number of models, HA). Cost is calculated individually — contact us for a project evaluation; it takes 1 business day.

Server configuration and infrastructure

Server setup

Before starting, check GPU availability and driver version:

nvidia-smi
nvcc --version

Install CUDA 12.1 and cuDNN. On Ubuntu 22.04:

apt-get install -y nvidia-driver-545
apt-get install -y cuda-toolkit-12-1

Check PyTorch: python3 -c "import torch; print(torch.cuda.get_device_name(0))"

Deploy vLLM as a systemd service

# /etc/systemd/system/vllm-llama.service
[Unit]
Description=vLLM LLaMA-3-8B Inference Server
After=network.target

[Service]
Type=simple
User=mlserving
WorkingDirectory=/opt/vllm
Environment="CUDA_VISIBLE_DEVICES=0,1"
Environment="HF_TOKEN=hf_xxx"
ExecStart=/opt/vllm/venv/bin/python -m vllm.entrypoints.openai.api_server \
    --model /data/models/llama-3-8b-instruct \
    --tensor-parallel-size 2 \
    --max-model-len 8192 \
    --max-num-seqs 128 \
    --gpu-memory-utilization 0.92 \
    --host 127.0.0.1 \
    --port 8000 \
    --log-level info
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Nginx reverse proxy

# /etc/nginx/sites-available/vllm
upstream vllm_backend {
    server 127.0.0.1:8000;
    keepalive 100;
}

limit_req_zone $binary_remote_addr zone=api_limit:10m rate=60r/m;

server {
    listen 443 ssl http2;
    server_name llm.company.internal;
    ssl_certificate /etc/nginx/ssl/cert.pem;
    ssl_certificate_key /etc/nginx/ssl/key.pem;

    location /v1/ {
        limit_req zone=api_limit burst=20 nodelay;
        proxy_pass http://vllm_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_read_timeout 300s;
        proxy_buffering off;
        chunked_transfer_encoding on;
    }

    location /health {
        proxy_pass http://vllm_backend/health;
    }
}

Monitoring and auto-restart

We use Prometheus and Grafana with nvidia_gpu_exporter to track GPU temperature, VRAM utilization, and throughput. Alerts: temperature > 85°C, VRAM > 95%, service unavailable > 30 seconds.

systemd Restart=always plus a watchdog script that checks the health endpoint every 30 seconds. After three failures, it restarts the service.

#!/bin/bash
while true; do
    if ! curl -sf http://127.0.0.1:8000/health > /dev/null; then
        systemctl restart vllm-llama
        echo "$(date) - vLLM restarted" >> /var/log/vllm-watchdog.log
    fi
    sleep 30
done

Our engineers have over 5 years of ML/infra experience and have deployed more than 30 inference systems, with a 99.9% uptime guarantee. Every project comes with a configuration guarantee and post-deployment support. If you need integration with a RAG pipeline, fine-tuning, or MLOps CI/CD — we can discuss it during a consultation. Get a preliminary estimate for your project — contact us, we'll respond within a day.

MLOps: Infrastructure for Training, Deploying, and Monitoring ML Models

The model is trained, metrics — F1 0.94 on validation. Three months later in production, quality drops by 12%. No one knows when — there is no monitoring. It's impossible to retrain quickly — the training script is in a Jupyter notebook of a data scientist who has already left. Data for retraining is collected manually from three disparate systems. About half of the projects come to us with this pain. We build a turnkey MLOps platform: from experiment tracking to automatic deployment and data drift monitoring. We will assess your infrastructure in 1–2 weeks, and in 4–6 weeks you will get a basic MLOps core running in production. Our team has 10+ years of experience in ML infrastructure, over 50 implementations.

How does MLOps infrastructure benefit your ML projects?

Experiment Tracking and Reproducibility

Without tracking, an ML project turns into chaos: it's unclear which checkpoint is better, which hyperparameters were used, which dataset. Reproducing a result a month later is a quest.

Why is experiment tracking the foundation of reproducibility?

MLflow is an open source standard for tracking. It logs parameters, metrics, artifacts (models, graphs), and code. MLflow Model Registry is a centralized model storage with versioning and lifecycle stages (Staging → Production → Archived). Deployment via MLflow Serving or integration with external systems.

Typical initialization in code:

import mlflow

mlflow.set_experiment("fraud-detection-v2")
with mlflow.start_run():
    mlflow.log_params({"learning_rate": 3e-4, "batch_size": 64, "epochs": 10})
    mlflow.log_metric("val_f1", val_f1, step=epoch)
    mlflow.pytorch.log_model(model, "model")

This is the minimum. In production, we add logging of system metrics (GPU utilization, memory), dataset (hash, version), code (git commit hash). Weights & Biases — richer UI, collaboration features, sweep for hyperparameter optimization. MLflow — for on-premise deployment without external dependencies.

DVC (Data Version Control) — versioning of data and models on top of git. Data is stored in S3/GCS/Azure Blob, only metadata (hashes) in git. dvc repro reproduces the entire pipeline from raw data to metrics.

To ensure reproducibility of training, fix random seeds (torch.manual_seed, numpy.random.seed, random.seed) and record them in experiment metadata. Without this, debugging irregular results is painful. Log the dataset version (DVC hash) and git commit — then any experiment can be reproduced down to the byte.

Pipeline Orchestration: Kubeflow, Airflow, Prefect

A pipeline orchestrator becomes necessary when: A 100-line training script in cron is fine for simple tasks. But as soon as you have a multi-step pipeline (data loading → preprocessing → feature engineering → training → validation → deployment if quality above threshold), you need an orchestrator with retry logic, visualization, and alerts.

Kubeflow — Kubernetes-native orchestrator for ML (see Kubeflow). Each step is a Docker container. Supports parallel steps, conditional branches, artifacts between steps. Integrates with Katib (AutoML), KServe (serving), Feast (feature store).

Apache Airflow — more general DAG orchestrator. Wide ecosystem of operators (S3, Spark, DBT, Kubernetes). Easier to deploy if Airflow already exists in the company.

Prefect / Metaflow — less boilerplate. Prefect 2.x with @flow and @task decorators — quick start for small teams.

Typical training pipeline architecture on Kubeflow:

1. Data ingestion component — fetches data from S3/DB, validates schema via Great Expectations
2. Preprocessing component — transformations, normalization, train/val/test split
3. Training component — training on GPU, logging to MLflow
4. Evaluation component — metric calculation, comparison with baseline in Model Registry
5. Conditional deployment — deploy only if new model is better than current by >2% F1

Each component is a separate Docker image. Pipeline is versioned in git. Scheduled run (retraining once a week on new data) or manual.

Model Registry and Lifecycle Management

Model Registry is not just a checkpoint store. It is a centralized system that knows:

• Which model is currently in production (and with what metrics)
• History of all versions with training parameters
• Metadata: dataset, git commit, validation results
• Lifecycle stage: None → Staging → Production → Archived

MLflow Model Registry — standard. For enterprise — Vertex AI Model Registry (GCP), SageMaker Model Registry (AWS), Azure ML Model Registry.

Model promotion through stages: automatically move model to Staging after successful eval, then manual or automatic (during A/B test) promotion to Production. Rollback — switch to previous Production version in seconds.

Serving: From FastAPI to Triton Inference Server

Simple case. FastAPI + PyTorch/ONNX on one server — 80% of production ML deployments are exactly that. Sufficient for most tasks with load up to 100 req/s.

from fastapi import FastAPI
import onnxruntime as ort

app = FastAPI()
session = ort.InferenceSession("model.onnx", providers=["CUDAExecutionProvider"])

@app.post("/predict")
async def predict(request: PredictRequest):
    inputs = preprocess(request.text)
    outputs = session.run(None, {"input_ids": inputs})
    return {"label": postprocess(outputs)}

Triton Inference Server — production standard for high loads (500+ req/s). Dynamic batching, concurrent model execution, model ensemble. Supports TensorRT, ONNX, PyTorch TorchScript, TensorFlow SavedModel.

KServe — Kubernetes-native ML serving with autoscaling, canary deployments, A/B testing out of the box. Scale-to-zero for inactive models — savings on infrastructure up to 40% annually for a project with 10 models.

Monitoring: Data Drift, Model Drift, Infrastructure Metrics

Monitoring — what is usually done last and regretted first. Three levels.

Infrastructure monitoring. Latency (P50/P95/P99), throughput (req/s), error rate (4xx, 5xx), GPU/CPU utilization. Prometheus + Grafana — standard. Alert when P99 latency > threshold or error rate > 1%.

Data drift monitoring. Distribution of input data changes over time. Detect via PSI (Population Stability Index) for numerical features: PSI > 0.2 — strong drift. Chi-squared test for categorical, Kolmogorov-Smirnov test for continuous. Evidently AI — open source library with ready-made drift tests.

Model drift monitoring. If ground truth is delayed (e.g., we know conversion after a week) — monitor real metrics. If not — surrogate metrics: distribution of prediction scores, proportion of confident predictions.

Alerting. Three levels: INFO (minor drift, log it), WARNING (significant, notify team), CRITICAL (quality dropped below threshold — automatic switch to fallback model).

Why is data drift monitoring important?

Without it, you learn about model degradation only from user complaints or ringing SLA. A drift alert allows you to retrain the model in advance, before errors start causing losses. In one of our projects, PSI monitoring detected drift 2 days after a data source change — this saved the campaign.

Feature Store

Feature Store solves the training-serving skew problem. If preprocessing during training and inference is implemented in two different places — divergence is inevitable.

A Feature Store is needed when:

• Several models use the same features
• Features are computed from streaming data (real-time)
• Large team with different people on feature engineering and model training

Feast — open source Feature Store. Offline store (S3 + Parquet) for training, online store (Redis, DynamoDB) for low-latency inference. Feature definitions as code, materialization job syncs offline → online.

Tecton (commercial), Vertex AI Feature Store (GCP), SageMaker Feature Store (AWS) — managed options with less ops overhead.

CI/CD for ML

ML CI/CD is regular CI/CD plus specific ML steps.

ML-specific checks in CI:

• Reproducibility check: run training with a fixed seed, result must match
• Data validation: Great Expectations or Pandera on schema/distribution checks
• Model performance check: automatic eval on holdout, block merge if degradation > threshold
• Latency regression test: inference must meet SLA

GitOps for deployment. Merge to main → CI triggers training → eval → if passes → automatic deployment to Staging → smoke tests → manual promotion to Production or automatic upon successful canary.

Tools: GitHub Actions / GitLab CI for CI, ArgoCD for GitOps deployment on Kubernetes.

What's Included in MLOps Platform Development

We provide a full cycle of work, documentation, and team training.

Total time from audit to full MLOps platform: 3–5 months. Also possible phased launch: basic level (tracking + serving) in 4–6 weeks.

Cost is calculated individually based on data volume, number of models, and infrastructure requirements. Order an MLOps infrastructure audit — get a roadmap in 1–2 weeks. Contact us for a project assessment — we will send a preliminary estimate within 2 business days.

Note: warranty on architectural solutions — 12 months. We provide integration certificates with major cloud providers (AWS, GCP, Azure). During our work, we have not lost a single client after the first implementation — the experience of 50+ successful MLOps projects speaks for itself. Get a consultation on building an MLOps platform today.