How to Deploy Large Language Models on Yandex Cloud

Typical situation: a company developed an RAG bot based on domestic LLMs, but Western clouds are unavailable. We have already deployed Mistral 7B and LLaMA 3 on Yandex Cloud using vLLM and know how to bypass pitfalls — from CUDA configuration to load balancing. With over 5 years of experience in AI/ML and 50+ successful deployments, we guarantee efficient solutions. Get a consultation for your project — we will assess it within 2 days.

Why Yandex Cloud for LLM?

Yandex Cloud is the main domestic provider with GPU instances, Yandex ML Platform, and its own LLMs (YandexGPT). For Russian companies with data residency and import substitution requirements, this is the optimal choice. We get access to GPUs on demand, managed Jupyter (DataSphere), and integration with Object Storage. Support for CUDA and Tensor Cores ensures maximum performance. According to the official Yandex Cloud documentation, g3 GPU instances on A100 provide up to 80% performance improvement over V100 for LLMs.

Setting Up a GPU Instance

GPU clusters: g2 (Tesla V100 32GB) and g3 (A100 80GB). For most open-source 7B parameter models, one V100 is enough. Creating a VM via YC CLI:

yc compute instance create \ --name llm-server \ --zone ru-central1-a \ --platform gpu-standard-v3 \ --gpus 1 \ --memory 48GB \ --cores 14 \ --core-fraction 100 \ --image-family ubuntu-2204-lts-gpu \ --image-folder-id standard-images \ --disk-type network-ssd \ --disk-size 300GB \ --network-interface subnet-name=default,nat-ip-version=ipv4 \ --ssh-key ~/.ssh/id_rsa.pub 

Choosing the Right GPU

For 7B models in INT4, V100 provides acceptable speed — up to 150 tokens/sec. If you plan to serve multiple requests in parallel or work with 13B+ models, take A100. Also consider budget: A100 costs 2–3 times more. For experiments, V100 suffices; for production with high load, use A100.

Deploying with vLLM

vLLM is the most performant inference engine for LLMs, 2–3 times faster than Hugging Face Transformers on the same GPU. It supports continuous batching and tensor parallelism. Installation and launch:

sudo apt-get update && sudo apt-get install -y python3-pip pip install vllm # Download model from Yandex Object Storage aws s3 sync s3://my-bucket/models/mistral-7b/ /data/models/mistral-7b/ \ --endpoint-url https://storage.yandexcloud.net \ --profile yandex # Launch server python -m vllm.entrypoints.openai.api_server \ --model /data/models/mistral-7b/ \ --tensor-parallel-size 1 \ --max-model-len 8192 \ --max-num-seqs 128 \ --port 8000 \ --host 0.0.0.0 

For optimization, use flags --quantize awq for INT4 and --kv-cache-dtype auto. With multiple GPUs, enable --tensor-parallel-size. Monitor metrics via the /metrics endpoint. Compared to Hugging Face Transformers, vLLM achieves >90% GPU utilization and p50 latency below 100ms, while HF typically uses 60–70% GPU and ~300ms latency.

Model Storage and Prototyping

We store model weights in S3-compatible Object Storage: cheap (≈ 2–3 ₽/GB/month) and reliable. Example upload using Python boto3:

import boto3 s3 = boto3.client( "s3", endpoint_url="https://storage.yandexcloud.net", aws_access_key_id=os.getenv("YC_ACCESS_KEY"), aws_secret_access_key=os.getenv("YC_SECRET_KEY"), region_name="ru-central1" ) for file in model_files: s3.upload_file( Filename=f"/local/models/{file}", Bucket="llm-models-bucket", Key=f"mistral-7b/{file}", ExtraArgs={"StorageClass": "COLD"} ) 

For prototyping, DataSphere provides managed Jupyter with on-demand GPUs. Ideal for testing quantization parameters (INT4/INT8) before production deployment.

Scaling and Monitoring

To scale horizontally, place multiple vLLM instances behind an Application Load Balancer:

yc alb target-group create llm-targets \ --target subnet-name=default,ip-address=10.0.0.10 \ --target subnet-name=default,ip-address=10.0.0.11 yc alb backend-group create llm-backends \ --http-backend name=vllm-backend,port=8000,target-group-id=xxx,healthcheck-path=/health yc alb http-router create llm-router \ --virtual-host name=llm,authority=llm.company.ru \ --route name=api,path-prefix=/v1,backend-group-id=xxx 

Monitoring uses Yandex Monitoring for basic metrics (CPU, RAM, GPU utilization via DCGM exporter) and custom metrics via Unified Agent for p99 latency and tokens per second.

Deployment Checklist

1. Requirements analysis — determine load, latency, budget.
2. Model and quantization selection — choose the optimal one for the task.
3. Create GPU instance — configure VM via YC CLI.
4. Install vLLM — set up and optimize the inference engine.
5. Upload model to Object Storage — transfer weights to S3 storage.
6. Launch and configure API — open endpoint, integrate with the application.
7. Load balancing and monitoring — add ALB, set up alerts.

Our Expertise and Deliverables

• Audit of your task and configuration selection (model, GPU, budget)
• Deployment of vLLM / TGI on Yandex Cloud with monitoring
• Configuration of API gateway and autoscaling
• Operation documentation and team training
• Technical support during launch
• Guaranteed p99 latency <200ms for models up to 13B, uptime 99.9%

With over 5 years of experience in AI/ML and 50+ successful projects, we are certified Yandex Cloud specialists. Contact us for a consultation — we'll assess your project within 2 days. Order LLM deployment on Yandex Cloud right now!