Fine-Tuning Llama: On-Premise, LoRA, QLoRA

Fine-Tuning Llama (Meta)

Imagine your company processes thousands of legal documents. You want to automate data extraction with 95% accuracy. GPT-4o does the job, but API costs grow with token volume. The solution: fine-tuning Llama 3.1 on-premise. You get weight files, deploy the model on your own infrastructure, and fine-tune without API limitations.

We are a team of AI/ML engineers with 5+ years of experience in fine-tuning open models (Transformer, GPT, LLaMA). We provide a turnkey service: from analyzing your data to deploying the model in production. Estimate the savings — self-hosted inference on Llama 3.1 8B costs 10–15 times less than an equivalent quality OpenAI API call at high loads.

Llama 3.x Model Family

For most fine-tuning tasks, Llama 3.1 8B or 70B is optimal. The former trains on a single A100 80GB, the latter requires 2–4 GPUs.

Why Choose Llama for On-Premise Deployment?

Unlike GPT-4o or Claude, you get the weight files. You can deploy the model on your own infrastructure and fine-tune without API limitations. This gives you full control over your data. According to our estimates, self-hosted inference on Llama 3.1 8B costs 10–15 times less. Compare that to an equivalent quality OpenAI API call. The difference is especially noticeable at high loads. Additionally, we have certified specialists and experience implementing in industries with strict security requirements.

Fine-Tuning Methods

Full Fine-Tuning updates all weights — maximum quality, but requires significant resources. LoRA (Low-Rank Adaptation) (Hu et al., 2021) updates only low-rank adapters on top of frozen weights. QLoRA additionally quantizes the base model to 4-bit. For 95% of tasks, LoRA or QLoRA is sufficient: they deliver quality close to full training at 5–15% of the cost.

Which target_modules to Choose for LoRA?

The target_modules parameter determines which layers receive LoRA adapters. Llama 3 architecture is a transformer with GQA (Grouped Query Attention). Typical targets:

• q_proj, k_proj, v_proj, o_proj — attention layers (minimum set)
• gate_proj, up_proj, down_proj — MLP layers (adds expressiveness)
• All 6 together — maximum quality, more adapter parameters

LoRA rank r determines the adapter size: r=8 gives ~0.1% additional parameters, r=64 gives ~0.8%. For style specialization, r=8–16 is enough; for complex knowledge extraction tasks, r=32–64.

Tech Stack: TRL + PEFT + Hugging Face

The main toolkit is the trl library paired with peft. Example QLoRA configuration:

from datasets import load_dataset
from trl import SFTTrainer, SFTConfig
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

# QLoRA configuration
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3.1-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto"
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

trainer = SFTTrainer(
    model=model,
    args=SFTConfig(
        output_dir="./llama3-finetuned",
        num_train_epochs=3,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        bf16=True,
        logging_steps=10,
    ),
    train_dataset=dataset["train"],
)

trainer.train()

Practical Example: Legal Assistant

Task: fine-tune Llama 3.1 8B to analyze Russian arbitration decisions and extract structured data (parties, subject of dispute, court decision, amount).

Dataset: 3200 pairs (decision text → JSON). Data obtained from the public database kad.arbitr.ru with 20% manual annotation and synthetic labeling with GPT-4o for the rest (with manual verification of a sample).

Infrastructure: one A100 80GB, training 4 hours (3 epochs).

Results:

• F1 for claim amount extraction: 0.58 → 0.91
• Accuracy of determining claimant/defendant: 82% → 97%
• Token generation speed: 47 tok/s (vLLM, A100)
• Inference cost vs GPT-4o API: 12x lower when self-hosted

Inference of the Fine-Tuned Model

After training, the LoRA adapter can be:

1. Used separately (PEFT inference): load base model + adapter
2. Merged into one model (merge_and_unload()): simplifies deployment, removes PEFT overhead
3. Quantized after merge: GGUF via llama.cpp, AWQ via autoawq, GPTQ — to reduce VRAM requirements

# Merge adapter into base model
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./llama3-merged")
tokenizer.save_pretrained("./llama3-merged")

For production deployment, we use vLLM — it provides PagedAttention and continuous batching, increasing throughput by 2–5× compared to naive inference via transformers.

What's Included in the Work

• Data preparation (collection, cleaning, annotation, augmentation)
• Model and fine-tuning method selection
• Training and validation (metrics, tests, baseline)
• Adapter merging, quantization, and inference optimization
• Deployment on your infrastructure (vLLM, TGI, llama.cpp)
• Documentation and training for your team
• 1-month support guarantee after delivery

Timeline and Infrastructure

• Data preparation and annotation: 2–6 weeks
• Training (8B, LoRA, A100): 2–8 hours
• Training (70B, QLoRA, 2×A100): 12–48 hours
• Evaluation and iterations: 1–2 weeks
• Deployment with vLLM/TGI: 3–5 days

Total from start to production: 4–10 weeks

How to Evaluate the Fine-Tuned Model's Effectiveness?

Use metrics depending on the task: for generation — ROUGE, BLEU, F1; for QA — precision, recall; for instructions — human eval or LLM-as-judge. We include an A/B testing phase: we compare the fine-tuned model with the baseline (API or base Llama) on a representative sample. Request a consultation — we will help determine metrics for your case.

Why Choose Us

5+ years of commercial experience in NLP and Computer Vision. 30+ successful fine-tuning projects for clients in fintech, legal, and healthcare. Certified specialists in Hugging Face, PyTorch, Triton Inference Server. We work with local installations, guarantee data confidentiality, and provide full documentation.

Interested? Contact us to discuss your project and get a preliminary estimate. We will help implement Llama into your infrastructure with maximum impact.

LLM Development: Fine-Tuning, RAG, Agents, and Production Deployment

Using GPT‑4 or Claude 3.5 Sonnet through a public API is not a solution — it's just a tool. When the requirement is to "make it like ChatGPT, but on our data," there is a real engineering challenge behind it: from prompt engineering to training a 70B model on your own infrastructure. End-to-end LLM solution development is a complex stack, and we have been doing it for over 5 years. During this time, we have completed over 20 projects in generative AI: from RAG systems for legal departments to custom support agents. Where exactly your task falls depends on data, latency requirements, budget, and how critical confidentiality is.

A typical situation: the client has already tried ChatGPT, but results are unstable — sometimes accurate, sometimes hallucinating. Or they need integration into a corporate portal while complying with security policies. Let's break down each layer of the stack in detail — from RAG to production deployment.

Why Do RAG Systems Break and How to Fix It?

RAG (Retrieval-Augmented Generation) looks simple: find relevant documents, put them in context, get an answer. In practice, it fails in several places.

Chunking without overlap. Classic mistake: chunk_size=512, overlap=0. If the answer lies across two chunks, retrieval won't find either with sufficient confidence. Solution: overlap 15–25% of chunk_size, or better yet, sentence-aware splitting with spaCy or NLTK instead of naive character splitting.

Poor embedder. text-embedding-ada-002 is good for general use, but on legal or medical texts, specialized models like E5-large-v2, BGE-M3, or fine-tuned sentence-transformers on domain data outperform it. Recall@5 differences can be 15–25%.

No re-ranking. Vector search optimizes for speed, not relevance. A cross-encoder re-ranker (ms-marco-MiniLM-L-6-v2, bge-reranker-large) after initial retrieval improves top-3 accuracy with acceptable latency (+50–150ms). This is often more impactful than improving the embedding model.

Hybrid search. Dense vectors alone work poorly on exact queries: names, SKUs, codes. BM25 (sparse) finds exact matches but misses semantics. Hybrid via RRF (Reciprocal Rank Fusion) is the optimal compromise. Qdrant, Weaviate, and pgvector 0.7+ support hybrid search natively.

When to Fine-Tune Instead of Prompt Engineering?

Prompt engineering solves ~70% of LLM adaptation tasks for a domain. The remaining 30% require fine-tuning. Three indicators: the model ignores a specific output format even with detailed prompting; the task requires deep knowledge of specialized vocabulary (medicine, law); you need to significantly reduce token costs by replacing a large model with a smaller specialized one.

LoRA and QLoRA are the standard for SFT. LoRA adds trainable low-rank matrices to attention layers. A typical configuration for Llama-3 8B: r=64, lora_alpha=128, target_modules=["q_proj","v_proj","k_proj","o_proj"] yields ~0.8% trainable parameters, training on one A100 40GB. QLoRA adds 4-bit quantization (NF4) and allows fine-tuning 70B models on two A100 40GB, though speed drops by half compared to bf16.

DPO instead of RLHF. Direct Preference Optimization requires only (chosen, rejected) pairs, not scalar reward signals. DPOTrainer from the trl library (Hugging Face) implements it in a few dozen lines.

Common mistake. A dataset of 500 examples, 5 epochs, validation loss 0.8 — seems fine. But on test, the model degrades on general instructions. Cause: catastrophic forgetting. Solution: add 10–20% general instruction-following examples (Alpaca, FLAN) to the training set to preserve original capabilities.

How to Choose a Base Model: 8B or 70B?

For most tasks, fine-tuning an 8B model is sufficient. 70B is needed when deep reasoning is required or the 8B baseline does not reach the required quality even after fine-tuning. Inference cost for Llama-3 8B via vLLM on A100 is efficient; the exact cost depends on volume.

What Does PagedAttention Bring to Production?

vLLM is the first choice for serving open-source models. PagedAttention is the key technical innovation: KV-cache is managed like virtual memory in an OS, without fragmentation. This yields 2–4x higher throughput compared to naive HuggingFace Transformers inference. The vLLM documentation confirms that continuous batching and PagedAttention are the standard for high-load LLM services.

Typical numbers on A100 80GB for Llama-3 8B (bf16): 400–600 req/s, P50 latency 200–400ms, P99 latency 600–900ms at concurrency 64. For 70B on two A100 with tensor parallelism: 80–120 req/s, P99 latency 1.5–2.5s. AWQ or GPTQ quantization reduces memory consumption by 2x with quality loss within 1–3%.

Multi-Agent Systems

Agents are LLMs with access to tools: search, code execution, API calls, database interaction. Common patterns:

• ReAct (Reason + Act): the model reasons → chooses a tool → observes the result → reasons again. LangChain and LlamaIndex implement it out of the box.
• Multi-agent orchestration: multiple specialized agents with a coordinator on top. Example: coordinator → researcher (search + summarization) → coder (code generation and execution) → critic (verification). Tools: AutoGen (Microsoft), CrewAI, custom implementation on LangGraph.

In production, agent systems are non-deterministic. Essential: guardrails, step limits, logging of each step, human-in-the-loop for critical actions.

How We Work: Stages, Timeline, Deliverables

What Is Included

We deliver:

• Technical documentation (model card, configs, deployment instructions)
• Access to infrastructure (code repository, trained weights)
• 1 month of post-deployment support (consultations, bug fixes)
• Customer team training (2–3 sessions on system operation)

Timeline: basic RAG prototype — 1–2 weeks. Fine-tuning with customer data — 3–6 weeks (including data preparation). Production system with monitoring and retraining — 2–4 months. Cost is calculated individually based on data volume, model complexity, and infrastructure requirements.

We guarantee the quality of the final model with performance benchmarks and ongoing monitoring. Our engineers have hands‑on experience with dozens of production LLM systems.

Want to evaluate your project? Leave a request — we will prepare a preliminary summary within 1–2 business days. Or get a consultation on choosing the approach: RAG, fine-tuning, or hybrid — we will tell you what works best for you. Contact us to discuss your LLM development needs. Schedule a free consultation today.