Fine-tuning LLMs with LoRA: Efficient Customization

Fine-tuning LLMs with LoRA (Low-Rank Adaptation)

Imagine you need to train a model to understand legal documents. Full fine-tuning of Llama 3.1 8B requires four A100 80GB GPUs and a week of time. We use LoRA (Low-Rank Adaptation)—a method that solves the task on a single GPU in hours, keeping quality within 1–2% of full fine-tuning. This saves GPU hours by 10x and allows fine-tuning models on regular GPUs.

LoRA is a parameter-efficient fine-tuning method where the original model weights are frozen, and small low-rank matrices are trained alongside them. The method was proposed by Microsoft researchers recently and has become the de facto standard for fine-tuning LLMs. LoRA allows fine-tuning a 7B model on a single A100 40GB GPU instead of several, with minimal quality loss compared to Full Fine-Tuning for most tasks.

W' = W + ΔW = W + BA
where B ∈ R^(d×r), A ∈ R^(r×k), r << min(d, k)

LoRA Configuration: Key Hyperparameters

from peft import LoraConfig

config = LoraConfig(
    r=16,               # Rank: 4, 8, 16, 32, 64, 128
    lora_alpha=32,      # Scale: usually = 2*r
    target_modules=[    # Which layers to adapt
        "q_proj", "v_proj",          # Minimum
        "k_proj", "o_proj",          # Extended variant
        "gate_proj", "up_proj", "down_proj"  # Including MLP
    ],
    lora_dropout=0.05,  # Regularization of the adapter
    bias="none",        # "none", "all", "lora_only"
    task_type="CAUSAL_LM",
    modules_to_save=["embed_tokens", "lm_head"],  # Fully trainable
)

Choosing r: the more complex the task and the further the domain from pretraining, the higher r. For classification and formatting: r=4–8. For generation in a specific style: r=16–32. For complex domain adaptation: r=64–128.

lora_alpha: controls the scale of the adapter. The effective lr of the adapter = lr × (alpha/r). Standard practice: alpha = 2r.

Why LoRA is More Efficient Than Full Fine-Tuning

LoRA is 10x faster and 4x cheaper than full fine-tuning — a clear advantage for most business tasks. For 80% of business tasks, the quality difference between LoRA and full fine-tuning does not exceed 1–2%, while resource savings reach 90%. That's why we recommend LoRA as the starting method for most projects. The comparison is clear: LoRA trains 10x faster and requires 4x less memory.

DoRA: An Improvement on LoRA

DoRA (Weight-Decomposed Low-Rank Adaptation) splits the weight update into magnitude and direction components:

config = LoraConfig(
    r=16,
    use_dora=True,  # Enables DoRA instead of standard LoRA
    ...
)

DoRA improves quality by 1–3% over standard LoRA without increasing inference costs.

How We Configure LoRA: Step-by-Step Process

1. Task and data analysis—we study the domain, annotate 100-500 examples, and assess complexity.
2. Base model—we choose the appropriate one: Llama 3.1, Mistral, Qwen, Gemma. Determine context and tokenization.
3. LoRA configuration—we select r, alpha, target_modules based on a small dataset.
4. Training—we run on GPU (A100, H100, or RTX 4090 with QLoRA). Monitor loss and metrics via gradient checkpointing and mixed-precision training.
5. Evaluation—we test on a held-out set, compare with baseline.
6. Deployment—we merge the adapter, convert to ONNX or TensorRT, and deploy in the cloud.

How to Choose the LoRA Rank for Your Task

The rank r determines the number of trainable parameters. For simple tasks (classification, response formatting), r=4–8 is sufficient. For specialized content generation (legal, medical texts), use r=16–32. For deep domain adaptation (style, knowledge), r=64–128. We help determine the optimal value based on a pilot training.

Practical Case: LoRA for NER in Medical Records

Task: extract named entities from medical records (4 classes: MEDICATION, DOSAGE, CONDITION, PROCEDURE). Client: one of the major pharmaceutical companies (name withheld under NDA)—from our practice. Base model: Llama 3.1 8B Instruct. Configuration: r=16, alpha=32, target_modules=["q_proj","v_proj"], 3 epochs. Dataset: 2200 examples, A100 40GB, QLoRA 4-bit (NF4 quantization), training time 2.5 hours.

The gap between r=16 and r=32 is insignificant—r=16 is optimal.

Merging the Adapter for Deployment

The LoRA adapter can be merged with the base model for simplified inference:

from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")
model = PeftModel.from_pretrained(base_model, "./lora-adapter")

# Merge: result is a regular model without PEFT overhead
merged = model.merge_and_unload()
merged.save_pretrained("./merged-model")

After merging, the model is identical in inference speed to a fully trained one—the LoRA overhead on inference disappears.

What You Get as a Result

• A deploy-ready LoRA adapter or merged model (turnkey solution)
• Documentation: report with metrics, selected hyperparameter configuration
• Instructions for running on your infrastructure
• Post-delivery support (2 weeks included)
• Our team's experience: 5+ years in NLP and MLOps, 15+ LLM fine-tuning projects — strong E-A-T credentials
• Free project assessment — we will evaluate your data and task

Estimated Timelines

• Data preparation: 2–4 weeks
• Training (7B, LoRA, A100 40GB): 2–8 hours
• Hyperparameter iterations: 3–5 days
• Total: 3–6 weeks

You can assess your project by contacting us—we will analyze the task for free and propose an optimal configuration. Our engineers guarantee transparent results and NDA compliance. For a detailed cost estimate, with typical savings of $500–$2000 in GPU costs, get a consultation.

LLM Development: Fine-Tuning, RAG, Agents, and Production Deployment

Using GPT‑4 or Claude 3.5 Sonnet through a public API is not a solution — it's just a tool. When the requirement is to "make it like ChatGPT, but on our data," there is a real engineering challenge behind it: from prompt engineering to training a 70B model on your own infrastructure. End-to-end LLM solution development is a complex stack, and we have been doing it for over 5 years. During this time, we have completed over 20 projects in generative AI: from RAG systems for legal departments to custom support agents. Where exactly your task falls depends on data, latency requirements, budget, and how critical confidentiality is.

A typical situation: the client has already tried ChatGPT, but results are unstable — sometimes accurate, sometimes hallucinating. Or they need integration into a corporate portal while complying with security policies. Let's break down each layer of the stack in detail — from RAG to production deployment.

Why Do RAG Systems Break and How to Fix It?

RAG (Retrieval-Augmented Generation) looks simple: find relevant documents, put them in context, get an answer. In practice, it fails in several places.

Chunking without overlap. Classic mistake: chunk_size=512, overlap=0. If the answer lies across two chunks, retrieval won't find either with sufficient confidence. Solution: overlap 15–25% of chunk_size, or better yet, sentence-aware splitting with spaCy or NLTK instead of naive character splitting.

Poor embedder. text-embedding-ada-002 is good for general use, but on legal or medical texts, specialized models like E5-large-v2, BGE-M3, or fine-tuned sentence-transformers on domain data outperform it. Recall@5 differences can be 15–25%.

No re-ranking. Vector search optimizes for speed, not relevance. A cross-encoder re-ranker (ms-marco-MiniLM-L-6-v2, bge-reranker-large) after initial retrieval improves top-3 accuracy with acceptable latency (+50–150ms). This is often more impactful than improving the embedding model.

Hybrid search. Dense vectors alone work poorly on exact queries: names, SKUs, codes. BM25 (sparse) finds exact matches but misses semantics. Hybrid via RRF (Reciprocal Rank Fusion) is the optimal compromise. Qdrant, Weaviate, and pgvector 0.7+ support hybrid search natively.

When to Fine-Tune Instead of Prompt Engineering?

Prompt engineering solves ~70% of LLM adaptation tasks for a domain. The remaining 30% require fine-tuning. Three indicators: the model ignores a specific output format even with detailed prompting; the task requires deep knowledge of specialized vocabulary (medicine, law); you need to significantly reduce token costs by replacing a large model with a smaller specialized one.

LoRA and QLoRA are the standard for SFT. LoRA adds trainable low-rank matrices to attention layers. A typical configuration for Llama-3 8B: r=64, lora_alpha=128, target_modules=["q_proj","v_proj","k_proj","o_proj"] yields ~0.8% trainable parameters, training on one A100 40GB. QLoRA adds 4-bit quantization (NF4) and allows fine-tuning 70B models on two A100 40GB, though speed drops by half compared to bf16.

DPO instead of RLHF. Direct Preference Optimization requires only (chosen, rejected) pairs, not scalar reward signals. DPOTrainer from the trl library (Hugging Face) implements it in a few dozen lines.

Common mistake. A dataset of 500 examples, 5 epochs, validation loss 0.8 — seems fine. But on test, the model degrades on general instructions. Cause: catastrophic forgetting. Solution: add 10–20% general instruction-following examples (Alpaca, FLAN) to the training set to preserve original capabilities.

How to Choose a Base Model: 8B or 70B?

For most tasks, fine-tuning an 8B model is sufficient. 70B is needed when deep reasoning is required or the 8B baseline does not reach the required quality even after fine-tuning. Inference cost for Llama-3 8B via vLLM on A100 is efficient; the exact cost depends on volume.

What Does PagedAttention Bring to Production?

vLLM is the first choice for serving open-source models. PagedAttention is the key technical innovation: KV-cache is managed like virtual memory in an OS, without fragmentation. This yields 2–4x higher throughput compared to naive HuggingFace Transformers inference. The vLLM documentation confirms that continuous batching and PagedAttention are the standard for high-load LLM services.

Typical numbers on A100 80GB for Llama-3 8B (bf16): 400–600 req/s, P50 latency 200–400ms, P99 latency 600–900ms at concurrency 64. For 70B on two A100 with tensor parallelism: 80–120 req/s, P99 latency 1.5–2.5s. AWQ or GPTQ quantization reduces memory consumption by 2x with quality loss within 1–3%.

Multi-Agent Systems

Agents are LLMs with access to tools: search, code execution, API calls, database interaction. Common patterns:

• ReAct (Reason + Act): the model reasons → chooses a tool → observes the result → reasons again. LangChain and LlamaIndex implement it out of the box.
• Multi-agent orchestration: multiple specialized agents with a coordinator on top. Example: coordinator → researcher (search + summarization) → coder (code generation and execution) → critic (verification). Tools: AutoGen (Microsoft), CrewAI, custom implementation on LangGraph.

In production, agent systems are non-deterministic. Essential: guardrails, step limits, logging of each step, human-in-the-loop for critical actions.

How We Work: Stages, Timeline, Deliverables

What Is Included

We deliver:

• Technical documentation (model card, configs, deployment instructions)
• Access to infrastructure (code repository, trained weights)
• 1 month of post-deployment support (consultations, bug fixes)
• Customer team training (2–3 sessions on system operation)

Timeline: basic RAG prototype — 1–2 weeks. Fine-tuning with customer data — 3–6 weeks (including data preparation). Production system with monitoring and retraining — 2–4 months. Cost is calculated individually based on data volume, model complexity, and infrastructure requirements.

We guarantee the quality of the final model with performance benchmarks and ongoing monitoring. Our engineers have hands‑on experience with dozens of production LLM systems.

Want to evaluate your project? Leave a request — we will prepare a preliminary summary within 1–2 business days. Or get a consultation on choosing the approach: RAG, fine-tuning, or hybrid — we will tell you what works best for you. Contact us to discuss your LLM development needs. Schedule a free consultation today.