Fine-Tuning an Open LLM for Customer Tasks

You trained an LLM on your dataset. But the model hallucinates ICD-10 codes? Fine-tuning an open LLM is a way to turn a general model into a specialist in your domain. We fine-tune models with full control over the data. You own the weights. You deploy on-premise.

You scale inference without paying per token. Accumulated experience — more than 50 projects: from classification to generating complex reports. QLoRA allows you to save up to 75% of the budget on computing resources. We will evaluate your project. Write to us, we will discuss the details and choose a strategy. Get a consultation on your project.

How to choose a base model for the task?

Choosing a base model is a critical decision. The wrong choice leads to rework at the iteration stage. We assess which class of tasks the model performs best on and select the optimal option.

Why is fine-tuning more effective than RAG for some tasks?

RAG (Retrieval-Augmented Generation) is good when you need to pull current facts from an external source. But if a task requires a deep understanding of domain logic, fine-tuning delivers more consistent quality.

For example, in medical diagnostics, a model must accurately classify an ICD-10 code, while RAG can get confused due to noise in the retrieved documents. Fine-tuning "bakes" knowledge directly into the weights, eliminating the problem.

At the same time, LoRA fine-tuning is 4 times more memory-efficient than full fine-tuning, which makes it accessible for most projects.

What is synthetic data generation and why is it needed?

Often a client doesn't have labeled data, but does have unstructured sources: documents, regulations, FAQs. We use GPT-4o or Claude to automatically generate training pairs. This makes it possible to get thousands of examples in days rather than months. However, it's important to verify the sample manually — at least 10–15% for quality control.

from openai import OpenAI
import json

client = OpenAI()

def generate_training_example(document_chunk: str, num_examples: int = 5) -> list:
    """Generates training pairs from a document fragment"""
    prompt = f"""You are an expert at creating datasets for training language models.
Based on the document fragment below, create {num_examples} "question-answer" pairs in JSON format.
The questions should be diverse: factual, analytical, practical.
The answers should be precise, based only on the text of the document.
Document: {document_chunk}
Return a JSON array: [{"question": "...", "answer": "..."}]"""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
        temperature=0.7
    )
    return json.loads(response.choices[0].message.content)["pairs"]

How is a typical fine-tuning project structured?

Phase 1: Task and data audit (1–2 weeks)

• Formalization of the task (classification/generation/extraction)
• Inventory of available data, assessment of volume and quality
• Selection of the base model and training method

Phase 2: Data preparation (2–6 weeks)

• Collection and aggregation of sources
• Cleaning (duplicates, noise, PII)
• Labeling (manual/synthetic/combined)
• Formatting for the chat template
• Train/val/test split (80/10/10)

Phase 3: Training (1–4 weeks)

• Baseline evaluation of the base model
• First run of LoRA/QLoRA with default parameters
• Analysis of training/val loss curves
• Hyperparameter tuning
• Full Fine-Tuning if necessary

Phase 4: Evaluation and iterations (1–3 weeks)

• Automatic metrics (F1, BLEU, ROUGE, accuracy)
• LLM-as-judge (GPT-4o or another strong model as a judge)
• Human evaluation of a sample
• Analysis of failure cases → data refinement

Phase 5: Deployment and monitoring (1–2 weeks)

• Quantization (optional)
• Deployment via vLLM/TGI
• Setting up monitoring and an A/B test vs baseline

What is included in the work

• Analysis of the task and data, formalization of success metrics
• Selection of the optimal base model and training method
• Data preparation: collection, cleaning, labeling (manually or synthetically)
• Training with hyperparameter tuning, logging in MLflow
• Evaluation: autopilot + LLM-as-judge + expert holdout set
• Deployment (vLLM, TGI) and integration into your pipeline
• Documentation and training of the customer's team

Practical case: specialization for telemedicine (from our practice)

Task: an assistant for primary care physicians — differential diagnosis based on patient complaints, suggesting a list of examinations, selecting an ICD-10 code. Our client is a large medical center.

Input data:

• 450 clinical cases with conclusions (from the medical system, anonymized)
• Clinical recommendations of the Ministry of Health of the Russian Federation for 12 nosologies (PDF, 3200 pages)
• ICD-10 reference book

Strategy:

1. Converting clinical recommendations into chunks
2. Synthetic generation of 3200 examples via GPT-4o (complaints → diagnostics)
3. Verification of 15% of the sample by practicing therapists
4. Fine-tuning Qwen2.5-14B (best Russian language for medical terminology)

Results (after 4 epochs of QLoRA, r=32):

• Top-3 accuracy of ICD-10: 71% → 89%
• Completeness of the list of recommended examinations (recall vs expert): 0.62 → 0.84
• Hallucination rate (fabricated drugs/procedures): 24% → 6%
• Latency (vLLM, A100): 1.8s per request

QLoRA made it possible to train the model on 1x A100 80GB, which is 4 times more cost-effective than full fine-tuning. This approach is also suitable for your tasks — get in touch, and we will estimate the project.

Production quality monitoring

After deployment, we set up continuous monitoring:

import mlflow # Logging predictions for drift analysis
with mlflow.start_run():
    mlflow.log_metrics({
        "avg_response_length": avg_len,
        "refusal_rate": refusal_rate,
        "latency_p95": latency_p95,
        "user_rating_avg": rating_avg,
    })

Signs of model degradation: rising refusal rate, declining user ratings, an increasing share of escalation in downstream systems.

Infrastructure Requirements

Full-cycle timelines

A minimal project (ready data, simple task): 3–5 weeks. A typical project (data preparation from scratch): 8–14 weeks. A complex project (specialized domain, iterative labeling): 16–24 weeks.