Fine-Tuning the DeepSeek Language Model
DeepSeek is a family of open LLMs from the Chinese company DeepSeek AI, released under the MIT license. The flagship DeepSeek-V3 and DeepSeek-R1 compete with GPT-4o and Claude 3.5 Sonnet on benchmarks. Inference is significantly cheaper — up to 40% savings on GPU hours.
Open weights and high performance make DeepSeek an ideal candidate for fine-tuning for enterprise tasks. In our practice, we have fine-tuned this model for financial audit, legal analysis, and code generation — each time achieving a 1.5–2x improvement in accuracy.
Clients achieve significant savings on compute resources by deploying a fine-tuned model instead of a commercial alternative.
According to independent testing, DeepSeek-R1 achieves 89% accuracy on a mathematical reasoning task, which is comparable to closed-source counterparts.
How to fine-tune DeepSeek for business tasks?
The DeepSeek family: navigating the models
| Model | Parameters | Architecture | Application |
|---|---|---|---|
| DeepSeek-V3 | 671B (MoE, ~37B active) | MoE | Flagship, general purpose |
| DeepSeek-R1 | 671B (MoE) | MoE + Chain-of-Thought | Reasoning, mathematics |
| DeepSeek-R1-Distill-Llama-70B | 70B | Dense | Reasoning, more affordable |
| DeepSeek-R1-Distill-Llama-8B | 8B | Dense | Lightweight reasoning |
| DeepSeek-R1-Distill-Qwen-32B | 32B | Dense | Balance of quality/resources |
| DeepSeek-Coder-V2 | 236B (MoE) | MoE | Code generation |
For practical fine-tuning, we more often use the distilled versions (8B, 32B, 70B) — they train on a standard GPU cluster and deliver good results for specialized tasks.
Architectural feature: Multi-head Latent Attention (MLA)
DeepSeek-V3 uses MLA — an attention mechanism with KV-cache compression. Compared to GQA (Grouped Query Attention in Llama), MLA reduces KV-cache size by 5–13x at comparable quality.
This is critical for inference with long context — DeepSeek supports 128K tokens with reasonable memory requirements.
During fine-tuning, MLA layers are processed in the standard way via peft, but when choosing target_modules we account for the specifics: in DeepSeek-V3 the attention projections are named q_proj, kv_a_proj_with_mqa, kv_b_proj, o_proj.
What results does fine-tuning DeepSeek deliver?
Compared to GPT-4o or Claude, DeepSeek offers open weights and an MIT license — you are not locked in to a provider. The cost of inference on your own GPUs is 2–4x lower than via the OpenAI API.
With fine-tuning, you have full control over the data and the model, which is critical for confidential information. Our clients achieve significant savings on compute resources by deploying a fine-tuned model instead of a commercial alternative.
Stages of fine-tuning DeepSeek for a business task
- Analysis of the business task and collection of a representative dataset (3–8 weeks).
- Data labeling with think blocks to preserve reasoning.
- Selection of the fine-tuning architecture: LoRA or QLoRA with INT4 quantization.
- Training on a GPU cluster (12–24 hours for a 32B model).
- Quality evaluation by experts and iteration if needed (2 weeks).
- Deployment on your infrastructure via vLLM or Triton Inference Server.
LoRA fine-tuning of DeepSeek-R1-Distill-Qwen-32B
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, TaskType
import torch
model_name = "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map="auto",
trust_remote_code=True, # required for DeepSeek
torch_dtype=torch.bfloat16,
)
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
lora_dropout=0.05,
bias="none",
task_type=TaskType.CAUSAL_LM,
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
Working with the Chain-of-Thought when fine-tuning R1
DeepSeek-R1 generates reasoning in special <think>...</think> tags before the final answer. When fine-tuning on specialized data, it is important to:
- preserve the think blocks in the training examples — this maintains the model's reasoning ability;
- not suppress the
<think>token in the prompts during training; - if reasoning needs to be removed from inference — use
<think>\n\n</think>as a prefix or configure stopping criteria.
Example of a training example for R1:
{ "messages": [ {"role": "user", "content": "Is there a contradiction between clause 3.2 and clause 7.1 of the contract?"}, {"role": "assistant", "content": "<think>\nClause 3.2 establishes the payment term — 30 days from the moment of delivery. Clause 7.1 refers to the supplier's right to suspend deliveries in case of payment delay of more than 15 days. There is no contradiction — clause 7.1 creates a supplier's right that arises earlier than the expiration of the general payment term.\n</think>\n\nThere is no direct contradiction between the clauses..."} ] }
From our practice: financial audit on DeepSeek-R1
Task: fine-tune DeepSeek-R1-Distill-Qwen-32B to identify violations in financial documentation — discrepancies between balance sheet line items, violations of the double-entry principle, anomalous transactions. Dataset: 2,100 examples with think blocks, prepared jointly with practicing auditors. Training: QLoRA (r=32), 3 epochs, 4×A100 40GB, 18 hours.
| Metric | Before fine-tuning | After fine-tuning |
|---|---|---|
| Precision | 0.61 | 0.89 |
| Recall | 0.54 | 0.84 |
| F1 | 0.57 | 0.87 |
| Reasoning score (1–5) | 2.8 | 4.3 |
This case is from our team's practice: we deployed the solution at a company processing 500+ financial documents per day and reduced manual review time by 70%.
What is included in the work
We provide the full cycle: business task analysis, dataset collection and labeling, architecture selection (LoRA/QLoRA), training, testing, deployment on your infrastructure. As a result, you receive:
- a trained model together with weights and configs
- the dataset and training scripts
- deployment and monitoring documentation
- training for your team on working with the model
- support for 3 months after launch
Project timeline
- Dataset preparation with think blocks: 3–8 weeks (significantly more complex than ordinary SFT)
- Training (32B, 4×A100): 12–24 hours
- Reasoning quality evaluation: 2 weeks (requires expert assessment)
- Deployment and monitoring: 1–2 weeks
- Total: 7–14 weeks
Contact us — we will assess your scenario and propose the optimal turnkey fine-tuning strategy. Request a consultation: tell us about the task, and we will select the model, architecture, and work plan. More than 5 years of experience in LLM fine-tuning, 30+ completed projects on fine-tuning models for business.
