Why a base model does not solve the task on your data
A base model is trained on a broad slice of the internet and confidently reasons about weather or cooking, but your subject domain is represented only fragmentarily in that slice.
It does not know your nomenclature, internal regulations, history of requests, and the abbreviations adopted at your company. In a demo this goes unnoticed: the questions asked are general, and the answers sound coherent.
The problem shows up on real data — where the answer must rely on a specific document rather than an averaged notion of the topic. It is precisely at this junction that deployment without fine-tuning Mistral breaks down.
The first thing that surfaces is general phrases instead of facts. The model answers "it depends on the terms of the contract" where you have an unambiguous wording in the regulation.
It is not formally wrong, but it is also of no use: the employee gets a retelling of the topic instead of an answer on the merits. The second is format.
Without fine-tuning a language model, it is hard to maintain the structure of the answer: fields, order of blocks, mandatory caveats, units of measurement. The model "drifts" from request to request, and the wrapper that parses its output starts failing on unexpected variants.
The third and most costly is terminology. In a narrow domain, the same words mean different things, and the model may confidently interpret your internal abbreviations in its own way.
Adapting the model to the subject domain is needed not for "beauty of style," but so that it distinguishes your entities and does not confuse them with similar ones from the general corpus.
Prompt engineering helps here only partially: it narrows behavior but does not add knowledge that is not in the weights. As soon as a question goes beyond the examples in the instruction, the quality of answers on domain data drops again.
Hence the practical conclusion: fine-tuning is not a replacement for document retrieval and not an attempt to "bake everything into the model." It is a way to fix in it the style, format, and terminology, while pulling facts from your sources.
Let us break down how this is covered: which fine-tuning methods suit different volumes of data and hardware, where the boundary lies between training and context injection, and how to measure that the model has truly started answering within your subject domain.
What fine-tuning Mistral gives you: LoRA and QLoRA modes
Fine-tuning Mistral through adapters solves the main problem of the base model: it answers "on average across the internet" rather than in the terms of your subject domain.
LoRA freezes the original weights and trains small low-rank matrices — the number of tunable parameters drops by two to three orders of magnitude, so training runs on a single GPU instead of a cluster, and the adaptation itself takes hours, not weeks.
QLoRA adds quantization of the base weights into a 4-bit format on top of this: memory requirements drop by roughly another factor of three, and fine-tuning becomes possible on consumer hardware without losing quality on the target task.
Where the effect is measurable
On typical scenarios — ticket classification, entity extraction from contracts, answers based on an internal knowledge base — a fine-tuned adapter raises accuracy on the target class from 60–70% to 85–92% with the same prompt.
The model stops "drifting" in its response format: the share of valid JSON without retries grows from 70% to 98%. The adapter weighs tens of megabytes, so it is versioned and rolled back like an ordinary artifact, not like a multi-gigabyte checkpoint.
Comparison of fine-tuning modes
| Option | When it fits | Limitations |
|---|---|---|
| LoRA | You have a GPU with memory to spare, you need maximum accuracy on a complex task | Requires more memory for the base weights and the optimizer |
| QLoRA | A single GPU, a limited hardware budget, a quick start | Quantization causes a slight drop on rare classes |
| Full fine-tuning | The very nature of the task changes, the domain is far from the training corpus | Expensive, slow, risk of catastrophic forgetting |
| Prompt engineering without training | The task is simple, there is little data, you need an answer within a day | It can't handle complex formats and narrow terminology |
The choice between LoRA and QLoRA is a trade-off between headroom on quality and the hardware available, not "best/worst". We run both modes on your dataset and show metrics on a held-out sample, not promises.
We work with open weights via Hugging Face and a PyTorch stack — this gives reproducibility and lets the adapter be moved into your environment without being tied to someone else's API.
How fine-tuning a pre-trained model works: stages of the project
Fine-tuning is not "start training and wait." Most of the time goes into data preparation and quality checks, while the training run itself is just one of the stages.
Below is the route our team follows: from data audit to handing the model over to production, with result control at every step.
- Data audit and task definition. We look at what is already available: request logs, correspondence, the knowledge base, ready-made question-answer pairs. We define what result we consider successful — accuracy on the test set, the share of correct refusals, the answer format. Without this criterion, training turns into guesswork.
- Dataset preparation. We clean up duplicates and noise, bring examples to a single format, and label complex cases. We split the data into training, validation, and held-out test parts — the latter does not participate in training and serves as an honest measurement.
- Choosing the fine-tuning method. We select the approach based on the task and the data volume: full fine-tuning, LoRA, or QLoRA. For most applied scenarios, adapters are enough — they deliver the required quality with lower hardware requirements.
- Training and parameter tuning. We run several configurations and watch for overfitting on the validation set. We lock in the data version and parameters so the result can be reproduced.
- Model validation. We check not only the metrics but also the behavior: robustness to paraphrasing, absence of fabricated facts, correct refusals on off-topic questions. We compare against the base model — the fine-tuned one should be better, not just different.
- Handover to production. We deploy the model in your environment or ours, set up quality monitoring and request logging. We provide documentation: how to update the data, how to retrain, how to roll back to the previous version.
- Post-launch support. We collect feedback from real requests, label disputed cases further, and plan the next fine-tuning iteration.
Each stage ends with an artifact: a labeled dataset, a locked configuration, a metrics report, an operations manual. Describe the task and the data you have at the input — we will come back with an estimate and a work plan.
How LoRA differs from QLoRA and what to choose under load
LoRA trains small low-rank adapter matrices on top of the frozen weights of the base model: the weights themselves are not touched, only the added layer is updated.
This reduces the volume of trainable parameters by orders of magnitude and makes it possible to fine-tune a model on one or two GPUs without spinning up a full cluster.
Classic LoRA keeps the base weights at full precision — usually 16 bits — and requires that they fit entirely in memory along with the adapter gradients and activations.
QLoRA adds model quantization on top of this: the base weights are stored in 4-bit format, while the adapters are still trained in 16 bits. The memory for the weights drops by roughly a factor of four, and a model that previously required several accelerators fits into one.
The price for this is unpacking the weights at every step of the forward and backward pass: training runs slower, and on long sequences the gap in speed is more noticeable.
The choice of adaptation mode comes down to resource requirements and to how sensitive the task is to precision. If there are enough accelerators and iteration speed matters, classic LoRA is used.
If memory is limited and the model is large, QLoRA makes it possible to run fine-tuning where the full variant simply does not fit.
In terms of final quality, with a carefully selected rank and a reasonable context length, the difference most often lies within the margin of error, but it should be checked on a validation set rather than taken on faith.
Separately, it is worth keeping in mind the rank of the adapters and the target layers: too small a rank underfits the task, too large a rank brings back the risk of overfitting and extra memory consumption.
Our engineers select these parameters for a specific dataset and load profile, rather than using the defaults from the documentation.
Case Study: Fine-Tuning Mistral for a Support Service
Fine-tuning was launched for a support service with a catalog of typical requests: operators responded according to the regulations, but the base model regularly drifted into free retelling, lost mandatory fields, and confused categories.
Manually editing every second response consumed first-line time, and the share of correct responses according to the regulations hovered around 60%.
We assembled a dataset from real dialogues with annotations: the reference response, the request category, mandatory fields.
We trained LoRA adapters on top of the base model — full fine-tuning mistral was not required, the weights were not rewritten entirely, so training ran on a single GPU and took hours rather than a day.
Separately, we set aside a validation set of requests that were not in the training data — so that the metrics would reflect behavior on new queries rather than memorization of examples.
What Changed in the Metrics
| Metric | Before | After |
|---|---|---|
| Correct response according to the regulations | ~60% | ~92% |
| Format compliance (fields, tone, structure) | ~55% | ~97% |
| Share of responses requiring manual editing | ~50% | ~12% |
| Responses with fabricated conditions | regularly | isolated cases |
Format compliance grew the most, because the model learned not "the right tone in general" but a specific response template with mandatory fields. Manual edits were reduced almost fourfold — the operator now reviews and confirms rather than rewriting from scratch.
How Quality Was Maintained
Fine-tuning does not leave everything to the model's discretion: responses undergo compliance checks against the regulations, and disputed cases are routed to an operator.
We built in a rollback mechanism — if quality on new requests degrades, the previous version of the adapters is restored without retraining.
For training we used Mistral AI — the open architecture allows keeping the model on our own infrastructure, without transmitting client requests externally.
Compatibility, access, and rollback: what to discuss before the start
Fine-tuning a model is not a standalone experiment in a vacuum, but a new artifact in your environment: it needs to be hosted somewhere, connected to something, and be rollback-capable.
That is why before the start we lock down three things: where the model will live (your closed environment, cloud, or hybrid), what data and services it will see, and how we will return the system to its original state if the quality of responses turns out to be lower than expected.
Adapters are lightweight and are stored separately from the base model — this is the rollback point: switching to the previous adapter restores the prior behavior without rebuilding the entire environment.
What happens if the fine-tuned model starts responding worse than the base one?
We do not replace the working model; instead, we connect an adapter on top of it. Rollback is switching to the previous adapter or to the base model, takes minutes, and does not affect the other services.
Before putting it into operation, we run a set of control queries and compare the responses with the original ones — the decision to switch is made based on numbers, not on gut feeling.
What if we have a closed environment with no external access? Training and inference are deployed inside your infrastructure; data does not leave the perimeter. Who will get access to the data?
We work on the principle of least privilege: access is granted only to engineers on the task, is logged, and is revoked upon completion. What if the model is already used in production?
The new adapter goes along a separate route; traffic migration is a controlled step with the ability to revert to the previous one.
After handover, you receive:
- training and inference configurations with a description of the parameters;
- a diagram of the model's placement in your environment and resource requirements;
- an access policy and delineation of rights to data and artifacts;
- a rollback procedure with a step-by-step description and return points;
- a set of control queries and reference responses for acceptance;
- documentation on maintenance and updating of adapters.
Describe your infrastructure and data constraints — we will come back with an assessment and an integration plan.
Timeline for fine-tuning a retrained model by stage
The timeline for fine-tuning is not made up of a single operation, but of a chain: data preparation, training setup, run, quality evaluation, rollout.
Most of the time goes not into the run itself, but into collecting and labeling the corpus and into accepting the result — these stages shape the work schedule more than the choice of method.
The duration of the stages depends on the volume of the corpus and the accuracy requirements: a small dataset of a few thousand examples completes the cycle in a couple of weeks, while a corpus of tens of thousands of examples with manual review stretches out preparation and acceptance.
In parallel with training, rollout infrastructure is being prepared — this does not lengthen the overall timeline if it starts at the labeling stage.
We fix the work schedule after the audit, when the real volume of data is visible, and we keep intermediate checkpoints: at each one the client sees the result and can adjust priorities. Describe the task — we will come back with an estimate and a plan by stage.
