Fine-Tuning GPT-4 and GPT-4o: Customizing LLMs for Business Tasks

Fine-tuning GPT-4: why the base model doesn't solve the task

Prompt engineering handles general scenarios well, but hits a ceiling where a task has its own logic.

As soon as customer terminology, a rare category of requests, or a rigidly specified response format appears in a query, the base model starts answering according to an "average" view of the world — and the result varies from run to run.

This manifests as two classes of failures. Unstable model responses: the same input yields different structure, tone, and level of detail, and on long instructions the model loses some requirements closer to the end of the query.

Domain hallucinations are more dangerous: the model confidently inserts plausible but nonexistent SKUs, codes, policy clauses, or internal system names, because such entities simply were not in its training corpus.

Language model fine-tuning here is not "improvement for improvement's sake" but a way to move rules from the prompt into the weights.

The model stops deriving the format from a description and reproduces it as a learned template: the share of responses that pass schema validation on the first try grows, and the variance between runs shrinks enough that the result can be put into a pipeline instead of manually checking every response.

The key part of the work is labeling training examples. You need not "roughly similar" dialogues but a consistent set: a single policy for edge cases, coverage of rare categories, a separate hold-out sample so metrics are calculated on data the model has not seen.

It is the quality of labeling, not the volume, that determines whether domain hallucinations go away or simply start sounding more accurate.

We'll break down which tasks truly require fine-tuning, where prompt configuration is enough, and how to measure the effect before releasing the model to users.

What fine-tuning provides: accuracy and controllability

Fine-tuning on a labeled set changes model behavior where instructions in the query no longer help.

The effect is visible on the hold-out portion of data that the model did not see during training: the share of correct answers grows from 70–75% to 90–95%, and the variance between runs shrinks to a few percent.

Fine-tuning GPT-4 on OpenAI's side delivers accuracy in your subject domain, not on an averaged set from open sources.

Terminology is the first thing that breaks the base model: internal product names, service code designations, and department abbreviations it paraphrases in its own way.

Labeling 300–1000 real dialogues closes this — the model reproduces your vocabulary and answers paraphrases of the same question consistently: the divergence between five formulations falls from 20–30% to 5%.

A single output format is a separate measurable effect.

A rigid response schema (fields, their order, values from reference books, absence of introductions) removes a layer of post-processing: responses are parsed directly by code, and the share of requests that an employee checks manually falls from 30–40% to 5–10%.

Controllability on rare categories is ensured by selecting examples: classes of questions where the base model erred in half the cases are pulled up to the average level.

Weights are fixed by checkpoints, so rolling back to previous behavior is replacing one set, not rebuilding the entire solution.

Model adaptation modes: comparison of approaches

The adaptation mode is chosen based on three inputs: how many labeled examples you have now, how narrow the subject domain is, and how often the task will change.

These are not mutually exclusive options but different in resources and risk ways to fine-tune GPT-4 — from editing behavior to rebuilding the model itself.

We compare along two practical axes: when a mode is truly needed and what you pay for it with.

The modes are compatible and form a ladder: first instructions, then adapters on top of the same base model, and only with a clear shortfall, full fine-tuning.

The model invocation interface does not change during the transition, so moving from instructions to adapters does not require rewriting the integration: the configuration and adapter version change, and rollback to the previous step remains available.

Implementation stages: from data audit to handover

The customer needs not a "trained model file" but predictable behavior in a working scenario. That is why the work follows a fixed sequence: at each step there is an artifact and a criterion by which it is accepted.

The order of stages below is not bureaucracy but a way to catch a problem before implementation, not after.

1. Data audit. We look at what has already been accumulated: correspondence, requests, internal documents, ready-made "question — reference answer" pairs. We assess volume by target scenarios, find duplicates, contradictions in labeling, and personal data that needs anonymization.
2. Setting the quality metric. Before training, we fix what we consider success: accuracy on the target class of queries, share of responses in the required format, share of correct "no data" refusals. The metric is agreed with you — otherwise acceptance turns into a dispute about tastes.
3. Preparing and labeling the set. We write instructions for labelers, review edge cases, and split data into training, validation, and hold-out samples. The hold-out sample is collected separately and does not participate in training until the very end.
4. Fine-tuning and mode selection. We compare fine-tuning with the base model and with instruction configuration, and watch for overfitting: growth on training examples while validation falls is a signal to stop.
5. Checking on the hold-out set. We run the model on examples it has not seen. This is the only honest evaluation: everything the training process saw shows memorization quality, not performance.
6. Implementation. We connect the model to your environment, set output format constraints, request logging, and rollback to the base version upon degradation.
7. Result acceptance and handover. A run through agreed scenarios, comparison with the metric, then documentation, access, and training for your team — further support is possible on your own.

Implementation artifacts: weights, schemas, regulations

A fine-tuning project is considered delivered not when the model answers on demonstration examples, but when you can reproduce its work and update it without our involvement. Therefore all results live in your repository and artifact registry, not on our side. Acceptance follows a list: every item can be opened, read, and checked.

• Model weights and fine-tuning adapters. For each scenario — a separate set, so that fine-tuning for one task does not worsen behavior in another. Alongside, the set's checksum and the version of the base model on which it was trained are recorded.
• Launch configurations. Generation parameters, system prompts, response length limits, and filtering rules — as files in the repository, not settings in someone else's panel. A configuration change goes through the same review path as code.
• Quality check suite. Test queries with expected answers and a run through them: the metric before and after for each new version, so degradation is caught before release.
• Environment schema. Where the weights are hosted, which services work with them, how traffic flows, who has access to data, and how request logging is organized.
• Version update and rollback regulation. Who initiates a release, which checks are mandatory, how many versions are stored, and how to return to the previous set. Rollback takes minutes, without retraining.
• Environment documentation. A component map, the procedure for deploying from scratch, integration points with your systems, and actions in case of quality degradation.
• Access and permissions. The repository, artifact registry, and infrastructure accounts are set up under your accounts, so that maintenance does not depend on one team.

Such a kit makes fine-tuning a reproducible process: the next version is released according to the regulation, not from scratch.

Case: automatic request classification

Starting point

Requests came from email, a website form, and messengers into one common inbox, and each operator manually decided which department to send it to.

The share of correct routes held around 71%: mixed topics in one message, complex wording, and different names for the same service threw off rule-based sorting.

Average time to transfer to the relevant department was about six minutes; during peak hours the queue grew, and the client waited for a response twice: first for classification, then for handling.

What was changed and what happened

We labeled the archive of requests by actual routes, discarded edge cases and duplicates, and kept a multiclass schema with a separate "requires a human" class.

We fine-tuned GPT-4o on this corpus and added a confidence threshold: below the threshold, classification hands the request to an operator together with the model's suggestion, rather than to a random department.

Quality was checked on the hold-out part of the archive that the model did not see during training — so metrics were calculated on unfamiliar texts, not memorized ones.

The share of correct routes grew from 71% to 93%, time to transfer to a department fell to tens of seconds, and manual sorting in the flow was eliminated.

Operator workload shifted from routine to complex cases: they handle composite requests and confirm edge cases instead of being a filter. Reprocessing due to an erroneous route stays within a few percent — the client no longer goes through a second round.

How is fine-tuning different from prompt instructions?

Instructions in a query and fine-tuning solve different tasks, and confusing them is costly. The former sets rules on top of a frozen model: response format, tone, prohibitions — and works only within the current request.

The latter changes weights on your examples, and the desired behavior becomes the model's own rather than suggested.

Instructions hold while there are few rules and they do not conflict. When the system part grows to dozens of clauses with caveats, drift begins: some requirements are followed, others are quietly ignored, and the entire text travels in context with every request.

A stable industry language or a single format across thousands of similar requests cannot be maintained this way.

Fine-tuning is taken where behavior is stable and confirmed by examples: dialogue labeling, field extraction from documents, a rigid response template. It compresses long instructions into a short prompt and levels quality on rare formulations.

The price is maintenance effort: a reproducible dataset, its versioning, a run on the reference set, and checking that previous scenarios have not degraded.

Maintenance effort is the selection criterion. A prompt is edited in minutes and rolled back with one line, but requires manual checking after every edit. A fine-tuned model is predictable within its set, but expanding topics means a new data preparation cycle and re-evaluation.

In practice, the approaches are combined in one environment: fine-tuning sets the base domain, style, and format, while instructions manage client-specific rules and fresh data from the knowledge base. Changing rules then does not trigger retraining — the environment is cheaper to maintain.

Technical doubts before starting work

Before starting, almost always the same set of questions comes up: how much data is needed, what will happen to quality on topics outside the training set, and how often to repeat training. Below are answers on the merits; they also determine the scope of work.

Can the model be trained in a closed environment?

Yes. We deploy computation on your hardware: data does not leave the perimeter, and the trained model is delivered as weights and configurations. If internal services do not have external access, training runs on local capacity, and updates are transferred inside manually.

Will the model lose general skills?

The risk is real: with aggressive training on a narrow set, the model loses general erudition and begins answering formulaically.

Therefore we tune part of the layers and add general examples to the training mixture, and on the control set we monitor both the target task and answers to unrelated topics.

How often is retraining needed?

By triggers, not by calendar: a shift in query topics, a drop in quality on the control set, new input data formats. Retraining takes hours, so the update cycle is built into the normal version release order.

Let's discuss the task: what is needed from your side

We dedicate the first call to analyzing the task, not a presentation: where the model is currently making mistakes, how much manual correction goes into one response, and what you consider an acceptable result.

So that the estimate is based on data, not feelings, it is useful to bring several things to the conversation.

• The task statement — which process we are automating, who uses it, and what a successful response looks like.
• Examples of "input → correct answer": even 50–100 labeled cases show whether fine-tuning GPT-4 on your data is needed or precise instruction configuration is enough.
• Acceptance criteria — share of correct answers on the hold-out set, allowable response latency, output format.
• Data constraints: what can be taken outside the perimeter, what remains on your side, what security requirements apply.
• Existing work — prompts, previously fine-tuned models, request logs: often part of the work has already been done.
• An engineer contact on your side for clarifications on labeling and edge cases.

In return, we provide not general words but an estimate of the volume of labeled data, a fine-tuning scheme — which set is needed, how we check quality, and what we do upon degradation — and a stage-by-stage work plan with points where you see the result on the test set.

Describe the task in a convenient form — we will get back to you with an estimate and work plan.