Multimodal AI system: processing text, images, and audio

Multimodal AI falls apart at the junction of text, photos, and audio

The client submits a request as text, attaches a photo of the defect and a voice message — three channels, three different processing paths. One model parses the text, a second handles the image, and a third transcribes the audio.

Each branch works on its own, but at the junction of the channels the result diverges: the email has one order number, the photo has another, and the recording names a third.

Multimodal processing breaks not inside the models but at the join. Recognition errors are rarely visible in the output of a single branch — a confident answer masks them.

The problem surfaces later: entities do not match, fields are filled inconsistently, and some data has no anchor by which to link it to the rest.

Different data types arrive with mismatched labeling. Text has structure and keys, an image has visual features and model confidence, and audio has fragments without linkage to entities and authors.

Bringing this into a single record is possible only manually, and each disputed request consumes an operator's time for reconciliation.

A unified request flow is built differently.

First, a common identification layer links channels by client, time, and request context, then features are reconciled with one another, and a confidence threshold cuts off what the system should not infer for a human — such data is marked as requiring review rather than silently moving on. Below, we break down how this is assembled.

What a unified multimodal AI provides instead of a set of services

When text, images, and audio live in three separate services, every junction between them is manual work: the document is recognized in one place, the details are transferred to a second, and the call transcript is pasted into a third.

At each join, context is lost and a class of errors appears that none of the services sees: the amount in the invoice and the amount in the email diverge, and no one is there to reconcile them — the discrepancy surfaces only during final review.

In a unified multimodal system, modalities converge on a common representation layer: text, image, and audio are brought to a comparable form, and the model parses the fragment as a whole rather than as three disconnected answers.

We convert speech to text with a separate layer — here we rely on the open model Whisper — but the result enters the same end-to-end data flow as the document fields.

Because of this, manual transfers disappear: instead of manually reconciling data from three sources, only review of disputed cases remains.

The share of errors at the junction of channels drops severalfold — discrepancies between modalities are caught automatically and flagged as suspicious rather than passing further through the process.

Response time is no longer determined by the speed of an individual model but by the disappearance of the cascade: modalities are processed in parallel and converge at a single point, without sequential requests with waiting at each step.

Operating modes: cloud, closed environment, hybrid scheme

The choice of mode is determined not by hardware capacity but by two questions: what data is allowed outside the perimeter, and how critical response latency is.

A multimodal system has different flows — speech transcription is latency-sensitive, while parsing an archive of scans can safely go into overnight batch processing.

Switching between modes does not mean rewriting the system. The processing logic remains unified; the execution layer changes: where the models physically reside, where intermediate recognition artifacts are stored, and where the request log is written.

The boundaries between modes are set at the level of individual flows, not the entire system: the sensitive audio stream remains inside, while text generation goes outside.

Implementation stages: from data audit to handover to operation

A multimodal system is not assembled from ready-made blocks: text, images, and audio arrive with different quality, and the output requires one agreed-upon answer.

Therefore, the work proceeds in stages, and each ends with an artifact that is checked before moving to the next — otherwise a defect in the data will surface during operation.

1. Data audit. We examine volumes for each modality, the share of manual labeling, defects and gaps, and desynchronization between text and the attached image or recording.
2. Agreement on quality criteria. We agree on what counts as a correct answer and which errors are critical: false positive, miss, incorrectly identified modality.
3. Scheme design. We decide how modalities converge into one chain: where speech recognition is, where text parsing is, where the image is, and at what point the results are combined.
4. Model tuning. We select base models for the task, fine-tune on prepared data, and set activation thresholds and rules for disputed cases.
5. Quality validation. We run it on a holdout set and on modality-specific slices, analyze errors, and if necessary return to labeling or the scheme.
6. Handover to operation. We provide configurations, a description of the scheme, a model update regulation, and a procedure for rolling back to the previous version.
7. Team training. We show how to read metrics and operation logs, and what to do if quality drops after new data arrives.

At the output, the customer receives not a set of disconnected scripts but a system with clear boundaries of applicability, measurable quality, and control points at every layer.

What remains with the customer: schemes, settings, and regulations

After launch, the system lives without developers, so everything that determines its behavior must be reproducible on the customer's side.

We hand over not a “working service” but a complete set: without it, any change turns into contacting the contractor, and a rollback turns into searching for someone who remembers the original parameters. We record the composition in the acceptance act and verify it on the customer's test bench.

• System diagram: processing contours for text, images, and audio, data flows between services, storage points, and external calls tied to network segments.
• Environment configurations: test, pre-production, and production parameter sets, including model confidence thresholds, modality routing rules, and queue limits.
• Access policies: roles, rights to storages and indexes, service accounts, and the procedure for key rotation without stopping processing.
• Data schema: input and output formats, retention periods for raw files, and the list of fields that are masked before being written to logs.
• Documentation: description of the contours, instructions for typical operations, known model limitations, and the boundaries beyond which answer quality drops.
• Incident review regulation: how a failure is recorded, which logs and traces to inspect, and who decides to roll back the model version.
• Update regulation: the procedure for rolling out new versions and prompt templates, a set of validation scenarios, and the criterion under which an update is rolled back.

Acceptance runs by scenarios: we run typical and edge cases, compare quality metrics and recovery time after a training failure. Training the customer's team is part of the handover, not a separate option.

Case study: requests with photos and voice in a single processing flow

The client is a service company with a distributed network of locations: several hundred requests arrive per day, and almost half of them are not pure text. The client attaches a photo of the defect, records a voice message, and adds a comment in words.

An operator reviewed this manually: read the text, examined the image, listened to the audio — an average of 14 minutes per request, and by the end of a shift the classification error rate reached 18%.

What we built

Text, image, and audio now enter a single entry point, are brought to a common form, and are processed under one request identifier.

Speech transcription runs through Whisper, image classification uses a separate model, and a rules layer operates on top: it assigns priority by defect type and urgency.

The operator receives an already labeled request and confirms the decision rather than assembling it from scratch.

Erroneous reviews dropped from 18% to 3% — the effect came from modalities no longer conflicting: voice and photo end up in one card rather than in different systems.

Manual review remains where model confidence is below the threshold — that is 28% of requests, and that is exactly where operators' freed-up time goes. Primary review time decreased from 14 to 4 minutes, and peak hours no longer require bringing in an additional shift.

What to do if part of the data is already manually labeled?

We do not discard manually labeled data — it becomes a reference set for the new system. First, we assemble a labeling passport: a dictionary of classes, rules for assigning to each, a list of disputed examples, and agreements on them.

If such a document does not exist, we reconstruct it from the annotations themselves and conversations with those who labeled.

Next comes migration. A converter maps the fields of the current storage to the new system's schema, and we check compatibility on the intersection — we take examples labeled independently by two people and see whether the result matches after migration.

Classes that are not in the new schema are not deleted but moved to a separate layer: some will go into fine-tuning as is, and some as a hint for the labeler.

We build fine-tuning in iterations: old annotations set the starting quality, we add new batches, and after each one we compare metrics on the holdout part.

Manual work gradually shifts from simple cases to complex ones — the share of manual labeling drops, while quality on base classes does not degrade.

We make the transition parallel: the old and new processes run on the same document stream, results are compared on the same examples, and we switch over in parts — first one data type, then the rest. Rollback to the previous process is possible until the last step.

Process migration is a regulation: who labels, who verifies, what is considered disputed, and where it goes. Tools can remain the same if they provide exports; for new data types, we deploy Label Studio and transfer the rules there as well.

Common technical doubts before starting work

Before the start, engineers and department heads ask almost the same set of questions — about data, recognition quality, integrations, and responsibility after handover. Below are the answers as we give them in technical sessions, without vague wording.

What accuracy can we expect on our materials?

Public model metrics say little about your flow, so we calculate quality on a pilot sample from your texts, images, and recordings. We record the share of operations that pass without manual correction, and separately the error class that is critical for you.

What happens when the model becomes outdated?

The model version is pinned, and quality is monitored on a control sample. When metrics degrade, we start fine-tuning on accumulated examples: the update goes through the same validation as the first release, and rollback is possible at any moment.

How will this integrate into our processes, and who is responsible after launch?

We determine integration points during the assessment: API, queues, or file exchange — to match what already works for you.

We hand over configurations, schemes, and an operating regulation, train your team, provide a warranty for defect fixes, and provide support with metric monitoring.

Let's discuss the task: what to clarify on the first call

The first call is not for a presentation but to gather the inputs without which estimating a multimodal AI system turns into guesswork. From experience, this set of data is enough to come back with a work plan, timeline estimate, and team composition for a specific environment.

• Modalities and data: what is at the input — text, images, audio, in what proportion, whether labeling exists and in what volume.
• Quality metric: what error rate is considered acceptable and on which sample we calculate it.
• Infrastructure and access: cloud or closed environment, data isolation requirements, who issues access.
• Integrations: what we connect to — storage, queues, accounting systems, where the result goes.
• Load: how many requests at peak, what response latency is critical, whether batch processing is needed.
• Boundaries of the first version: what is mandatory in the first launch and what can be postponed without losing meaning.
• Acceptance: who on the customer's side accepts the result and by what scenario we validate.

After discussing the task, we come back with a stage-by-stage work plan, a timeline estimate with allowance for fine-tuning and quality tuning, team composition, and a clearly described next step.

Then comes a short validation on your data so that the estimate is based on measurements rather than assumptions.