Configuring request and response logging for an AI model

LLM logging: where collection of requests and responses breaks

A typical failure looks the same: the log has the time, response code, and call duration, but no request or response bodies. Logging is usually enabled at the reverse proxy level, while the prompt is assembled in the service and never reaches it. As a result, the model call log consists of metadata that makes incident investigation impossible.

The second bottleneck is streaming response delivery. The model returns text in chunks, and the collector either saves only the first one or concatenates them without message boundaries.

During timeouts and retries, one record ends up in the log instead of several — the cause of failure is not visible. The end-to-end identifier between services is lost, so there is no way to link the request, response, and subsequent user action.

Another reason is personal data and keys inside prompts: because of them, LLM logging is often disabled entirely or masked so heavily that only a set of characters unusable for debugging remains from the text. This harms debugging of the AI integration.

A working solution is redaction rules on the application side before writing: sensitive fields are replaced, and everything else is stored in full.

It is convenient to build export on the open standard OpenTelemetry — a unified format removes the log's dependency on a single model provider.

For support and product, this turns into a blind spot: a complaint cannot be reproduced, response quality cannot be measured, and the share of failed calls is unknown. We'll examine how request and response collection is addressed at each of these stages.

What changes after setup: investigation time and failure classes

Before data collection is set up, investigating a request to an AI service resembles archaeology: the user describes the symptom in words, support reconstructs the chain from screenshots and application logs, and the model request itself is no longer stored anywhere.

Reproduction takes two to three hours, and in half the cases the cause remains unproven — a change to the prompt or response handling is made "just in case."

Request tracing addresses this directly. Each call is assigned an identifier that links the user request, the final prompt, the model response, the invoked tools, and the time of each step. An engineer opens a single card and sees the entire sequence — without guesswork.

In practice, incident investigation time drops from hours to 10–15 minutes, and disputed cases are investigated using factual data rather than a description.

AI service observability yields a second consequence — managing model failure classes.

Instead of a single “service did not respond,” separable categories appear in the statistics: provider timeout, response structure parsing failure, mismatch with the required format, failure of our own validation.

Each class has its own configured behavior: retry, fallback to a backup model, or a proper failure with a clear message. Some failures stop reaching the user simply because they are now detected on our side.

Response quality control changes no less. When every request–response pair is stored and labeled, continuous manual review is unnecessary: selective sample checks and automated rules for format and prohibited wording are enough.

The share of requests requiring human involvement drops to a few percent. We build this on the distributed tracing standard OpenTelemetry — compatible exports remain with you rather than dissolving in someone else's dashboard.

Collection methods: intermediary layer, call wrapper, event stream

The collection method determines exactly where the model call is intercepted and who is responsible for the recording schema.

An intermediary layer sits at the boundary between the service and the model and sees all traffic, including calls from legacy services that no one wants to modify.

A call wrapper lives inside the application, so it knows the user, scenario, and session identifier, but it requires changes at every call point.

An event stream decouples the application and storage: the service only publishes an event, while writing, enrichment, and retention live in a separate pipeline.

In terms of load, the options differ: the intermediary layer adds a network hop, the wrapper adds latency in the model response itself, and the event stream adds a risk of loss when the buffer overflows if flushing to disk is not configured.

Maintenance also differs: when the request format changes, the schema is modified in one place in the gateway, at several points in the code, or in the event contract.

In practice, the methods are combined: the wrapper provides context, the intermediary layer provides coverage completeness, and the event stream provides resilience under load.

Implementation stages: from reconnaissance to handover of policies

The order of steps here matters more than speed: if you start with interception, you get a stream of disconnected records with no owner, retention period, or user linkage. That is why we go from the flow map to the storage schema, and only then enable collection. Each stage ends with an artifact that can be verified before moving to the next.

1. Flow reconnaissance. Inventory: which services call the model, where calls are synchronous and where they are batch, what peak volumes are, and in which request fields personal data appears.
2. Data flow schema. We define the record composition (request, response, duration, error flag), storage location and retention period, access control, and rules for masking sensitive fields before recording.
3. Interception setup. We place collection on a separate layer before the model so the application does not become overgrown with scaffolding. We bring records to a unified format and link them with the request identifier, template version, and initiator account.
4. Validation on live traffic. We enable it gradually: first a small share of requests, then the full stream. We measure latency, record loss rate, and field completeness, comparing the log against actual calls.
5. Team training. We show how to find a specific dialogue, reproduce a failure, and investigate a disputed model response using the stored data, without exports from the production environment.
6. Documentation handover. We provide the flow schema, field descriptions, storage and access policies, and the update procedure when new scenarios appear.

The output is not a one-time setup but a policy: it is clear who owns the log, what it contains, and how to use it during incident investigation.

Artifacts: schemas, access rules, policies, and labeling

The customer receives not a set of settings but working documentation that their team can use to change filters, add fields, and investigate requests without our involvement.

Everything is tied to configurations in the repository: each change is described so that it can be made and verified in a test environment.

• Record flow schema — from the request entry point to storage: where anonymization occurs, at which step the record becomes irreversible, and which services write to the same stream.
• Collection configurations — which request and response fields are recorded, detail levels, sampling rate, tags for service, environment, and model version.
• Access policies — roles and scopes: who can access the response body, who can access only metadata, and how keys are issued and revoked.
• Retention and deletion rules — timeframes in the hot and archive tiers, lifecycle rules, and the procedure for deleting records upon a user request.
• Request investigation policy — how to find the full processing chain by identifier, who is responsible, and within what timeframes.
• Collection documentation — a list of fields with the purpose of each, the list of masked data, and a way to verify that masking worked.
• Acceptance checklist — test scenarios: the record appears, access is limited by role, deletion works, and the schema does not break when the model changes.

The package is stored in the customer's repository and updated together with collection changes — this way, the documentation does not diverge from the actual configuration.

Case study: how support request investigation time was reduced

Before end-to-end logging was implemented, investigating a support request looked like this: the ticket contains the user's question and a strange model response, but which prompt version was used at the time of the call, with what generation parameters, and what happened before and after is unknown.

The cause was sought by trying hypotheses one by one, and some requests were closed with the wording “could not reproduce.” A defect found once could return after the next prompt change and go through the same path again.

What was changed

Each request received an end-to-end identifier that links the ticket in Jira to the model call record.

The record retains the full prompt text with a version label, generation parameters, the complete response, the stop reason, the filters that triggered, and the outcome of retries on failure.

Personal data is masked before recording, access to the archive is role-based, and the retention period is set by policy and requires no manual cleanup.

Defect reproducibility is no longer a lottery: similar failures are grouped by prompt version and error class, and a change is verified against the same set of requests from the archive.

Response reproducibility has become a property of the system rather than the luck of a particular engineer.

How does LLM logging differ from a regular application log?

A regular application log is written around an event: a fixed set of fields, severity level, request identifier, time.

LLM logging is built around a dialogue — the unit of record here is not “the service returned an error” but the complete exchange with the model, which has no stable structure or predictable size.

Request context is the main reason the recording schema cannot be taken from an application log.

A single call includes the system instruction, conversation history, inserted fragments from the knowledge base, and the user's text; the length ranges from a couple of lines to tens of thousands of characters.

That is why the record is broken down into fields with roles, and the parts of the context that actually influenced the response are marked separately — without this, incident investigation turns into reading one gigantic string.

A streaming response breaks the familiar model of “one record — one fact.” Chunks arrive piece by piece, and the log must assemble them into a whole while preserving the time to the first chunk and a flag indicating that generation completed rather than stopped midway.

A mandatory layer is sensitive data filtering: personal data, access keys, and internal documents enter the request along with the user's question. Masking is performed before storage, not at read time; otherwise, what leaked is already sitting in storage.

Storage volume grows not with the number of requests but with context length: a day of operation of a dialogue assistant with conversation history outweighs months of a regular log.

A working schema is to separate a hot tier with full request bodies and a cold tier with metadata and metrics, store failed requests in full, and successful ones selectively.

Common technical concerns before starting work

Concerns before starting are usually not about “whether LLM request logging is needed,” but about how it coexists with production: sensitive data in texts, storage volume, and impact on response time. We address the five objections that come up most often.

Will the logs quickly fill up all storage?

We write not the full request but a normalized structure: session identifier, prompt fingerprint, length, status. The hot window is short — it is enough for incident investigation, and after that only aggregates remain. Volume grows with the number of scenarios, not linearly with traffic.

Will production load handle this?

Writing is asynchronous: the response goes to the user without waiting for storage. If the receiver is unavailable, records are buffered, and the main path is not blocked. Selective recording works for mass scenarios, and full recording for critical ones.

How do I find all steps of a single dialogue?

An end-to-end session identifier is created at the entrance to the chain and passed through all calls, including retries and calls to external services. It is used to assemble the dialogue as a whole, rather than as a set of unrelated lines.

How do I understand what exactly failed?

The record captures the failure class: failure of the model itself, an integration error — transport, access, response parsing — or an error in our code. Metrics and alerts for them are separate, so the on-call engineer immediately sees the search area.

Let's discuss your task: what is needed for an estimate and plan

To come back with an estimate and work plan rather than general considerations, we need inputs about your system. The full list is not mandatory — we'll start with what is already described and clarify the rest during the review.

The more precisely the scope is defined, the fewer assumptions will make it into the implementation plan.

• Call schema: which services call the model, synchronously or through a queue, and whether there is an intermediary gateway between the application and the model.
• Load profile: how many requests per day, peak windows, and the share of streaming responses and long sessions.
• Storage requirements: log lifetime, whether full request and response texts are needed or metadata is sufficient, and where the data physically resides.
• Roles and access: who reads the log, how permissions are separated, and what needs to be masked before recording.
• Request investigation process: how complaints about response quality are currently investigated, who makes the decision, and by when the result is needed.
• Infrastructure: where the components are hosted, which storages and queues are already in use, and whether there are network restrictions.
• Current tools: what already collects metrics and links records between services, so we do not duplicate collection.

Based on these inputs, we describe the logging schema: interception points, recording format, access control, and retention and cleanup rules. After that, we come back with an estimate and work plan — stages, areas of responsibility, and the request investigation policy.