Why bot trade logging breaks at peaks
Trade logging most often breaks not in a calm market, but at a spike: the strategy generates dozens of times more events than usual, while recording goes through an in-memory buffer with batch flush to disk. While the stream is steady, everything looks intact.
As soon as the buffer overflows or the process restarts due to health, some records are lost irrevocably — silently, with no error in the log itself.
The second failure is a break in the record sequence. Each event must have an end-to-end number and a link to the previous one; otherwise the log ceases to be a reproducible chain.
If numbering is end-to-end only within a process, not within a trading session, after a restart the numbers start over and overlap the old ones — during analysis it is impossible to distinguish two different orders with the same identifier.
The third and most expensive is signal-execution divergence. The signal is recorded on the strategy side, execution on the broker side, each with its own time source.
Result: the log shows the entry condition, but not the actual price and placement time, and the stop in the report triggered differently from the record.
Reconstructing the course of a trade turns into guesswork, and there is nothing to answer the question "why did the position close at the wrong place".
Below we break down how this is closed.
What an end-to-end trade trace in a trading system provides
An end-to-end trade trace is a record for every order state: generated strategy signal, send moment, venue response, fact and volume of execution, partial close, cancellation.
Events are stitched together by a single correlation identifier and enter the log append-only — history cannot be rewritten retroactively, only a new event can be added. Such a log becomes the source of truth both for failure investigation and for reporting.
Incident investigation time is determined by how long it takes to answer the question "why did the bot do exactly that".
Using the identifier, the entire chain from signal to execution is retrieved: you can see which rule fired, what data was at the input, what the venue responded. Investigation moves from manual log reconciliation to a single query — tens of minutes instead of half a working day.
Restoring transaction history relies on the log being written in two places: next to the trading core, for instant access, and in an analytical store on ClickHouse — for month-by-month queries.
Divergence between the system position and venue data stops being inexplicable: the share of trades with fully recoverable history approaches the limit, except when the venue did not return a response at all.
Order execution control and trading strategy verification rely on the same record.
The log is replayed on accumulated history: the same set of rules and parameters produces the same sequence of decisions as in live trading, so a logic change is verified in one run instead of weeks of observation.
At the same time, a class of failures that previously could not be closed by anything goes away — the dispute over whose failure caused the divergence.
Log write modes: comparison of options
The log write mode determines what happens to data on channel break, process restart, or disk overflow.
Synchronous and batch writing are a trade-off between per-trade latency and the risk of losing recent events, so we choose the mode not "by eye" but from the requirements for record integrity in a specific environment.
| Option | When it fits | Limitations |
|---|---|---|
| Synchronous write | Low trade frequency, strict completeness requirements: an event is considered confirmed only after being committed | Each trade waits for the write to storage; if the receiver is unavailable, latency grows, and some signals arrive late |
| Buffered (batch) write | Dense flow of events and quotes, event buffering and grouping before sending are needed | On abnormal termination, the last unsent batch is lost — timeout flushing and buffer size control are needed |
| Local file log on the bot side | Autonomous mode, unstable channel: writing goes to a file next to the process | Files grow, rotation and cleanup are needed; incident investigation requires export and comparison with the centralized log |
| Hybrid scheme | Production where both reaction speed and history completeness matter: local commit plus asynchronous sending to the centralized log | More complex setup: two storage points, merge and deduplication rules, monitoring for divergence between them |
Hybrid option we usually set by default: an event first lands in a local segment, and from there goes in batches to the centralized log.
Record integrity relies on the event sequence number and retransmission of unconfirmed batches, so a communication break does not turn into a hole in history.
Switching between modes does not require rewriting the bot — it is set by configuration, and this allows increasing write frequency without rebuilding trading logic.
How we build trade logging: work stages
Trade logs are not needed just for logs: they are used to investigate divergences between strategy intent and actual execution, restore the chain order → execution → cancellation, and confirm operations before auditors.
Therefore the record format is approved before connecting the source — otherwise history would have to be rewritten retroactively.
- Audit of event sources. We inventory where data comes from: trading module, broker gateway, quote stream, risk control. For each source, we record the event moment, timestamp precision, and the attribute by which a record is recognized again.
- Record format design. We agree on the schema: trade identifier, time, instrument, side, volume, price, status, rejection reason, link to parent event. Separately — versioning rules so records from previous periods are read by new code.
- Connecting event sources. We install adapters, write the log append-only, and buffer to disk: a channel break must not "eat" events, and redelivery must not create duplicates.
- Log completeness verification. We reconcile the number of records against independent counters — broker confirmations, balance movement, number of sent orders. We replay the log on historical data and reproduce a trading day.
- Failure behavior verification. We examine restart, communication break, and disk overflow: what is lost, what is restored, how long reconciliation takes after restart.
- Handover of the system to the customer. We hand over the storage schema, accesses, rotation and archiving regulations, incident investigation procedure — and train the team to read the log without us.
Log artifacts: schemas, rules, regulations
A log without description turns into a set of tables that no one dares touch six months later. Therefore, together with the system, we hand over a set of artifacts by which records can be read, extended, and investigated without the author's involvement.
- Event record format. Fields, types, requiredness, empty value rule, and schema versioning: new fields are added without breaking reading of old records.
- Event flow schema. From order placement to execution: write points, behavior on channel break, the rule "an event is either recorded in full or not recorded at all".
- Log access policies. Roles and rights, separation of reading and export, prohibition of editing records, separate environment for sensitive fields.
- Record retention periods. Hot window for daily investigation, archive for reconciliations, automatic cleanup after expiry with a log of the deletions themselves.
- Divergence investigation regulations. How divergence between intent and execution is recorded, who escalates, within what time, and what goes into the fix list.
- Rules for reconciliation with venue reports. Matching keys, source priority on conflict, procedure for investigating mismatches.
- Recovery instructions. From which records and in what order position state is restored after a failure and how completeness is checked.
Each artifact is versioned and lives next to the code: a logic change and a description update are one task, not two unrelated matters. This is what makes the log suitable for handover to another team and for auditing.
Case: how a trade divergence was found via the log
The divergence was not noticed immediately: the bot showed the position closed, while at the broker it remained open.
At that time, the log recorded only the fact that an order was sent — without the broker identifier, the state before and after, and the version of risk-control rules, so the cause was sought by indirect signs and took on average a day and a half.
The share of unexplained divergences held at about 9% of the daily number of trades, and some operations were closed manually.
What was changed in the log schema
Each trade was assigned an end-to-end identifier, the same in the bot and in broker responses.
The log stopped being overwritten: it is only appended with state transitions (created, sent, confirmed, partially executed, closed), and next to them lie raw broker responses, the version of risk rules at the time of the trade, and a timestamp.
Storage — PostgreSQL with immutable records and a chain checksum: substitution or a lost fragment is visible immediately, not at a monthly reconciliation.
Replay from the log appeared: bot state is restored to any timestamp, so a failure is localized by comparing two runs. The first investigation took 20 minutes instead of a day and a half — the cause turned out to be a conflict between cancellation and partial execution.
Result in metrics
The share of unexplained divergences fell from 9% to 0.4%, cause localization time — from 36 hours to 20 minutes. 340 operations per quarter were restored; not one had to be reconciled manually.
What to do if the bot already writes events to files?
A file log on a trading server survives until the first rotation failure. The bot writes to the active file, the old one is deleted on schedule — and with it go trade records for the previous period that no one will restore.
The second source of loss is buffering: on abnormal process termination, the last lines do not reach disk, and a gap remains in the file exactly where strategy behavior needs to be investigated.
| Option | When it fits | Limitations |
|---|---|---|
| Keep the file log | One process, manual investigation, short retention | Losses on rotation and crash, no unified search across multiple bots |
| Add record export | An archive is needed without rebuilding the bot, the file remains primary | Delay between event and archive; if export breaks, the gap remains |
| Centralized storage | Multiple bots, disputed trade investigation, long-term storage | Requires refinement of event sending and delivery control |
Event log migration begins with transferring already accumulated files: records are parsed line by line, brought to a unified schema, and loaded in batches so as not to mix formats of old and new events.
Then the bot is switched to sending events immediately on write, while the file remains a fallback copy during the transition.
The key point is delivery control: if an event is not confirmed by storage, it remains in the local queue and goes again after communication is restored. Thus file rotation stops being a loss point, and record export works as a catch-up channel, not the only one.
Technical doubts before starting implementation
Before the start, the same questions come up: will the trading loop degrade, what will happen to lost events, who will get access to the log, and how to make sure records are not duplicated. Below is how each of them is closed.
Will logging create load on the trading loop?
We move writing off the critical path: the bot hands the event to an asynchronous queue, and storage goes through a separate loop. Trade decision time does not change, and queue depth and failure share are visible on Prometheus graphs.
What to do with missed events after a failure?
Each event gets a monotonic identifier, and the bot stores the last confirmed one. After restart, it resends the missed range and reconciles it with execution history — the log converges without manual stitching.
How is access to the trade log separated?
Roles are separated: the strategist sees their signals, risk control sees aggregates, audit sees the full log. Access is granted via service accounts with a limited term, and export is recorded with a separate trace.
How to make sure records are not duplicated?
An idempotent key made from the trade and event identifier cuts off repetition at the level of the storage unique index. Additionally, reconciliation is performed: the number of records for the period against the number of executed orders.
Can it be implemented without stopping trading?
Yes. First the log is enabled in shadow mode: it writes but does not affect bot decisions, then report reading is switched over. Rollback is removing the flag; trading logic is not affected.
Let's discuss the task and come back with an estimate and plan
So that the estimate and work plan rely on facts rather than assumptions, the first conversation is enough to show how trades are recorded now.
We look at event sources, the storage format, and what questions the log must close: investigation of erroneous entries, restoration of history for an arbitrary period, reconciliation with broker statements.
For a substantive estimate, source data is needed:
- Trading loop schema: where the decision is made, where the order is placed, which system confirms execution.
- Examples of current records — raw exports from tables or logs, even incomplete and heterogeneous.
- List of events mandatory for recording: order, execution, partial close, cancellation, broker rejection.
- Storage and access requirements: record lifetime, who reads them, whether protection against retroactive editing is needed.
- Environment constraints: how the bot is deployed, which external systems are unavailable in the test environment.
- Acceptance criteria: record completeness, history restoration speed, behavior on communication break.
After the conversation, we come back with an estimate and work plan: stages, log composition, intermediate checkpoints, and start order.
Usually it is like this — we agree on the event record schema, assemble the log on historical data, test on failures, and hand it over with a format description. Describe the task — we will come back with a plan.
