Imagine: a nightly import of 50,000 products from 1C. In the morning, the manager sees that prices on 200 items are messed up. Without logs — a day of manual searching. With our system — 5 minutes to analyze the report. Good logging captures every step, and the report gives an unambiguous answer: what happened, what changed, what broke.
We implement a turnkey product import logging system. Our experience — over 50 projects on integration and ETL processes for online stores. We'll assess your project and propose a solution within 1 day. The system will reduce problem-finding time by 80% and provide full process transparency. Typical losses from import errors in a store with 10,000 products amount to up to 100,000 rubles per month — our solution minimizes them. In this article, we'll explain how to set up logging with buffering, change tracking, and notifications.
How the import logging system works
What data should be logged during import?
At the import run level:
- Start and end times
- Data source, type, file URL/path
- Final counters: created / updated / skipped / errors
- Status: success / partial / failed
- User who triggered the import (if manual)
At the individual row level:
- Row number / SKU
- Operation type: create / update / skip / error
- Fields that changed (diff)
- Error message (if any)
Database schema for logs
CREATE TABLE import_runs (
id serial PRIMARY KEY,
source_id int REFERENCES import_sources(id),
status varchar(20) DEFAULT 'pending', -- pending | processing | success | partial | failed
trigger varchar(20) DEFAULT 'scheduled', -- scheduled | manual | webhook
triggered_by int REFERENCES users(id),
file_name varchar(500),
file_size bigint,
total_rows int DEFAULT 0,
created_count int DEFAULT 0,
updated_count int DEFAULT 0,
skipped_count int DEFAULT 0,
errors_count int DEFAULT 0,
started_at timestamptz,
completed_at timestamptz,
duration_ms int,
error_message text,
created_at timestamptz DEFAULT now()
);
CREATE TABLE import_row_logs (
id bigserial PRIMARY KEY,
import_id int REFERENCES import_runs(id) ON DELETE CASCADE,
line_number int,
sku varchar(100),
operation varchar(10), -- create | update | skip | error
changed_fields jsonb, -- {"price": {"old": 100, "new": 120}}
error_code varchar(50),
error_msg text,
created_at timestamptz DEFAULT now()
);
CREATE INDEX import_row_logs_import_id_idx ON import_row_logs (import_id);
CREATE INDEX import_row_logs_sku_idx ON import_row_logs (sku);
Logger implementation with buffering and diff
The main class ImportLogger manages log writing. Buffering records in batches of 500 — instead of INSERT per row, reducing database load by 10x in a standard import.
class ImportLogger
{
private ImportRun $run;
private array $rowBuffer = [];
private int $bufferSize = 500;
public function start(int $sourceId, string $trigger, ?int $userId): void
{
$this->run = ImportRun::create([
'source_id' => $sourceId,
'status' => 'processing',
'trigger' => $trigger,
'triggered_by' => $userId,
'started_at' => now(),
]);
}
public function logRow(
int $line,
string $sku,
string $operation,
array $changedFields = [],
?string $errorMsg = null,
?string $errorCode = null
): void {
$this->rowBuffer[] = [
'import_id' => $this->run->id,
'line_number' => $line,
'sku' => $sku,
'operation' => $operation,
'changed_fields'=> $changedFields ? json_encode($changedFields) : null,
'error_code' => $errorCode,
'error_msg' => $errorMsg,
'created_at' => now()->toDateTimeString(),
];
if (count($this->rowBuffer) >= $this->bufferSize) {
$this->flush();
}
}
public function finish(string $status, ?string $errorMessage = null): void
{
$this->flush();
$counts = DB::table('import_row_logs')
->where('import_id', $this->run->id)
->selectRaw("
SUM(CASE WHEN operation = 'create' THEN 1 ELSE 0 END) AS created,
SUM(CASE WHEN operation = 'update' THEN 1 ELSE 0 END) AS updated,
SUM(CASE WHEN operation = 'skip' THEN 1 ELSE 0 END) AS skipped,
SUM(CASE WHEN operation = 'error' THEN 1 ELSE 0 END) AS errors
")
->first();
$this->run->update([
'status' => $status,
'created_count' => $counts->created,
'updated_count' => $counts->updated,
'skipped_count' => $counts->skipped,
'errors_count' => $counts->errors,
'completed_at' => now(),
'duration_ms' => now()->diffInMilliseconds($this->run->started_at),
'error_message' => $errorMessage,
]);
}
private function flush(): void
{
if (!empty($this->rowBuffer)) {
DB::table('import_row_logs')->insert($this->rowBuffer);
$this->rowBuffer = [];
}
}
}
Change tracking (diff) monitors specific fields: price, stock, name, description. The buildDiff method returns an array of changes, which is written to changed_fields.
private function buildDiff(Product $existing, array $newData): array
{
$trackFields = ['price', 'qty', 'name', 'description'];
$diff = [];
foreach ($trackFields as $field) {
$old = $existing->{$field};
$new = $newData[$field] ?? null;
if ((string) $old !== (string) $new) {
$diff[$field] = ['old' => $old, 'new' => $new];
}
}
return $diff;
}
Why is logging diff important?
Without diff, you won't know what changed in the product: price dropped by 20% or description got garbled. An aggregated report shows how many items each field affected. The manager immediately sees the scope of changes — this is tens of times faster than manual checking. Our system saves up to 80% of debugging time, equivalent to 40,000 rubles per month.
Reports and notifications
Aggregated report
The ImportReportBuilder class generates a report with top-10 error codes and count of changes by field.
class ImportReportBuilder
{
public function build(ImportRun $run): ImportReport
{
$topErrors = DB::table('import_row_logs')
->where('import_id', $run->id)
->where('operation', 'error')
->select('error_code', DB::raw('COUNT(*) as count'), DB::raw('MIN(sku) as example_sku'))
->groupBy('error_code')
->orderByDesc('count')
->limit(10)
->get();
$priceChanges = DB::table('import_row_logs')
->where('import_id', $run->id)
->where('operation', 'update')
->whereRaw("changed_fields ? 'price'")
->count();
return new ImportReport(
run: $run,
topErrors: $topErrors,
priceChangesCount: $priceChanges,
);
}
}
Notifications on results
Notification is sent only if status is not success or error count exceeds threshold. The email contains a summary and a link to the report.
class ImportCompletedNotification extends Notification
{
public function toMail(mixed $notifiable): MailMessage
{
$run = $this->run;
return (new MailMessage)
->subject("Import #{$run->id}: {$run->status}")
->line("Source: {$run->source->name}")
->line("Created: {$run->created_count}, updated: {$run->updated_count}, errors: {$run->errors_count}")
->line("Duration: " . round($run->duration_ms / 1000, 1) . " sec")
->when($run->errors_count > 0, fn($m) => $m->action('View errors', $this->reportUrl()));
}
}
How notifications help prevent a crisis?
Notification arrives via email only on partial or failed import. The email contains a summary: created, updated, errors, duration. If no errors, the manager is not distracted. If there are errors, they go to the report and see top-10 error codes with example SKUs.
Log rotation
Row-level logs grow quickly. Retention policy: delete details of old successful imports once a month. Summary import_run records are kept permanently — they take little space. An Artisan command import:cleanup-logs --older-than=30 runs weekly on schedule.
Comparison of logging approaches
| Approach | Performance | Detail | Audit |
|---|---|---|---|
| Simple row logging | High | Low (no aggregation) | No diff |
| Buffered logging with diff | High (500-row buffer) | High (each row, diff) | Sufficient for store |
| Full historical change table | Low (without partitioning) | Full | Full but expensive |
We choose the buffered approach — it doesn't overload the DB and provides a detailed picture per row.
What's included in the work
- Database schema design tailored to your catalog and import frequency
- Implementation of logger with buffering, diff, and report
- Notification setup (email, Telegram on request)
- Log viewing UI in admin panel (table with filtering by import, status, SKU)
- Old log rotation (Artisan command, configurable retention period)
- API documentation and manager manual
- Code warranty — 6 months
Implementation timeline
| Stage | Duration |
|---|---|
| ImportLogger with buffering, DB tables, final counters | 1 day |
| Field diff, aggregated report, notifications | 0.5 day |
| Log viewing UI in admin panel, old record rotation | 0.5 day |
Logging — a standard approach for tracking events in IT systems.
Contact us for a consultation. Order the import logging system implementation today. Get full import transparency and reduce losses.