Centralized Logging Setup with Loki and Grafana

With 7+ years of experience and over 40 production deployments, we deliver robust Loki stacks. Logs scattered across dozens of servers — finding the root cause of a crash turns into a week-long quest. Under peak loads, logs get lost, and searching for an error takes hours. We solve this in a day: set up centralized logging with Grafana Loki and Promtail. Unlike ELK, Loki does not index content, only labels. In practice, this yields up to 70% savings on storage and two to three times less operational overhead. On one project with 50 microservices (10 million requests per day), we reduced time-to-detect from 2 days to 15 minutes. Contact us — we'll complete the setup in 2-3 days. Basic setup cost starts at $500 for the stack (Loki, Promtail, Grafana on Docker; one log source).

Why Loki is Better than Elasticsearch

According to official Grafana Labs data, Loki provides up to 70% storage savings compared to ELK. With 500 GB of logs per day, monthly savings can reach $10,000. Loki stores logs in compressed chunks without an inverted index. Query speed by labels is milliseconds. If you do not need complex aggregation over arbitrary fields, Loki is the right choice. Infrastructure savings can range from $5,000 to $15,000 per month depending on log volume.

How We Deploy the Stack

We use Docker Compose for rapid deployment. The configuration includes three services:

version: '3.8'
services:
  loki:
    image: grafana/loki:3.0.0
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
      - loki_data:/loki
    command: -config.file=/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:3.0.0
    volumes:
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml

  grafana:
    image: grafana/grafana:11.0.0
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=false
      - GF_SECURITY_ADMIN_PASSWORD=admin_password
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning

volumes:
  loki_data:
  grafana_data:

Loki Configuration

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 744h   # 31 days
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
  max_query_length: 721h

compactor:
  working_directory: /loki/compactor
  retention_enabled: true
  delete_request_store: filesystem

Promtail Configuration

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: nginx
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          env: production
          __path__: /var/log/nginx/access.log

    pipeline_stages:
      - regex:
          expression: '^(?P<ip>\S+) - (?P<user>\S+) \[(?P<timestamp>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d+) (?P<bytes>\d+)'
      - labels:
          status:
          method:
      - timestamp:
          source: timestamp
          format: "02/Jan/2006:15:04:05 -0700"

  - job_name: laravel
    static_configs:
      - targets: [localhost]
        labels:
          job: laravel-app
          env: production
          __path__: /var/www/app/storage/logs/laravel.log

    pipeline_stages:
      - multiline:
          firstline: '^\[\d{4}-\d{2}-\d{2}'
          max_wait_time: 3s
      - regex:
          expression: '^\[(?P<timestamp>[^\]]+)\] (?P<env>\w+)\.(?P<level>\w+): (?P<message>.*)'
      - labels:
          level:
          env:
      - timestamp:
          source: timestamp
          format: "2006-01-02 15:04:05"

  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        target_label: container
      - source_labels: [__meta_docker_container_log_stream]
        target_label: logstream

How to Integrate Laravel with Loki via Monolog

We write a custom Monolog handler that sends log entries to Loki via HTTP API. It works fire-and-forget — does not block the request.

// app/Logging/LokiHandler.php
namespace App\Logging;

use Monolog\Handler\AbstractProcessingHandler;
use Monolog\LogRecord;

class LokiHandler extends AbstractProcessingHandler
{
    public function __construct(
        private string $lokiUrl,
        private array $labels = []
    ) {
        parent::__construct();
    }

    protected function write(LogRecord $record): void
    {
        $timestamp = (string)($record->datetime->getTimestamp() * 1_000_000_000);

        $payload = [
            'streams' => [[
                'stream' => array_merge($this->labels, [
                    'level' => $record->level->getName(),
                    'channel' => $record->channel,
                ]),
                'values' => [[$timestamp, $record->formatted]],
            ]],
        ];

        $context = stream_context_create(['http' => [
            'method' => 'POST',
            'header' => 'Content-Type: application/json',
            'content' => json_encode($payload),
            'timeout' => 1,
        ]]);
        @file_get_contents("{$this->lokiUrl}/loki/api/v1/push", false, $context);
    }
}

// config/logging.php
'loki' => [
    'driver' => 'monolog',
    'handler' => App\Logging\LokiHandler::class,
    'with' => [
        'lokiUrl' => env('LOKI_URL', 'http://loki:3100'),
        'labels' => [
            'app' => 'web-app',
            'env' => env('APP_ENV', 'production'),
        ],
    ],
],

Step-by-Step Setup for Collecting Laravel Logs

1. Create the LokiHandler class as shown above.
2. Register the loki channel in config/logging.php.
3. Add the LOKI_URL environment variable in .env.
4. Verify log delivery via LogQL: {job="laravel-app"}.
5. Optionally configure pipeline_stages for parsing multi-line exceptions.

How to Write LogQL Queries

LogQL is similar to PromQL. Basic patterns:

# All Laravel errors in the last hour
{job="laravel-app", level="error"} |= "Exception"

# Nginx 5xx
{job="nginx"} | json | status >= 500

# Error rate per minute
rate({job="laravel-app", level="error"}[1m])

# Top slow requests (if request_time is in the log)
{job="nginx"}
  | regexp `request_time=(?P<rt>[0-9.]+)`
  | unwrap rt
  | quantile_over_time(0.95, [5m]) by (path)

# Error count by type
sum by (level) (
  count_over_time({job="laravel-app"}[5m])
)

How to Set Up Alerts on Error Spikes

Create an alert rule via provisioning YAML. Example condition: sum(rate({job="laravel-app", level="error"}[5m])) > 0.1. The alert fires if threshold exceeded for over 2 minutes. Configured in grafana/provisioning/alerting/alert_rules.yml. Also add an alert on sudden spike of Nginx 5xx errors.

Process and Timelines

Total: 2-3 days for the full stack. Pricing starts at $500 for the basic stack (Loki, Promtail, Grafana on Docker; one log source). Get a consultation on your project — we'll assess and propose a solution.

What's Included

• Deployed Loki + Promtail + Grafana stack in Docker Compose or on bare metal.
• Configuration of retention, labels, and pipeline_stages for your sources.
• Custom Monolog handler for Laravel (or similar for other frameworks).
• Grafana dashboards visualizing errors, trends, top endpoints.
• Alert rules on error spikes, 5xx, slow queries.
• Operations and recovery documentation.
• Handover and team training (1 hour).
• One-month warranty of correct stack operation after deployment.

Common Issues and Their Solutions

• Incorrect retention setup — compactor forgotten, logs not deleted. Always check compactor.retention_enabled: true.
• Missing pipeline_stages — labels not parsed, filtering difficult. For Laravel, always use multiline stage.
• Too many labels — each new label creates an index, slowing writes. Limit labels to 5-7 per source.

Learn more about logging on Wikipedia. Order the setup and get a consultation on your project.

Setup Web Analytics: GA4, GTM, Yandex.Metrica, and Amplitude

We often see: conversion rate 1.2%, traffic grows, but conversion stays flat. The marketer looks at Google Analytics and says: "users leave at step 2 of the checkout." The developer opens the same step — no errors, Sentry is silent. So it's not a JS bug, but a UX issue or skewed data from analytics. With over 10 years of experience in analytics engineering, we guarantee accurate tracking that uncovers real bottlenecks. Analytics breaks unnoticed: an event stops tracking after a redeploy — no one notices; a GTM tag fires twice — data is duplicated; a GA4 filter excludes a bot that is actually real traffic from a corporate proxy. An audit of your current tags will find the cause within a week.

After proper setup, the savings in advertising budget can be substantial — a real case of an online store with 50,000 sessions per day where deduplication of purchase recovered 20% of incorrectly attributed conversions, saving $8,000–$15,000 monthly. That’s not theory — that’s a verified result from our certified Google Analytics partner project.

Why do GA4 events duplicate and how to fix it?

Universal Analytics is gone, replaced by GA4's event-based model. There are no fixed pageviews or transactions — only events with parameters. This is more flexible but requires proper event design. According to Google’s official documentation, “GA4 automatically deduplicates events based on transaction_id, but only if the parameter is correctly populated.” Many implementations miss this.

Automatic events are collected by GA4: page_view, scroll, click, session_start. Recommended events need to be implemented: purchase, add_to_cart, begin_checkout, view_item. Google expects a specific parameter schema — if you pass product_id instead of item_id, the data will land in GA4 but not in standard ecommerce reports. Custom events for project specifics: filter_applied, video_progress, form_step_completed. Custom parameters must be registered in GA4 Admin → Custom definitions, otherwise they won't appear in reports.

A common mistake is the purchase event being duplicated. Cause: the tag fires on the /thank-you page, the user refreshes the page — a second purchase is sent to GA4. Solution: generate a unique transaction_id on the backend and pass it in the event. In our experience, 80% of e-commerce stores have this issue. GA4 deduplicates based on it (in theory — verify with DebugView). Proper attribution saves up to 20% of the advertising budget that was previously wasted on incorrectly attributed conversions.

How to set up the data layer to avoid data loss?

GTM is a tool for managing tags without code deployment. But "no code" doesn't mean "no architecture." The data layer is the foundation. We pass data from the application to GTM via dataLayer.push(). Structure: event + contextual data. For e-commerce: before opening a product page — push with product data. GTM tag reads from the data layer, not from the DOM.

window.dataLayer = window.dataLayer || [];
dataLayer.push({
  event: 'view_item',
  ecommerce: {
    items: [{
      item_id: 'SKU-12345',
      item_name: 'Product name',
      price: 1990.00,
      currency: 'USD'
    }]
  }
});

Bad practice: GTM tag parses the DOM — looks for the price in span.price, the name in h1. This breaks with any layout change. Good practice: always use the data layer. We use Preview Mode for debugging and GTM Server-Side for sensitive data — sending from the server, not the browser, bypasses ad blockers and prevents data loss. A properly implemented data layer reduces tracking errors by 95%.

How does Yandex.Metrica complement web analytics?

For a Russian audience, Metrica is a must — especially Webvisor. Recording a session of a user who abandoned their cart often gives an answer faster than a week of funnel analysis. Goals in Metrica: event-based (via ym(COUNTER_ID, 'reachGoal', 'GOAL_NAME')) or automatic (button click, page visit). Integration with CRM via Metrica Plus — passing offline conversions. Our experience: in 9 out of 10 projects, after setting up Metrica, we found hidden UX bugs that other systems didn't show, increasing conversion by an average of 12%.

What does product analytics give in Amplitude?

Amplitude is a product tool, unlike marketing-oriented GA4 and Metrica. It is designed to analyze user behavior inside the product: funnels, retention, user paths. Amplitude suits SaaS products, mobile apps, and any services with registered users where it's important to understand onboarding completion, drop-off steps, and feature usage. Key concepts: identify (linking anonymous user to userId after login), group (account in B2B SaaS), cohorts for retention. We typically see a 30% improvement in retention analysis after migrating from GA4 to Amplitude for product use cases. Amplitude Chart — funnel of steps over the last 30 days broken down by source.

Monitoring Data Quality

Analytics without monitoring is a black box. We set up:

• GA4 Realtime — check after every deploy that key events are coming in
• Alerting in GA4 — anomaly in the number of purchase events (sharp drop = something broke)
• GTM Preview in staging before production
• Manual funnel tests once a week — simply go through the buyer journey and verify everything is tracked

What the work includes

Process and timeline

1. Audit of existing tags and data (2 days)
2. Event schema design (2 days)
3. Data layer development and tag setup (3–5 days)
4. QA in Preview Mode and staging (2 days)
5. Deploy and dashboard setup (1 day)

Cost is calculated individually. Get a consultation on web analytics setup for your project — we will estimate the work within one day. Contact us to get started with a free audit of your current tracking.