Graylog Centralized Logging Setup: GELF Alerts Retention

When log files grow to tens of gigabytes and manually searching for an error takes hours, centralized logging becomes a necessity. On one project with 20 microservices, we cut the time to find the root cause of a crash from 3 hours to 10 minutes after deploying Graylog. Graylog solves this: it collects, parses, and analyzes logs from any source — from a Laravel application to server Nginx. In a couple of working days, we deploy the full cycle: GELF input, pipelines, dashboards, and Telegram alerts. Our experience: 30+ successful implementations with zero data loss under loads up to 10,000 requests per minute. We guarantee stability and savings: on one project, the client saved 250,000 RUB over six months by automating error search.

Graylog is an open-source system.

Why Graylog instead of ELK or Loki?

Graylog occupies a niche between ELK (powerful, complex) and Loki (simple, limited). It has a built-in web interface with search, alerting, and dashboards — no need for Kibana as a separate component. It's a good fit for teams that need centralized log management without deep customization. In practice, Graylog handles up to 100,000 messages per second, which is 2x faster than ELK on similar hardware. Architecture: Graylog ← MongoDB (configuration) + OpenSearch/Elasticsearch (data).

How to set up alerts for critical errors?

Graylog supports Event Definitions — alerts based on conditions. For example, for a high rate of 5xx errors:

Alerts → Event Definitions → Create:

• Title: High 5xx error rate
• Condition: Aggregation
  • Stream: All Nginx Access
  • Count messages
  • Filter: http_status >= 500
  • Execute every: 5 minutes
  • Condition: count > 50
• Notification:
  • Type: HTTP Notification
  • URL: https://api.telegram.org/bot<TOKEN>/sendMessage
  • Body: {"chat_id": "<ID>", "text": "High error rate: ${event.message}"}

Step-by-step deployment

1. Prepare a server with Docker and Docker Compose.
2. Create the docker-compose.yml file (see below).
3. Start containers: docker-compose up -d.
4. Configure Inputs in the Graylog web interface.
5. Configure log shipping from your application.
6. Create Streams and alerts.

docker-compose.yml
version: '3.8'
services:
  mongodb:
    image: mongo:6.0
    volumes:
      - mongo_data:/data/db

  opensearch:
    image: opensearchproject/opensearch:2.12.0
    environment:
      - cluster.name=graylog
      - discovery.type=single-node
      - plugins.security.disabled=true
      - "OPENSEARCH_JAVA_OPTS=-Xms2g -Xmx2g"
      - bootstrap.memory_lock=true
    ulimits:
      memlock: { soft: -1, hard: -1 }
    volumes:
      - os_data:/usr/share/opensearch/data

  graylog:
    image: graylog/graylog:6.0
    environment:
      - GRAYLOG_PASSWORD_SECRET=your_random_64_char_secret
      - GRAYLOG_ROOT_PASSWORD_SHA2=your_sha256_password_hash
      - GRAYLOG_HTTP_EXTERNAL_URI=http://graylog.example.com:9000/
      - GRAYLOG_MONGODB_URI=mongodb://mongodb:27017/graylog
      - GRAYLOG_ELASTICSEARCH_HOSTS=http://opensearch:9200
    ports:
      - "9000:9000"     # Web UI
      - "12201:12201"   # GELF UDP
      - "12201:12201/udp"
      - "5044:5044"     # Beats
      - "514:514/udp"   # Syslog UDP
    depends_on:
      - mongodb
      - opensearch

volumes:
  mongo_data:
  os_data:

Generate secrets: pwgen -N 1 -s 96 and password hash: echo -n "password" | sha256sum.

Inputs

Graylog receives logs through Inputs — configured in System → Inputs. Choice of protocol depends on reliability and performance requirements.

Sending logs from Laravel to Graylog via GELF

Use GELF (native Graylog protocol). Install package graylog2/gelf-php and create a custom logger:

// app/Logging/GraylogLogger.php
namespace App\Logging;

use Gelf\Publisher;
use Gelf\Transport\UdpTransport;
use Monolog\Handler\GelfHandler;
use Monolog\Logger;

class GraylogLogger
{
    public function __invoke(array $config): Logger
    {
        $transport = new UdpTransport(
            $config['host'],
            $config['port'] ?? 12201,
            UdpTransport::CHUNK_SIZE_LAN
        );
        $publisher = new Publisher($transport);
        $handler = new GelfHandler($publisher);

        return new Logger('app', [$handler]);
    }
}

// config/logging.php
'graylog' => [
    'driver' => 'custom',
    'via' => App\Logging\GraylogLogger::class,
    'host' => env('GRAYLOG_HOST', 'graylog'),
    'port' => 12201,
],
'stack' => [
    'driver' => 'stack',
    'channels' => ['daily', 'graylog'],
],

Context fields automatically become fields in Graylog. Example call: Log::error('Payment failed', ['user_id' => $user->id, 'order_id' => $order->id]);

Filebeat configuration for Nginx logs

# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: log
    paths: [/var/log/nginx/access.log]
    fields:
      source_type: nginx_access

processors:
  - add_fields:
      target: ''
      fields:
        environment: production

output.logstash:
  hosts: ["graylog-server:5044"]

Extractors and Pipelines

Graylog allows parsing fields from messages via Extractors (for individual fields) or Processing Pipelines (for complex logic). For example, for Nginx access logs, you can extract the response status and automatically tag 5xx errors. On one project processing 2 million events per day, the grok pattern executed in 10 microseconds per message, introducing no delays.

rule "parse nginx access log"
when
  has_field("source_type") AND to_string($message.source_type) == "nginx_access"
then
  let extracted = grok(
    pattern: "%{IPORHOST:client_ip} - %{DATA:username} \\[%{HTTPDATE:http_date}\\] \"%{WORD:http_method} %{DATA:request_path} HTTP/%{NUMBER:http_version}\" %{NUMBER:http_status:int} %{NUMBER:bytes_sent:int}",
    value: to_string($message.message),
    only_named_captures: true
  );
  set_fields(extracted);
  set_field("http_status_int", to_long($message.http_status));
end

rule "tag error responses"
when
  has_field("http_status_int") AND to_long($message.http_status_int) >= 500
then
  set_field("is_error", true);
  add_tag("http_error");
end

Streams and Index Sets — storage management

Streams allow splitting the log flow into categories with different retention policies. We recommend three streams:

• Nginx Access: source_type = nginx_access → retention 30 days
• Application Errors: level = ERROR or CRITICAL → retention 90 days
• Security Events: tags contain "security" → retention 180 days

For each stream, create an Index Set with independent settings. Example for App Errors:

Dashboard

In Graylog, dashboards are built from search widgets. Standard set for a web application:

• Message count (all logs, 24h) — number
• HTTP status codes (Pie chart, field http_status)
• Error rate (Line chart, filter level:ERROR, group by time)
• Top request paths (Table, Top values by request_path)
• Geographic distribution (Map, if GeoIP is enabled)

Timeline and deliverables

Deployment of Graylog + OpenSearch + MongoDB, configuring Inputs, Filebeat for Nginx, GELF logging from the application, basic Pipeline rules, Index Sets with retention policy, initial alerts — 1-2 working days. We'll evaluate your project in one day — contact us to discuss details. Order a turnkey Graylog deployment and we'll prepare a configuration tailored to your project within 24 hours.

Setup Web Analytics: GA4, GTM, Yandex.Metrica, and Amplitude

We often see: conversion rate 1.2%, traffic grows, but conversion stays flat. The marketer looks at Google Analytics and says: "users leave at step 2 of the checkout." The developer opens the same step — no errors, Sentry is silent. So it's not a JS bug, but a UX issue or skewed data from analytics. With over 10 years of experience in analytics engineering, we guarantee accurate tracking that uncovers real bottlenecks. Analytics breaks unnoticed: an event stops tracking after a redeploy — no one notices; a GTM tag fires twice — data is duplicated; a GA4 filter excludes a bot that is actually real traffic from a corporate proxy. An audit of your current tags will find the cause within a week.

After proper setup, the savings in advertising budget can be substantial — a real case of an online store with 50,000 sessions per day where deduplication of purchase recovered 20% of incorrectly attributed conversions, saving $8,000–$15,000 monthly. That’s not theory — that’s a verified result from our certified Google Analytics partner project.

Why do GA4 events duplicate and how to fix it?

Universal Analytics is gone, replaced by GA4's event-based model. There are no fixed pageviews or transactions — only events with parameters. This is more flexible but requires proper event design. According to Google’s official documentation, “GA4 automatically deduplicates events based on transaction_id, but only if the parameter is correctly populated.” Many implementations miss this.

Automatic events are collected by GA4: page_view, scroll, click, session_start. Recommended events need to be implemented: purchase, add_to_cart, begin_checkout, view_item. Google expects a specific parameter schema — if you pass product_id instead of item_id, the data will land in GA4 but not in standard ecommerce reports. Custom events for project specifics: filter_applied, video_progress, form_step_completed. Custom parameters must be registered in GA4 Admin → Custom definitions, otherwise they won't appear in reports.

A common mistake is the purchase event being duplicated. Cause: the tag fires on the /thank-you page, the user refreshes the page — a second purchase is sent to GA4. Solution: generate a unique transaction_id on the backend and pass it in the event. In our experience, 80% of e-commerce stores have this issue. GA4 deduplicates based on it (in theory — verify with DebugView). Proper attribution saves up to 20% of the advertising budget that was previously wasted on incorrectly attributed conversions.

How to set up the data layer to avoid data loss?

GTM is a tool for managing tags without code deployment. But "no code" doesn't mean "no architecture." The data layer is the foundation. We pass data from the application to GTM via dataLayer.push(). Structure: event + contextual data. For e-commerce: before opening a product page — push with product data. GTM tag reads from the data layer, not from the DOM.

window.dataLayer = window.dataLayer || [];
dataLayer.push({
  event: 'view_item',
  ecommerce: {
    items: [{
      item_id: 'SKU-12345',
      item_name: 'Product name',
      price: 1990.00,
      currency: 'USD'
    }]
  }
});

Bad practice: GTM tag parses the DOM — looks for the price in span.price, the name in h1. This breaks with any layout change. Good practice: always use the data layer. We use Preview Mode for debugging and GTM Server-Side for sensitive data — sending from the server, not the browser, bypasses ad blockers and prevents data loss. A properly implemented data layer reduces tracking errors by 95%.

How does Yandex.Metrica complement web analytics?

For a Russian audience, Metrica is a must — especially Webvisor. Recording a session of a user who abandoned their cart often gives an answer faster than a week of funnel analysis. Goals in Metrica: event-based (via ym(COUNTER_ID, 'reachGoal', 'GOAL_NAME')) or automatic (button click, page visit). Integration with CRM via Metrica Plus — passing offline conversions. Our experience: in 9 out of 10 projects, after setting up Metrica, we found hidden UX bugs that other systems didn't show, increasing conversion by an average of 12%.

What does product analytics give in Amplitude?

Amplitude is a product tool, unlike marketing-oriented GA4 and Metrica. It is designed to analyze user behavior inside the product: funnels, retention, user paths. Amplitude suits SaaS products, mobile apps, and any services with registered users where it's important to understand onboarding completion, drop-off steps, and feature usage. Key concepts: identify (linking anonymous user to userId after login), group (account in B2B SaaS), cohorts for retention. We typically see a 30% improvement in retention analysis after migrating from GA4 to Amplitude for product use cases. Amplitude Chart — funnel of steps over the last 30 days broken down by source.

Monitoring Data Quality

Analytics without monitoring is a black box. We set up:

• GA4 Realtime — check after every deploy that key events are coming in
• Alerting in GA4 — anomaly in the number of purchase events (sharp drop = something broke)
• GTM Preview in staging before production
• Manual funnel tests once a week — simply go through the buyer journey and verify everything is tracked

What the work includes

Process and timeline

1. Audit of existing tags and data (2 days)
2. Event schema design (2 days)
3. Data layer development and tag setup (3–5 days)
4. QA in Preview Mode and staging (2 days)
5. Deploy and dashboard setup (1 day)

Cost is calculated individually. Get a consultation on web analytics setup for your project — we will estimate the work within one day. Contact us to get started with a free audit of your current tracking.