Centralized Logging with ELK Stack for Web Applications
After yet another production incident with 5xx errors, we spent 3 hours digging through logs on ten servers. This repeated every month. When a web application serves thousands of users, logs are generated in huge volumes—up to 50 GB per day on 10 servers. Without centralization, finding an error is like looking for a needle in a haystack. The ELK Stack solves this: all logs flow into a central storage with second-fast search. In our practice, incident response time drops by 70% after ELK adoption. Telegram alerts notify about 5xx, slow requests, application errors. We handle the full cycle: from Elasticsearch deployment to dashboards and alerts. Duration: 2 to 7 days depending on complexity.
How ELK Stack Solves Log Collection and Analysis Problems
ELK is a combination of Elasticsearch (storage and search), Logstash (parsing and transformation), and Kibana (visualization). Filebeat delivers logs from servers. The result is a single entry point for all logs, significantly simplifying log monitoring and error investigation.
Choosing a Scheme: ELK vs EFK vs Without Logstash
| Scheme | Complexity | Performance | Flexibility |
|---|---|---|---|
| ELK | High | Medium | High |
| EFK | Medium | Higher | Medium |
| Without Logstash | Low | High | Low |
Logstash excels in capabilities: it parses legacy log formats using grok, enriches data (geoip, useragent). We use it in 80% of projects. However, Elasticsearch Ingest Pipelines are faster: they process up to 15,000 events/s—3 times more than Logstash (5,000). But Logstash handles unstructured data where Ingest falls short.
How We Set Up ELK for Your Project
Docker Compose for a Test Environment
We use the following compose file:
version: '3.8'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
environment:
- discovery.type=single-node
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=false
- ELASTIC_PASSWORD=changeme
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
volumes:
- esdata:/usr/share/elasticsearch/data
ports:
- "9200:9200"
ulimits:
memlock:
soft: -1
hard: -1
kibana:
image: docker.elastic.co/kibana/kibana:8.13.0
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
- ELASTICSEARCH_USERNAME=kibana_system
- ELASTICSEARCH_PASSWORD=changeme
ports:
- "5601:5601"
depends_on:
- elasticsearch
logstash:
image: docker.elastic.co/logstash/logstash:8.13.0
volumes:
- ./logstash/pipeline:/usr/share/logstash/pipeline
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
ports:
- "5044:5044"
- "5000:5000"
depends_on:
- elasticsearch
volumes:
esdata:
Logstash Pipeline: Parsing Nginx Access and JSON Logs
input {
beats { port => 5044 }
tcp { port => 5000; codec => json_lines }
}
filter {
if [fields][log_type] == "nginx_access" {
grok { match => { "message" => '%{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status_code:int} %{NUMBER:bytes_sent:int} "%{DATA:referrer}" "%{DATA:user_agent}" %{NUMBER:request_time:float}' } }
date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]; target => "@timestamp" }
geoip { source => "client_ip"; target => "geoip" }
useragent { source => "user_agent"; target => "ua" }
mutate { remove_field => ["message", "timestamp"] }
}
if [fields][log_type] == "app_json" {
json { source => "message"; target => "app" }
mutate { remove_field => ["message"] }
}
}
output {
if [fields][log_type] == "nginx_access" {
elasticsearch { hosts => ["http://elasticsearch:9200"]; user => "elastic"; password => "changeme"; index => "nginx-access-%{+YYYY.MM.dd}" }
} else {
elasticsearch { hosts => ["http://elasticsearch:9200"]; user => "elastic"; password => "changeme"; index => "app-logs-%{+YYYY.MM.dd}" }
}
}
Sending Logs from Laravel
Through a custom Monolog handler:
class LogstashLogger
{
public function __invoke(array $config): Logger
{
$handler = new SocketHandler("tcp://{$config['host']}:{$config['port']}");
$handler->setFormatter(new JsonFormatter());
return new Logger('app', [$handler]);
}
}
Now Log::error(...) sends JSON directly to Logstash.
On one project with 10,000 RPS load, we configured a 3-node Elasticsearch cluster with ILM and Logstash with grok patterns for parsing specific application logs. As a result, error search time dropped from 40 minutes to 10 seconds. This allowed the team to respond faster to incidents and reduce MTTR by 65%.
Why ILM Is Mandatory
Without ILM, indices grow uncontrollably, filling the disk in a month. We configure a policy: hot (5 GB or 1 day) → warm (3 days) → cold (30 days) → delete (90 days). This is done via an index template. It's the foundation of cost-effective log storage. Additionally, you can set rollover by size or age to avoid node overload.
Elasticsearch Performance: Practical Tips
- Heap: no more than 50% of RAM and never exceed 31 GB (due to compressed oops)
- Number of shards: 1 shard ≈ 20–40 GB of data. Oversharding is a common mistake
- Slow log:
index.search.slowlog.threshold.query.warn: 2s - Disable swap:
bootstrap.memory_lock: true
Comparison: Logstash vs Ingest Pipelines
| Parameter | Logstash | Ingest Pipelines |
|---|---|---|
| Performance | ~5k events/s | ~15k events/s |
| Flexibility | Grok, enrich, routing | Only simple parsing |
| Complexity | Requires server setup | Built into ES |
For complex transformations, Logstash is irreplaceable. Built-in pipeline processors handle typical tasks but cannot work with arbitrary text patterns.
What's Included in ELK Setup
- Deploying an Elasticsearch cluster with optimal settings (shards, ILM, security)
- Configuring Logstash to parse Nginx, PHP, and application logs
- Connecting Filebeat on all servers
- Creating Kibana dashboards for log monitoring (5xx, latency, traffic)
- Setting up ILM to save disk space
- Integrating alerts to Telegram or Email
- Documentation for operation and team training
Workflow
- Analysis — we study your current logs, sources, and storage requirements
- Design — choose the scheme (ELK/EFK), outline ILM and dashboards
- Implementation — deploy the cluster, configure pipelines
- Testing — verify data ingestion, alerts, search speed
- Deployment — roll out to production, hand over documentation
We guarantee 99.9% SLA for the cluster. Our engineers have over 5 years of experience and have completed 30+ projects. Contact us to discuss your project and schedule an ELK implementation to reduce error search time. Pricing is determined after an infrastructure analysis.