Centralized Mobile Backend Logging with ELK Stack: Setup Guide
Imagine your mobile backend generates tens of thousands of log lines per minute across multiple servers. A production error hits — you spend hours grepping via SSH. ELK Stack (Elasticsearch, Logstash, Kibana) turns that chaos into centralized search with filters in seconds. We've integrated such a solution for over 30 projects, from startups to enterprise, cutting diagnosis time by 10x. Here's how it works.
Why Set Up ELK Stack for Mobile Backend Logging?
Because the time and resource savings are obvious. On one project (gaming, 5M users), we reduced infrastructure costs by 30% and sped up error search from hours to seconds. Developers stopped digging through logs manually — Kibana dashboards show error rate in real-time, and alerts land in Telegram when thresholds are exceeded. The average project pays for itself in 2 months through reduced incident time. Guaranteed results or your money back.
What We Configure
The log collection stack consists of three parts: structured logging at the application level, an agent for collection and forwarding (Fluent Bit or Logstash), Elasticsearch storage with search engine, and Kibana UI.
Structured Logs
Applications must write logs in JSON — not plain text. A line like [ERROR] Jan 15 14:23:11 UserService: null pointer is junk for Elasticsearch. JSON-formatted logs are parseable and filterable:
import structlog
logger = structlog.get_logger()
def authenticate_user(user_id: str, device_id: str):
logger.info(
"auth_attempt",
user_id=user_id,
device_id=device_id,
platform="ios",
app_version="3.2.1",
)
try:
token = auth_service.verify(user_id)
logger.info("auth_success", user_id=user_id, token_expires_in=3600)
return token
except InvalidTokenError as e:
logger.warning("auth_failed", user_id=user_id, reason=str(e))
raise
import "github.com/rs/zerolog/log"
log.Error().
Str("user_id", userID).
Str("device_id", deviceID).
Str("endpoint", "/api/v1/auth").
Int("status_code", 401).
Dur("duration_ms", elapsed).
Msg("authentication failed")
Mandatory fields in every log: timestamp (ISO 8601), level, service, request_id (for request tracing across services), user_id (if applicable). request_id is a UUID generated at the incoming request middleware and propagated to all child calls via context.
| Field | Type | Description |
|---|---|---|
| timestamp | date | ISO 8601 |
| level | keyword | error, warn, info, debug |
| service | keyword | Service name |
| user_id | keyword | User identifier |
| request_id | keyword | Request UUID |
| message | text | Event description |
| duration_ms | long | Execution time (ms) |
| status_code | integer | HTTP status |
Fluent Bit vs Logstash
| Parameter | Fluent Bit | Logstash |
|---|---|---|
| RAM consumption | ~1 MB | ~500 MB |
| Configuration | YAML/INI | Ruby DSL |
| Deployment | DaemonSet, sidecar | DaemonSet |
| Performance | 50K events/sec per 1 vCPU | 30K events/sec per 1 vCPU |
According to Fluent Bit official documentation, memory consumption does not exceed 1MB, making it 500x more memory-efficient than Logstash. Fluent Bit is also 1.6x more performant on a single vCPU. For most setups, Fluent Bit is the optimal choice. Example Fluent Bit configuration
[INPUT]
Name tail
Path /var/log/app/*.log
Parser json
Tag app.*
Refresh_Interval 5
[FILTER]
Name grep
Match app.*
Regex level (warn|error|fatal)
[OUTPUT]
Name es
Match app.*
Host elasticsearch
Port 9200
Index mobile-backend-logs
Type _doc
Logstash_Format On
Logstash_Prefix mobile-backend
The grep filter at the Fluent Bit level reduces data volume in Elasticsearch — debug logs do not enter storage.
Advantages of Fluent Bit
At typical mobile backend load (up to 10M events per day), infrastructure cost savings reach 30% compared to Logstash. Fluent Bit is easier to maintain — updates via Rolling Update in Kubernetes without downtime.
How to Protect Data in Logs from Leaks
Logs contain user_id, device information, sometimes request data fragments. Never log: passwords, full tokens, card numbers, personal data. PII in logs is a GDPR violation. We mask at the logger level with a simple filter: sensitive fields are replaced with ***. This ensures no confidential data enters Elasticsearch. Our proven track record includes 30+ projects with zero data breaches.
Elasticsearch and Index
For a mobile backend with moderate load (up to 10M events per day), a single Elasticsearch node or managed cluster (Elastic Cloud, AWS OpenSearch) suffices. Index lifecycle management (ILM) is mandatory: logs older than 30 days moved to cold tier or deleted, otherwise disk fills within a week.
{
"mappings": {
"properties": {
"timestamp": { "type": "date" },
"level": { "type": "keyword" },
"service": { "type": "keyword" },
"user_id": { "type": "keyword" },
"request_id": { "type": "keyword" },
"message": { "type": "text" },
"duration_ms": { "type": "long" },
"status_code": { "type": "integer" }
}
}
}
Dynamic mapping is a source of problems: Elasticsearch auto-detects the field type from the first value, and if status_code first arrives as a string, all subsequent numeric values cause mapping errors. Therefore we define the mapping template upfront.
Kibana: Search and Dashboards
After collection setup, create an Index Pattern in Kibana, configure Discover for field search, build Lens dashboards:
- Error rate by endpoint for the last hour
- Top-10 slow queries (duration_ms > 1000)
- Active users by user_id in real-time
- Error heatmap by hour and day
KQL (Kibana Query Language) for search is simpler than SQL: level: error AND service: auth-service AND duration_ms > 500 — instantly you see all slow authentication errors.
Logstash: Use Cases
Logstash is justified when complex parsing (e.g., multiline logs) or integration with legacy systems over TCP/UDP is required. However, for most mobile backends, Fluent Bit covers all scenarios and saves resources.
Scope of Work
- Audit current logging: identify problems (missing request_id, unstructured output)
- Configure JSON-formatted logs in code (with mandatory fields)
- Deploy EFK/ELK stack: Docker Compose for development, Kubernetes Helm charts for production
- Configure Index Lifecycle Management (ILM) — storage optimization
- Create Kibana dashboards for your key metrics
- Set up alerts (Kibana Watcher or Prometheus Alertmanager)
- Operator documentation and team training
- One month of post-deployment support
How We Do It: Step-by-Step Plan
- Analysis: review current architecture, log types, load level.
- Structuring: add JSON logger with required fields.
- Agent deployment: install Fluent Bit via DaemonSet or sidecar container.
- Elasticsearch setup: create mapping template, ILM policies.
- Kibana: configure Index Pattern and build dashboards.
- Testing: verify collection, search, and alert correctness.
- Deploy: roll out to staging and production.
Timelines
Basic EFK stack with Docker Compose, JSON-formatted logs for one service, basic Kibana dashboards: 2 to 3 days. Production-ready setup with ILM, alerts, multiple services, and security: 5 to 8 days. Cost is calculated individually — starting from $5,000, typically pays for itself within 2 months. Guaranteed 30% reduction in log-related infrastructure costs.
We'll assess your project in one business day — just write to us. In your message, specify your current stack and approximate load, and we'll propose the optimal turnkey solution. Get a consultation today and start seeing all backend errors in one window.