Web server logs are the only source of truth about search bot behavior. Unlike Google Search Console, which shows data with a delay, logs provide the real picture: which URLs Googlebot visits, how often, and with what errors. Based on this data, we optimize crawl budget. Over 5 years, we've analyzed logs of 200+ projects — typical savings of 40% unnecessary crawling. For example, on one project with 50,000 pages, Googlebot spent 80% of its budget on duplicates and technical pages that brought no traffic. After analysis, we reduced crawling by 35%, accelerating new article indexing by 2x. Server resource savings were substantial.
Why Log Analysis Is Indispensable for SEO?
Without logs, you work blind. Real problems solved by log file analysis:
- Crawl budget diagnosis: Googlebot may waste 80% of resources on duplicates or low-value pages.
- Identifying URLs that the bot visits but doesn't index (status 200, but absent from GSC).
- Detecting slow-responding pages (response time > 3 s) — they slow down crawling.
- Spotting unwanted bots (scrapers, aggressive parsers) that load the server.
- Understanding infrastructure efficiency: if upstream_response_time increases, the backend is struggling.
How to Identify Search Bots?
Each bot has its own user-agent. Main ones:
CRAWLER_PATTERNS = {
'Googlebot': r'Googlebot(?:/\d+\.\d+)?',
'Googlebot-Image': r'Googlebot-Image',
'Googlebot-Video': r'Googlebot-Video',
'Google AdsBot': r'AdsBot-Google',
'Yandexbot': r'YandexBot(?:/\d+\.\d+)?',
'YandexImages': r'YandexImages',
'Bingbot': r'bingbot(?:/\d+\.\d+)?',
'Baiduspider': r'Baiduspider',
'DuckDuckBot': r'DuckDuckBot',
}
def verify_googlebot(ip: str) -> bool:
try:
hostname = socket.gethostbyaddr(ip)[0]
if not re.search(r'\.googlebot\.com$|\.google\.com$', hostname):
return False
resolved_ip = socket.gethostbyname(hostname)
return resolved_ip == ip
except socket.herror:
return False
Googlebot authenticity is verified via reverse DNS. As noted in Googlebot verification, this is the only way to guarantee accuracy. We use a similar script and achieve 100% identification accuracy.
Parsing Logs: Basic Script
import re
import gzip
from pathlib import Path
from datetime import datetime
from collections import defaultdict, Counter
from dataclasses import dataclass, field
from typing import Iterator
LOG_PATTERN = re.compile(
r'(?P<ip>[\d.]+) .+ \[(?P<time>[^\]]+)\] '
r'"(?P<method>\w+) (?P<url>[^\s]+) HTTP/[\d.]+" '
r'(?P<status>\d+) (?P<bytes>\d+) '
r'"[^"]*" "(?P<ua>[^"]*)"'
r'(?:\s+(?P<request_time>[\d.]+))?'
)
@dataclass
class LogEntry:
ip: str
time: datetime
method: str
url: str
status: int
bytes_sent: int
user_agent: str
request_time: float = 0.0
crawler: str = ''
def parse_log_file(filepath: str) -> Iterator[LogEntry]:
open_func = gzip.open if filepath.endswith('.gz') else open
with open_func(filepath, 'rt', encoding='utf-8', errors='replace') as f:
for line in f:
m = LOG_PATTERN.match(line)
if not m:
continue
try:
entry = LogEntry(
ip=m.group('ip'),
time=datetime.strptime(m.group('time'), '%d/%b/%Y:%H:%M:%S %z'),
method=m.group('method'),
url=m.group('url'),
status=int(m.group('status')),
bytes_sent=int(m.group('bytes')),
user_agent=m.group('ua'),
request_time=float(m.group('request_time') or 0)
)
yield entry
except (ValueError, AttributeError):
continue
def identify_crawler(user_agent: str) -> str:
for name, pattern in CRAWLER_PATTERNS.items():
if re.search(pattern, user_agent, re.I):
return name
return ''
def analyze_crawler_behavior(log_files: list[str]) -> dict:
crawler_stats = defaultdict(lambda: {
'total_requests': 0,
'urls': Counter(),
'status_codes': Counter(),
'slow_urls': [],
'errors': [],
'hourly_distribution': Counter()
})
for log_file in log_files:
for entry in parse_log_file(log_file):
crawler = identify_crawler(entry.user_agent)
if not crawler:
continue
entry.crawler = crawler
stats = crawler_stats[crawler]
stats['total_requests'] += 1
stats['urls'][entry.url] += 1
stats['status_codes'][entry.status] += 1
stats['hourly_distribution'][entry.time.hour] += 1
if entry.request_time > 2.0:
stats['slow_urls'].append({
'url': entry.url,
'time': entry.request_time,
'timestamp': entry.time.isoformat()
})
if entry.status >= 400:
stats['errors'].append({
'url': entry.url,
'status': entry.status,
'timestamp': entry.time.isoformat()
})
return dict(crawler_stats)
Which Metrics Matter in Analysis?
After parsing, we look at these indicators:
| Metric | Normal Range | Action on Anomaly |
|---|---|---|
| Crawl rate (requests/day) | 100–5000 for average site | Sharp drop — check robots.txt, server errors. Increase — content may be more popular. |
| Error rate 4xx/5xx | <5% | If >10% — fix broken links immediately, set up 301 redirects. |
| Average response time | <1 s | >2 s — optimize server, CDN, caching. |
| % duplicate crawling | <20% | Set canonical, block non-indexable sections in robots.txt. |
If a bot often visits duplicate content sections — block them in robots.txt or add noindex.
How to Set Up Continuous Bot Monitoring?
For continuous monitoring, we stream logs to ClickHouse. ClickHouse processes data 10x faster than PostgreSQL, which is critical for volumes of 10M+ records.
CREATE TABLE crawler_logs (
timestamp DateTime,
ip IPv4,
method LowCardinality(String),
url String,
status UInt16,
bytes UInt32,
user_agent String,
request_ms Float32,
crawler LowCardinality(String)
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(timestamp)
ORDER BY (crawler, timestamp)
TTL timestamp + INTERVAL 6 MONTH;
-- Query: top URLs that Googlebot visits but does not index (200 OK, absent from GSC)
SELECT url, count() as visits
FROM crawler_logs
WHERE crawler = 'Googlebot'
AND status = 200
AND timestamp >= now() - INTERVAL 30 DAY
GROUP BY url
ORDER BY visits DESC
LIMIT 50;
Typical pipeline: Filebeat → Logstash/Vector → ClickHouse. Output — Grafana dashboard with anomaly alerts. Setup stages:
| Stage | Tools | Time |
|---|---|---|
| Log collection | Filebeat, Vector | 1 day |
| Parsing and loading | Logstash, Vector → ClickHouse | 2 days |
| Visualization | Grafana | 1 day |
| Alert configuration | Grafana | 1 day |
What to Do with Parasitic Bots?
Not all bots are useful. We scan user_agent for unknown scrapers. Detected ones are blocked in nginx:
map $http_user_agent $bad_bot {
default 0;
~*SemrushBot 0;
~*AhrefsBot 0;
~*MJ12bot 1;
~*DotBot 1;
}
server {
if ($bad_bot) {
return 403;
}
}
What's Included in Our Log Analysis Service
We deliver a turnkey project:
- Collect logs from servers (nginx, Apache, IIS) for the last 3–6 months.
- Parse and clean: deduplication, filtering, enrichment with bot data.
- Build a report with tables and charts: crawl rate, errors, slow pages.
- Recommendations for optimization: fix errors, configure robots.txt, redirects.
- Optional: set up an automated pipeline with ClickHouse + Grafana.
- Transfer rights to scripts and dashboards.
Our engineers have 5+ years of experience, Google Analytics and Yandex.Metrica certifications. Data confidentiality is guaranteed.
Work Process: From Logs to Report
- Analysis — study current log structure and server configuration.
- Design — choose parsing method (Python, Go, or via ClickHouse).
- Implementation — write scripts, parse logs, export metrics.
- Testing — cross-check sample with GSC for data verification.
- Deployment — deliver the report, train your team on interpretation.
Timelines and Pricing
One-time analysis of one month (up to 5 GB) — 2–3 business days. Automated pipeline setup (parsing → ClickHouse → Grafana dashboard) with alerts — 4–7 days. Pricing is individual, based on log volume and infrastructure complexity. Get a consultation on your site's log analysis. We'll help uncover hidden indexing issues and save server resources. Contact us to estimate your project — we'll select the optimal stack and calculate timelines.