Broken links are not the only headache for a tech lead. Crooked canonicals, duplicate titles, N+1 redirect chains—each of these bugs drags down search rankings. Screaming Frog SEO Spider is the de facto standard for technical audits. We use it on every project and configure automated crawling for regular monitoring. With our 10+ years of production experience and over 40 projects completed, we confirm: no issue goes unnoticed. Automating manual audits saves 70–90% of the effort. The configuration effort is estimated after a brief analysis—contact us for a tailored setup.
What problems does Screaming Frog solve?
- Broken links and redirects: find 4xx and 5xx errors, redirect chains, cyclic redirects. On one project with a 20,000-product catalog we discovered a 12-hop redirect chain—ranking dropped by 30%.
- Meta-tag issues: missing or duplicate titles, overly long meta descriptions, wonky H1s. The crawler highlights all discrepancies.
- Indexing issues: pages without canonical, incorrect hreflang, URLs blocked in robots.txt that should be indexed. For example, blocking filter pages led to a 40% traffic loss.
Automated crawling via CLI is 5× faster than manual GUI launches and allows integrating the audit into CI/CD. For a 20,000-URL site with JS rendering, expect 4-8 GB RAM usage; downtime due to undetected errors is 3–5× higher. Contact us to estimate the savings on your project.
Why automated crawling is the standard for technical audits
GUI version is fine for one-off checks, but regular monitoring requires CLI. Comparison:
| Criterion | GUI | CLI |
|---|---|---|
| Launch | Manual each time | Scheduled automatic |
| Speed | Depends on UI | 5× faster |
| Integration | None | CI/CD, cron, notifications |
| Resources | Fixed heap | Configurable -Xmx |
| Repeatability | Varies each run | Identical config |
Common configuration mistakes:
| Mistake | Consequence | Solution |
|---|---|---|
| No User-Agent set | Crawler blocked | Set --user-agent appropriately |
| Excessive crawl depth | Long scan time | Limit --max-crawl-depth to 5-10 |
| JS rendering disabled | Missed links | Use --use-rendered |
| No exclusions | Looping | Configure --exclude for parameters |
Example cron configuration
0 3 * * 1 /usr/bin/screamingfrogseospider --crawl https://example.com --headless --config /etc/screamingfrog/standard-audit.seospiderconfig --output-folder /var/reports/$(date +\%Y\%m\%d) --export-tabs "Internal:All,Response Codes:All,Page Titles:All,Meta Description:All,H1:All,Canonical:All,Directives:All,Hreflang:All,Structured Data:All,Links:All,Sitemaps:All" --timestamped-output
Setting up automated crawling
Installation and license
Screaming Frog is a Java application. The free version is limited to 500 URLs; the paid license is for commercial projects.
# Ubuntu/Debian
wget https://download.screamingfrog.co.uk/products/seo-spider/screamingfrogseospider_21.0_all.deb
sudo dpkg -i screamingfrogseospider_21.0_all.deb
CLI launch
Key mode: --headless. Parameters: --crawl, --save-crawl, --export-tabs, --output-folder.
screamingfrogseospider \
--crawl https://example.com \
--headless \
--save-crawl \
--export-tabs "Internal:All,Response Codes:All,Page Titles:All,Meta Description:All,H1:All,Canonical:All,Directives:All,Hreflang:All,Structured Data:All,Links:All,Sitemaps:All" \
--output-folder /var/reports/example-com/$(date +%Y%m%d) \
--timestamped-output
Configuration and rendering
To standardize, save a config in GUI and pass via --config. For JavaScript-based sites, enable rendering with the built-in Chromium (--use-rendered). Crawling time with JS increases 3–5×, so only use where critical.
Authentication
# Cookie
screamingfrogseospider --crawl https://example.com/admin/ --headless --set-cookies "PHPSESSID=abc123; laravel_session=xyz789" --output-folder /var/reports/
# HTTP Basic
screamingfrogseospider --crawl https://example.com --headless --auth-details "user:password" --output-folder /var/reports/
How to schedule CLI runs?
Use a cron job with the full binary path and parameters above. Ensure Java heap is sufficient: for a 50,000-URL site with JS rendering, 4-8 GB RAM (-Xmx). Add notifications (Telegram/email) for 4xx and 5xx errors.
When is JavaScript rendering needed?
For sites on React, Vue, or Angular where content is dynamically loaded. Without rendering, the crawler sees only an empty template—missing up to 80% of links. We recommend enabling --use-rendered for SPAs.
Why JS rendering is critical for SPAs
SPAs lack static HTML pages. Without rendering, the crawler cannot find internal links, meta-tags, or structured data. As per the official Screaming Frog documentation, a headless Chrome is essential for a complete SPA audit. This is the only way to uncover all technical issues.
Our work process
- Analysis: define audit goals, URL list, robots.txt, sitemap.
- Design: create crawler configuration (User-Agent, depth, exclusions).
- Implementation: set up CLI launch, data export, processing scripts.
- Testing: verify on a test subdomain, validate results.
- Deployment: deploy to server, configure cron, notifications via Telegram/email.
What’s included in the result
- Crawler configuration file (
standard-audit.seospiderconfig). - Bash script for automated launch with logging.
- Python script for analyzing key metrics (response codes, duplicate titles/H1, broken links).
- Notification setup (Telegram/email) for anomalies.
- Maintenance and update instructions.
Processing the results
Screaming Frog exports CSV for each tab. We provide Python scripts to load and summarize.
import pandas as pd
from pathlib import Path
def load_crawl_results(output_dir: str) -> dict[str, pd.DataFrame]:
results = {}
for csv_file in Path(output_dir).glob('*.csv'):
name = csv_file.stem.lower().replace(' ', '_')
try:
df = pd.read_csv(csv_file, encoding='utf-8-sig', low_memory=False)
results[name] = df
except Exception as e:
print(f'Could not load {csv_file}: {e}')
return results
def audit_summary(results: dict) -> dict:
summary = {}
if 'internal_all' in results:
internal = results['internal_all']
summary['total_urls'] = len(internal)
summary['indexable'] = len(internal[internal.get('Indexability', '') == 'Indexable'])
if 'response_codes_all' in results:
codes = results['response_codes_all']
for code, prefix in [('2xx','2'),('3xx','3'),('4xx','4'),('5xx','5')]:
summary[code] = len(codes[codes['Status Code'].astype(str).str.startswith(prefix)])
if 'page_titles_all' in results:
titles = results['page_titles_all']
summary['missing_title'] = titles['Title 1'].isna().sum() + (titles['Title 1']=='').sum()
summary['duplicate_title'] = titles['Title 1'].duplicated().sum()
if 'h1_all' in results:
h1 = results['h1_all']
summary['missing_h1'] = h1['H1-1'].isna().sum() + (h1['H1-1']=='').sum()
if 'H1-1 Count' in h1:
summary['multiple_h1'] = (h1['H1-1 Count'] > 1).sum()
return summary
The summary immediately highlights critical issues and scales the effort. For example, more than 5% 4xx errors signals an urgent structural redesign. Automating the crawl cuts the cost of technical audits by 70–90% compared to manual testing. Get a consultation and a sample configuration for your site — contact us for a preliminary assessment.