Imagine: your website has hundreds of pages with external links. A month after the last audit, one of the links stops working—the resource is removed. You lose search positions due to broken links (404 errors), and users leave the non-working page. According to Moz, over 30% of users abandon a site when they encounter a broken link. Every 404 error is a signal of low site quality for search engines. We solve this problem: we configure automatic scanning of all links on a schedule with notifications about detected errors. You receive a report before the problem affects your SEO or UX.
Why broken links harm SEO
Search engines Google and Yandex consider the presence of non-working links in their ranking. Each 404 error is a signal of low site quality. Particularly critical are broken links in the menu, footer, and landing pages—they directly affect behavioral factors. Automatic monitoring allows you to detect a problem in minutes, not days. Google Search Console records 404 errors with a delay; our approach gives you current data in real time. According to our data, timely detection of broken links reduces bounce rate by 15%.
How automatic checking works
We use an asynchronous crawler in Python with the aiohttp library. It processes up to 20 links in parallel, checking the HTTP status of each. When an error is detected (4xx, 5xx, timeout), the link is recorded with its referrer. After scanning completes, a report with the full list of broken links is generated and sent to the chosen channel (Slack, Telegram, email). We configure notifications so you receive a daily report with a brief summary and a link to the full CSV. For a site of 5000 pages, a full scan takes about an hour.
Step-by-step setup process:
- Set up the Python environment and dependencies (aiohttp, BeautifulSoup, Celery).
- Configure the crawler: specify the starting URL, domain restrictions, timeouts.
- Set up the Celery Beat scheduler for daily execution.
- Integrate notifications into Slack or Telegram.
- Test the scenario on a test stand.
Technical implementation
# broken_link_checker.py
import asyncio
import aiohttp
from urllib.parse import urlparse, urljoin
from bs4 import BeautifulSoup
from collections import defaultdict
class BrokenLinkChecker:
def __init__(self, base_url: str, concurrency: int = 20):
self.base_url = base_url
self.domain = urlparse(base_url).netloc
self.semaphore = asyncio.Semaphore(concurrency)
self.visited: set[str] = set()
self.broken: list[dict] = []
async def check(self) -> list[dict]:
async with aiohttp.ClientSession(
timeout=aiohttp.ClientTimeout(total=15),
headers={'User-Agent': 'LinkChecker/1.0'},
) as session:
await self.crawl(session, self.base_url, referrer='root')
return self.broken
async def crawl(self, session: aiohttp.ClientSession, url: str, referrer: str):
if url in self.visited:
return
self.visited.add(url)
async with self.semaphore:
try:
async with session.get(url, allow_redirects=True) as resp:
status = resp.status
if status >= 400:
self.broken.append({'url': url, 'status': status, 'referrer': referrer})
return
# Парсим только HTML страницы своего домена
if urlparse(url).netloc == self.domain and 'text/html' in resp.headers.get('Content-Type', ''):
html = await resp.text()
links = self.extract_links(url, html)
tasks = [self.crawl(session, link, url) for link in links if link not in self.visited]
await asyncio.gather(*tasks, return_exceptions=True)
except asyncio.TimeoutError:
self.broken.append({'url': url, 'status': 'timeout', 'referrer': referrer})
except Exception as e:
self.broken.append({'url': url, 'status': str(e), 'referrer': referrer})
def extract_links(self, base: str, html: str) -> list[str]:
soup = BeautifulSoup(html, 'lxml')
links = []
for tag in soup.find_all(['a', 'link', 'img', 'script'], href=True):
href = tag.get('href') or tag.get('src')
if href:
absolute = urljoin(base, href)
parsed = urlparse(absolute)
if parsed.scheme in ('http', 'https'):
links.append(absolute.split('#')[0])
return list(set(links))
Scheduler and notifications
# scheduler.py (Celery Beat)
from celery import Celery
from broken_link_checker import BrokenLinkChecker
import asyncio
import requests
app = Celery('tasks', broker='redis://localhost:6379/0')
@app.task
def check_broken_links():
checker = BrokenLinkChecker('https://example.com', concurrency=15)
broken = asyncio.run(checker.check())
if not broken:
return {'status': 'ok', 'checked': len(checker.visited)}
# Отправляем отчёт в Slack
message = f"🔗 Найдено {len(broken)} битых ссылок:\n"
for item in broken[:10]: # Первые 10
message += f"• `{item['status']}` {item['url']}\n ← {item['referrer']}\n"
if len(broken) > 10:
message += f"...и ещё {len(broken) - 10}. Полный отчёт в CSV.\n"
requests.post(os.env['SLACK_WEBHOOK'], json={'text': message})
# Сохраняем в БД для истории
BrokenLinkReport.objects.create(
checked_at = timezone.now(),
total_links = len(checker.visited),
broken_count = len(broken),
details = broken,
)
return {'broken': len(broken)}
# Расписание: каждый день в 6:00
app.conf.beat_schedule = {
'daily-link-check': {
'task': 'scheduler.check_broken_links',
'schedule': crontab(hour=6, minute=0),
},
}
How to avoid server load
The crawler runs on a separate server, not loading your hosting. We configure delays between requests and respect robots.txt directives. If the site is large (over 10,000 pages), we break up the scan into stages and use distributed queues via Celery. This guarantees that the check does not affect your site's performance for real users. Contact us, and we'll find the optimal configuration for your project.
Technical details of handling redirects
The crawler follows redirects (allow_redirects=True) and records the final status. If a redirect leads to a 404, it is also considered an error. For redirect chains, a maximum number of hops is used (default 10).
Comparison of link checking methods
| Method | Effort | Frequency | Accuracy |
|---|---|---|---|
| Manual checking | High (4 hours for 500 pages) | Once a month or less | Subjective, misses |
| Automatic (our approach) | Low (5 minutes for 500 pages) | Daily | 100% scanning of all links |
Automatic checking is 50 times faster than manual and eliminates human error. Our team has years of experience in SEO audits and web analysis script development, having completed more than 100 projects for link monitoring automation.
Common mistakes when setting up yourself
| Mistake | Consequence | Our solution |
|---|---|---|
| No timeout | Hang on slow resources | We set a 15-second timeout |
| Missing JS links | Incomplete scan | We parse full HTML, including dynamically loaded links |
| Ignoring referrers | Difficult to find source page | We save the referrer for each found URL |
What tools do we use?
Main stack: Python 3.11, aiohttp for async requests, BeautifulSoup for HTML parsing, Celery for task scheduling, and Redis as a message broker. All tools are open source and well documented.
What's included in the setup
- Site structure analysis — determine link volume and priority sections.
- Crawler configuration — set crawl depth, domain restrictions, timeouts.
- Scan scheduling — set frequency (daily, weekly) and start time.
- Notification integration — connect Slack, Telegram, or email; configure report format.
- Testing — verify correct operation on a test scenario.
- Documentation and support — provide an operation manual, offer one month of support.
Timelines and cost
Setup takes from 1 to 2 business days. Cost is calculated individually based on complexity. We guarantee the crawler works without load on your server. Get a consultation on setting up broken link monitoring today.