How a Bot Helps You Monitor Competitors' New Products: Setup and Results
Manual browsing of competitor catalogs takes hours daily. Instead of focusing on strategy, you open dozens of pages looking for new items. One competitor with 20 categories and 50 catalog pages takes about 30–40 minutes per day. With 5–10 competitors, this becomes a full work shift, and human error inevitably leads to missing 20–30% of new products.
We automate this process. Our bot performs automated competitor monitoring, new product tracking, and SKU catalog scraping. It scans product pages daily on a schedule and instantly notifies you of new items via Telegram. You get only real changes, no noise. This lets you react faster than competitors and not lose audience.
With over 5 years of experience, we have developed solutions for 40+ online stores, from small niche projects to large marketplaces. Our expertise guarantees stable operation even when competitor layout changes—we use semantic selectors and fallback strategies. Time savings on monitoring: up to 80%. The bot processes up to 10,000 SKU per day, generating real-time reports.
Why Manual Monitoring Is Inefficient?
Competitors have hundreds of catalog pages that update daily. Manually reviewing them wastes time and misses new items. The bot checks each page on a schedule, and you learn about new products within minutes.
Moreover, the human eye cannot track dozens of categories simultaneously. Misses are inevitable, especially if a product appears in the middle of a list. The bot captures every change by comparing the current state with a baseline snapshot.
How the Bot Detects New Products?
Monitoring new products differs from price monitoring. We track not a specific URL but catalog sections—categories, "New" pages, search results. The algorithm works as follows:
Configuration (catalog URL + selector) → Scraper → Snapshot → Diff → Alert
- Scraper loads the page and extracts the product list via CSS selectors.
- Each product is saved in the database with a field hash.
- On subsequent scans, the new list is compared with the previous one.
- Positions missing in the old snapshot are marked as new.
- The system sends a notification to Telegram.
Detailed detector operation algorithm
For each competitor, a JSON config with selectors and pagination rules is set up:
{
"pagination": {
"type": "url_param",
"param": "page",
"max_pages": 20
},
"item_selector": ".catalog-item",
"fields": {
"url": {"selector": "a.product-link", "attr": "href"},
"title": {"selector": ".product-name", "text": true},
"sku": {"selector": "[data-sku]", "attr": "data-sku"},
"price": {"selector": ".price", "text": true},
"image": {"selector": "img.product-image", "attr": "src"}
}
}
Data schema for storing snapshots and new items:
CREATE TABLE competitor_catalogs (
id BIGSERIAL PRIMARY KEY,
competitor_id INT REFERENCES competitors(id),
url TEXT NOT NULL,
scrape_config JSONB NOT NULL,
check_interval INTERVAL DEFAULT '6 hours',
last_checked_at TIMESTAMP,
is_active BOOLEAN DEFAULT TRUE
);
CREATE TABLE competitor_items (
id BIGSERIAL PRIMARY KEY,
catalog_id BIGINT REFERENCES competitor_catalogs(id),
external_url TEXT NOT NULL,
title TEXT,
external_sku VARCHAR(255),
price NUMERIC(12,2),
image_url TEXT,
first_seen_at TIMESTAMP DEFAULT NOW(),
last_seen_at TIMESTAMP DEFAULT NOW(),
is_new BOOLEAN DEFAULT TRUE,
UNIQUE(catalog_id, external_url)
);
CREATE INDEX idx_competitor_items_new
ON competitor_items(catalog_id, first_seen_at)
WHERE is_new = TRUE;
The scraper itself is implemented in PHP using Symfony DomCrawler:
class CatalogScraper
{
public function scrape(CompetitorCatalog $catalog): array
{
$config = $catalog->scrape_config;
$items = [];
$maxPages = $config['pagination']['max_pages'] ?? 1;
for ($page = 1; $page <= $maxPages; $page++) {
$url = $this->buildPageUrl($catalog->url, $config['pagination'], $page);
$html = $this->fetchWithRetry($url);
if (!$html) break;
$pageItems = $this->extractItems($html, $config);
if (empty($pageItems)) break;
$items = array_merge($items, $pageItems);
usleep(rand(1_500_000, 3_000_000));
}
return $items;
}
private function extractItems(string $html, array $config): array
{
$crawler = new \Symfony\Component\DomCrawler\Crawler($html);
$items = [];
$crawler->filter($config['item_selector'])->each(function ($node) use ($config, &$items) {
$item = [];
foreach ($config['fields'] as $field => $fieldConfig) {
try {
$el = $node->filter($fieldConfig['selector'])->first();
if ($el->count() === 0) continue;
$item[$field] = isset($fieldConfig['attr'])
? $el->attr($fieldConfig['attr'])
: $el->text();
} catch (\Exception $e) {
continue;
}
}
if (!empty($item['url'])) {
$items[] = $item;
}
});
return $items;
}
}
New product detection service:
class NewProductDetectionService
{
public function process(CompetitorCatalog $catalog): DetectionResult
{
$scraped = $this->scraper->scrape($catalog);
$newItems = [];
foreach ($scraped as $item) {
$url = $this->normalizeUrl($item['url'], $catalog->url);
$existing = CompetitorItem::where([
'catalog_id' => $catalog->id,
'external_url' => $url,
])->first();
if (!$existing) {
$created = CompetitorItem::create([
'catalog_id' => $catalog->id,
'external_url' => $url,
'title' => $item['title'] ?? null,
'external_sku' => $item['sku'] ?? null,
'price' => $this->parsePrice($item['price'] ?? ''),
'image_url' => $item['image'] ?? null,
'is_new' => true,
]);
$newItems[] = $created;
} else {
$existing->update(['last_seen_at' => now()]);
}
}
$disappeared = CompetitorItem::where('catalog_id', $catalog->id)
->where('last_seen_at', '<', now()->subDays(3))
->get();
$catalog->update(['last_checked_at' => now()]);
return new DetectionResult(newItems: $newItems, disappeared: $disappeared);
}
}
After detecting new items, the bot sends a Telegram message with brief information: name, price, link. Notifications arrive within a minute after scanning.
Bypassing Bot Protections
| Protection | Bypass Method |
|---|---|
| Cloudflare Bot Management | Playwright + stealth plugin |
| Rate limiting | Random delays 2–5 sec between requests |
| IP blocks | Proxy rotation (residential proxies) |
| Cookie/session requirement | Headless browser with session persistence |
| JS rendering | Playwright/Puppeteer instead of curl |
What's Included in the Work
- Configuration for up to 5 competitors (categories, selectors, intervals)
- Telegram notification setup (new items, disappearances)
- Deployment on your server (Docker image)
- Operations documentation
- Team training (1 hour)
- Support and modifications for 1 month
- Pricing starts from $500
Comparison of Approaches
| Parameter | Manual Monitoring | Bot |
|---|---|---|
| Time to check 1 competitor | 30 min/day | 1 sec/day |
| Missed new items | up to 30% | <1% |
| Reaction to changes | hours–days | minutes |
| Scaling cost | linear | fixed |
| Cost | Variable hours | Fixed from $500 |
Typical Monitoring Mistakes and How to Avoid Them
- Ignoring pagination. If the competitor has more than one page, the bot will go through all. Check the
max_pagessetting. - Too frequent requests. Risk of blocking. We recommend intervals of 6–12 hours.
- Rigid selectors. When layout changes, selectors may break. Use data attributes or identifier classes.
- Lack of error handling. If a page is temporarily unavailable, the bot should retry with a delay.
Timeline for Implementation
- CatalogScraper + JSON configuration: 1–2 days
- NewProductDetectionService + data schema: 1 day
- Telegram notifications: 0.5 day
- Playwright adapter for JS sites: +1 day
- Proxy rotation: +0.5 day
Total: 3–4 working days.
Get a free consultation on monitoring automation. Our engineers will audit your competitors and offer the optimal solution.
Contact us to discuss your task. We'll evaluate the project and select a configuration that fits your budget.