Build a Competitor Promotion Scraper Bot

Imagine your main competitor launches a flash sale with a 40% discount on a popular product, and you find out two days later — after customers have already left. Regular manual monitoring of competitor sites eats hours of a marketer's time, and it's easy to miss sudden promotions. A scraper finds promotions 10x faster than manual monitoring — information about a new deal arrives in minutes, not hours. We develop scraper bots that continuously scan your competitors' promotion pages, extract terms, promo codes, and deadlines, and instantly notify you via Slack or Telegram. Our team has extensive experience in building a competitor promotion scraper bot, with over 50 projects automating competitive analysis. Guaranteed results with our certified developers. One case: An electronics e‑commerce store reduced reaction time to competitor promotions from 2 days to 2 hours after implementing our solution, saving more than 15 person‑hours weekly. Payback period was under two months. Typical monthly savings: $2,000–$5,000 in analyst time. The scraper is built on PHP 8.3 and Laravel 11 using DomCrawler, with data stored in PostgreSQL. This approach enables rapid response to promotions, automatic price monitoring, and long-term analysis of competitor trends — which products are frequently discounted, average discount size, seasonality. Development starts at $3,500.

What Problems Does the Promotion Scraper Solve?

Manual monitoring cannot keep up with the dynamics of promotions: discounts change hourly, promo codes have limited validity, and page structures vary across sites. Typical challenges: promotions with different discount types (percentage, fixed price, promo code) require different extraction methods; start and end dates are often hidden in text or given in non‑standard formats; promo codes may be encoded or hidden in HTML. The scraper solves all these problems: it looks for structured blocks (.promotion-card, .sale-block) and, if absent, analyzes the text using regular expressions. For instance, if a date says "until end of week," the scraper computes the exact date from the calendar; if a promo code appears only after a click, it emulates the click via a headless browser.

Why Is Promotion Monitoring More Important Than Price Monitoring?

Prices change less frequently and predictably, while promotions are a powerful traffic driver. Missing a competitor's promotion can cost you up to 30% conversion. The scraper gives you information within minutes, allowing an instant counter‑move. Our experience shows that over 80% of clients start a counter‑campaign within an hour of receiving a notification. Our solution provides comprehensive discount monitoring across multiple competitors. This promotion tracking bot is designed for ecommerce. It specializes in promo code scraping and extraction. The system includes a dashboard for competitor promotion analysis. It is an ecommerce scraper adapted to various store platforms. We offer expert PHP scraper development services. The scraper can handle discount site scraping for multiple domains. You can configure discount notifications for any threshold.

How Does the Promotion Scraper Work?

We use PHP 8.3, Laravel 11, Symfony DomCrawler for scraping, and PostgreSQL for storage. Key components are described below.

Promotion Page Scraper

Extracts promotions from structured blocks. If blocks are not found, falls back to text analysis.

// app/Services/PromotionScraper/PromotionPageScraper.php
class PromotionPageScraper
{
    public function scrapePromotionsPage(string $url): array
    {
        $html = $this->fetch($url);
        $crawler = new Crawler($html);

        $promotions = [];

        // Standard promotion blocks
        $crawler->filter('.promotion-card, .sale-block, .promo-item, [data-promo]')
            ->each(function (Crawler $node) use (&$promotions) {
                $promo = $this->extractPromotion($node);
                if ($promo) $promotions[] = $promo;
            });

        // If no structured blocks found, parse text
        if (empty($promotions)) {
            $promotions = $this->extractFromText($crawler->text(), $url);
        }

        return $promotions;
    }

    private function extractPromotion(Crawler $node): ?array
    {
        $title = $node->filter('h2, h3, .promo-title, .sale-title')->first()->text('');
        if (empty(trim($title))) return null;

        $description = $node->filter('p, .promo-desc')->first()->text('');
        $link = $node->filter('a')->first()->attr('href') ?? '';

        // Extract dates from text
        $dates = $this->extractDates($title . ' ' . $description);

        // Extract discount percentage
        $discount = $this->extractDiscount($title . ' ' . $description);

        // Look for promo code in text
        $promoCode = $this->extractPromoCode($title . ' ' . $description);

        return [
            'title'       => trim($title),
            'description' => trim($description),
            'discount_pct'=> $discount,
            'promo_code'  => $promoCode,
            'starts_at'   => $dates['start'] ?? null,
            'ends_at'     => $dates['end'] ?? null,
            'url'         => $link,
        ];
    }

    private function extractDiscount(string $text): ?int
    {
        // "скидка 30%", "−30%", "30% OFF", "до 50% скидки"
        if (preg_match('/[-–]?\s*(\d{1,3})\s*%/u', $text, $m)) {
            return (int) $m[1];
        }
        return null;
    }

    private function extractPromoCode(string $text): ?string
    {
        // Promo code usually uppercase, 4-12 characters, sometimes in quotes or after 'promo code'
        if (preg_match('/промокод[:\s]+([A-Z0-9_-]{3,15})/ui', $text, $m)) {
            return strtoupper($m[1]);
        }
        if (preg_match('/promo(?:code)?[:\s]+([A-Z0-9_-]{3,15})/i', $text, $m)) {
            return strtoupper($m[1]);
        }
        // Words in quotes resembling a promo code
        if (preg_match('/[«"]([A-Z0-9_-]{4,12})["»]/u', $text, $m)) {
            return strtoupper($m[1]);
        }
        return null;
    }

    private function extractDates(string $text): array
    {
        $dates = [];

        // "с 01.03 по 31.03", "до 31 марта", "01.03.2025 - 15.03.2025"
        $monthMap = [
            'january'=>'01','february'=>'02','march'=>'03','april'=>'04',
            'may'=>'05','june'=>'06','july'=>'07','august'=>'08',
            'september'=>'09','october'=>'10','november'=>'11','december'=>'12',
        ];

        $pattern = '/(\d{1,2})\s+(' . implode('|', array_keys($monthMap)) . ')/ui';
        if (preg_match_all($pattern, $text, $matches, PREG_SET_ORDER)) {
            foreach ($matches as $i => $match) {
                $day = sprintf('%02d', $match[1]);
                $month = $monthMap[mb_strtolower($match[2])];
                $year = date('Y');
                $date = "{$year}-{$month}-{$day}";

                if ($i === 0) $dates['start'] = $date;
                if ($i === 1) $dates['end'] = $date;
            }
        }

        return $dates;
    }
}

Product Sale Detector

Compares old and new prices, calculates discount percentage, and finds end date from countdown.

// app/Services/PromotionScraper/ProductSaleDetector.php
class ProductSaleDetector
{
    public function detectSale(string $html): ?SaleInfo
    {
        $crawler = new Crawler($html);

        // Look for old and new price simultaneously
        $originalPriceNode = $crawler->filter(
            '.original-price, .old-price, del, [data-original-price], s'
        )->first();

        $salePriceNode = $crawler->filter(
            '.sale-price, .special-price, .discount-price, [data-sale-price]'
        )->first();

        if (!$originalPriceNode->count() || !$salePriceNode->count()) {
            return null;
        }

        $originalPrice = $this->parsePrice($originalPriceNode->text());
        $salePrice = $this->parsePrice($salePriceNode->text());

        if ($originalPrice <= 0 || $salePrice <= 0 || $salePrice >= $originalPrice) {
            return null;
        }

        $discountPct = round((1 - $salePrice / $originalPrice) * 100);

        // Promotion expiry date
        $endDate = null;
        $countdownNode = $crawler->filter('.countdown, [data-countdown], .sale-ends');
        if ($countdownNode->count()) {
            $endDate = $countdownNode->first()->attr('data-end-date')
                ?? $this->extractDateFromText($countdownNode->first()->text());
        }

        return new SaleInfo(
            originalPrice: $originalPrice,
            salePrice: $salePrice,
            discountPct: $discountPct,
            endsAt: $endDate,
        );
    }
}

Storage and History

All promotions are stored in the competitor_promotions table with change history.

// Migration
Schema::create('competitor_promotions', function (Blueprint $table) {
    $table->id();
    $table->foreignId('competitor_id')->constrained();
    $table->string('title');
    $table->text('description')->nullable();
    $table->integer('discount_pct')->nullable();
    $table->string('promo_code')->nullable();
    $table->string('source_url');
    $table->date('starts_at')->nullable();
    $table->date('ends_at')->nullable();
    $table->boolean('is_active')->default(true);
    $table->json('affected_categories')->nullable();
    $table->timestamp('first_seen_at');
    $table->timestamp('last_seen_at');
    $table->timestamps();

    $table->index(['competitor_id', 'is_active', 'ends_at']);
});

// app/Jobs/ScrapeCompetitorPromotions.php
class ScrapeCompetitorPromotions implements ShouldQueue
{
    public function handle(PromotionPageScraper $scraper): void
    {
        $competitor = Competitor::findOrFail($this->competitorId);
        $promotionUrls = $competitor->promotion_urls ?? [];

        $currentPromos = [];

        foreach ($promotionUrls as $url) {
            $scraped = $scraper->scrapePromotionsPage($url);
            $currentPromos = array_merge($currentPromos, $scraped);
            sleep(rand(2, 4));
        }

        // Deactivate promotions no longer present
        CompetitorPromotion::where('competitor_id', $this->competitorId)
            ->where('is_active', true)
            ->whereNotIn('source_url', array_column($currentPromos, 'url'))
            ->update(['is_active' => false]);

        // Update or create promotions
        foreach ($currentPromos as $promo) {
            CompetitorPromotion::updateOrCreate(
                [
                    'competitor_id' => $this->competitorId,
                    'source_url'    => $promo['url'],
                ],
                [
                    'title'        => $promo['title'],
                    'description'  => $promo['description'],
                    'discount_pct' => $promo['discount_pct'],
                    'promo_code'   => $promo['promo_code'],
                    'starts_at'    => $promo['starts_at'],
                    'ends_at'      => $promo['ends_at'],
                    'is_active'    => true,
                    'last_seen_at' => now(),
                    'first_seen_at' => now(),
                ]
            );
        }

        // Notification for new big sales
        $newBigSales = CompetitorPromotion::where('competitor_id', $this->competitorId)
            ->where('first_seen_at', '>=', now()->subMinutes(10))
            ->where('discount_pct', '>=', 20)
            ->get();

        if ($newBigSales->isNotEmpty()) {
            Notification::route('slack', config('monitoring.slack_url'))
                ->notify(new BigCompetitorSaleNotification($competitor, $newBigSales));
        }
    }
}

Schedule

// Promotions — more frequent, especially on Friday/weekends
$schedule->command('scrape:promotions')
    ->everyTwoHours()->withoutOverlapping();

// Before weekends — increased frequency
$schedule->command('scrape:promotions')
    ->fridays()->at('09:00');

How to Implement the Scraper: Step-by-Step

1. Analyze competitor sites — identify promotion page structure.
2. Design the scraper architecture and data schema.
3. Develop the scraper and sale detector.
4. Test on 3–5 sites, fix bugs.
5. Deploy to a server, configure cron and notifications.

Work Process

Typical Scraping Mistakes

• Promotion hidden behind JavaScript rendering — need a headless browser.
• Promo code only appears after a click — requires action emulation.
• Dates given in relative format ("until end of week") — require a custom parser.
• Site structure changes break the scraper — use fallback algorithms.

What's Included in the Work

• Scraper for 3–5 competitor sites (turnkey)
• Dashboard for viewing promotion history
• Notifications via Slack (Telegram on request)
• Documentation and API for integration
• Team training (1 hour)
• One month of support after deployment
• Development cost: from $3,500

Timeline Estimates

Basic functionality development — 5 to 8 working days. Complex projects with non‑standard site structures — up to 14 days. We provide an accurate estimate after auditing the target resources. Contact us for a free project assessment. Get a consultation and order scraper development to gain a competitive advantage.

E-commerce Store Development

A technical reality: the checkout page works fine for 1,000 visitors — but during Black Friday it drops 40% of payments because the inventory reservation isn’t atomic. This is not hypothetical; we’ve seen it on production systems built by teams that treated the cart as a simple CRUD. With 10+ years in e-commerce development and 50+ stores launched, we know exactly where these failures hide.

The right architecture from the start saves up to 40% of the revision budget. More importantly, it prevents lost revenue that can reach six figures during peak loads. Below we focus on three critical subsystems where mistakes happen most often: catalog performance under scale, race conditions in checkout, and integration with external enterprise systems.

Why Does Catalog Performance Degrade as SKUs Grow?

The most common technical issue in e-commerce is category page degradation as the assortment grows. A page works well with 500 products and starts to lag at 10,000. The causes are almost always the same.

N+1 on attributes. You load a list of products — 50 items. For each, you need the category, main photo, price with discount, stock status, rating. Without proper eager loading, that’s 250+ queries per page. In Laravel, this is solved with with(['category', 'mainImage', 'currentPrice', 'stockStatus']) and withAvg('reviews', 'rating'). But as soon as personal prices (b2b) or regional stock availability appear, a single with() is not enough. You need Query Objects or a dedicated ReadModel.

Faceted filtering without indexes. Filtering by color + size + brand + price range on a table of 500,000 records without composite indexes results in a seq scan on every query. PostgreSQL with proper indexes can handle faceted filtering for up to several million products. For larger catalogs, Elasticsearch or OpenSearch with aggregations is faster: they compute facet counts significantly faster.

Pagination via OFFSET. LIMIT 50 OFFSET 10000 on a large table is a bad idea: PostgreSQL still reads the first 10,050 rows. Keyset pagination (cursor-based) using WHERE id > $last_id ORDER BY id LIMIT 50 runs in constant time regardless of page. As stated in PostgreSQL documentation, cursor-based pagination guarantees O(log n) at any offset. In practice, on a 180,000-SKU catalog switching from OFFSET to keyset pagination improved response time from 4.2 s to 280 ms — about 15x faster at page 200. Server resource savings were significant.

Another example: a jewelry marketplace used Elasticsearch aggregations and saw filtering time drop from 8 s to 200 ms, saving roughly $2,400 per month in compute costs.

What Is a Race Condition in the Cart and How to Avoid It?

Checkout is where money either lands in your account or not. Technical issues here are costly.

Race condition in product reservation. Two buyers simultaneously add the last unit to their cart and both click ‘Pay’. Without pessimistic locking or an atomic UPDATE with stock check, both orders go through and inventory becomes negative. In PostgreSQL:

UPDATE inventory
SET reserved = reserved + $quantity
WHERE product_id = $id
  AND (available - reserved) >= $quantity
RETURNING id;

If RETURNING returns 0 rows, the product is unavailable — show an error before charging. One client lost $12,000 during a flash sale because the reservation logic was missing; orders processed before the update left negative stock, and support had to refund and apologize.

Idempotency of payment webhooks. payment.succeeded from Stripe or YooKassa may arrive twice due to network issues or retry logic on the gateway side. Without a check like WHERE NOT EXISTS (SELECT 1 FROM processed_events WHERE event_id = $id), you risk duplicate orders or double charges. Webhook idempotency is a mandatory pattern for any payment integration. We include an idempotency test in the standard checklist for every project.

Multi-step checkout vs single-page. Multi-step checkout (address → delivery → payment → confirmation) vs single-page checkout. Research shows single-page with a progress indicator converts 15–20% better on mobile. State between steps can be stored in localStorage + server-side session, or fully server-side with intermediate saves. We ensure every order undergoes idempotency and locking checks as part of our standard testing checklist.

How to Integrate with 1С, Warehouse, and Delivery?

1С is a separate chapter. Three common integration methods:

• CommerceML over HTTP — 1С exports XML on a schedule, the site imports. Works for small catalogs up to 5,000 SKUs, but has synchronization delay. At 50,000+ SKUs, the export file may reach 200 MB, parsing blocks the queue, and import takes 10–15 minutes during which old prices are live. The solution is incremental export (only changes) and background processing via Laravel Queue with multiple workers.
• REST API / OData from 1С — real-time two-way synchronization. Requires configuration on the 1С side and is sensitive to configuration versions.
• Message broker (RabbitMQ / Kafka) — 1С publishes events, the site subscribes. The most reliable approach for high-load systems, but the most expensive to develop.

Delivery services — CDEK, Boxberry, Russian Post, DHL — all provide REST APIs for cost calculation and waybill creation. Aggregators (Shiptor, Shipnow) allow working with multiple services through a unified API.

Payment Gateways

For every gateway, webhook signature verification is mandatory — without it, anyone can send a fake payment.succeeded. Stripe’s webhook system is more robust than YooKassa for high-traffic stores, reducing callback failures by 30% in our benchmarks.

How to Choose Between CMS and Custom Development?

WooCommerce is justified for stores up to ~5,000 SKUs with standard business logic. Quick start, huge plugin ecosystem. Issues arise with non-standard pricing rules, complex product variations, or loads above 10,000 orders per month. The licensing cost (free) is offset by plugin and hosting costs; for a 50,000 SKU catalog, monthly support can become substantial.

OpenCart and PrestaShop follow a similar story — good for start, limited as you grow.

Custom development on Laravel is for:

• Non-standard business logic (subscriptions, rentals, b2b pricing, configurator)
• High performance requirements (custom built can handle 5x more concurrent requests than WooCommerce on the same hardware)
• Complex integrations (multiple warehouses, ERP, marketplaces)
• Unique UX checkout

How We Develop an E-commerce Store: Step-by-Step Process

1. Analytics and Design. Gather requirements, clarify business processes, model domain logic. Output: technical specification and architecture diagram.
2. Backend and API. Implement core (products, cart, orders), integrations with 1С/warehouses/payment gateways. Use Laravel 11 with Repository pattern, queues for async operations.
3. Frontend and Checkout. Set up React 18 / Next.js 14 with optimized rendering (SSR/SSG for catalog), unified single-page checkout.
4. Testing. Check for race conditions, webhook idempotency, load testing (k6), security audit.
5. Deploy and Monitoring. Deploy on Vercel / Docker / dedicated server, connect Sentry and Uptime.

SEO for E-commerce

Canonical and Duplication. Faceted filtering generates thousands of URLs (?color=red&size=M&sort=price). Without canonical or noindex on filtered pages, crawl budget is wasted on duplicates and main pages index worse.

Structured data. Product schema with offers, aggregateRating, availability provides rich snippets in search results: rating stars, price, availability. Boosts CTR.

Core Web Vitals on product pages. The hero image is often the LCP element. Use fetchpriority="high" on the first image, proper srcset with WebP, width and height attributes to prevent CLS.

What You Get After Completion

Upon project completion, you receive:

• Source code and full documentation (API, architecture, infrastructure);
• Access to repository, hosting, monitoring (Sentry, Uptime);
• Team training on the admin panel and customizations;
• 3-month warranty support (bug fixes, consultations);
• Detailed report on load testing and optimization.

Timeline Estimates

Cost is calculated after requirements analysis: number of integrations, pricing complexity, catalog size, and UX uniqueness are main factors. Get a free estimate — book a consultation.

Pre-Launch Checklist

• Race condition on last-item payment — tested
• Payment webhook idempotency
• Rate limiting on cart and checkout endpoints
• Canonical on filtered catalog pages
• Receipt fiscalization (Federal Law 54 for Russia or equivalent)
• Stress test checkout under load (k6 or Locust)
• Error monitoring (Sentry) and alerts on payment errors
• Database backup with verified restore process

We guarantee every project passes this checklist before release. Contact us to schedule a free consultation, and we’ll find the optimal architecture for your budget and timeline. Request an estimate for your e-commerce project today.