Price Monitoring Bot: Automatic Competitor Price Parsing on Schedule
Imagine this: your competitor drops prices by 15% on their top 20 SKUs, and you find out a week later when sales have already plummeted. That's exactly what happened to one of our clients, losing 30% of their margin in a month. To prevent such scenarios, we build automated price parser bots that run on a schedule, store history, and can independently adjust your prices based on flexible rules. With 7+ years of experience and 40+ e-commerce projects delivered, we guarantee reliable parser operation at every stage. Contact us for a custom price monitoring bot tailored to your catalog.
Why Manual Price Monitoring Is Inefficient
- Stale data: you check prices once a week, while competitors change them daily. By the time you react, you've already lost sales or margin.
- Anti-bot measures: competitors use Cloudflare, captchas, dynamic selectors. Manual collection falls short—you need a headless browser with proxy rotation.
- Normalization: prices come in different formats: with/without tax, different currencies, whitespace, symbols. Without normalization, comparisons are meaningless.
How We Build the Monitoring System
We use Laravel 11 on PHP 8.3, PostgreSQL for storage, Redis for queues, and Playwright for dynamic content parsing. The architecture is microservice-based—each competitor is processed independently.
Scheduler
→ ScrapeCompetitorPrices Job (per competitor)
→ CompetitorScraper (HTTP/Playwright)
→ PriceNormalizer
→ PriceHistoryRepository (INSERT)
→ PriceChangeDetector
→ AlertDispatcher (if change > threshold)
→ RepricingEngine (if autoprice enabled)
Data Model
// database/migrations/create_competitor_prices_table.php
Schema::create('competitor_prices', function (Blueprint $table) {
$table->id();
$table->foreignId('product_id')->constrained();
$table->foreignId('competitor_id')->constrained();
$table->string('competitor_sku')->nullable();
$table->string('competitor_url');
$table->decimal('price', 12, 2);
$table->decimal('sale_price', 12, 2)->nullable();
$table->boolean('in_stock')->default(true);
$table->timestamp('scraped_at');
$table->timestamps();
$table->index(['product_id', 'competitor_id', 'scraped_at']);
});
// History for charts and analytics
Schema::create('competitor_price_history', function (Blueprint $table) {
$table->id();
$table->foreignId('product_id')->constrained();
$table->foreignId('competitor_id')->constrained();
$table->decimal('price', 12, 2);
$table->boolean('in_stock');
$table->date('recorded_date');
$table->timestamps();
$table->unique(['product_id', 'competitor_id', 'recorded_date']);
});
Competitor Configuration
// config/competitors.php
return [
'competitor_a' => [
'name' => 'Shop A',
'base_url' => '',
'selectors' => [
'price' => '.current-price',
'in_stock' => '.in-stock-badge',
],
'price_regex' => '/[\d\s]+/',
'url_pattern' => '/product/{sku}',
'requires_js' => false,
'request_delay' => [1000, 3000], // ms
],
'wildberries' => [
'name' => 'Wildberries',
'type' => 'api',
'scraper' => WildberriesPriceScraper::class,
'request_delay' => [500, 1500],
],
];
Base Price Scraper
// app/Services/PriceMonitor/CompetitorPriceScraper.php
class CompetitorPriceScraper
{
public function __construct(
private Client $httpClient,
private array $config
) {}
public function scrapePrice(string $url): ?PriceData
{
try {
$html = $this->fetch($url);
$crawler = new Crawler($html);
$priceText = $crawler->filter($this->config['selectors']['price'])
->first()
->text('');
$price = $this->extractPrice($priceText);
if ($price === null) return null;
$salePrice = null;
if (!empty($this->config['selectors']['sale_price'])) {
$salePriceText = $crawler->filter($this->config['selectors']['sale_price'])
->first()->text('');
$salePrice = $this->extractPrice($salePriceText);
}
$inStock = true;
if (!empty($this->config['selectors']['in_stock'])) {
$inStock = $crawler->filter($this->config['selectors']['in_stock'])->count() > 0;
}
return new PriceData(
price: $price,
salePrice: $salePrice,
inStock: $inStock,
);
} catch (\Exception $e) {
Log::warning("Price scrape failed for {$url}: {$e->getMessage()}");
return null;
}
}
private function extractPrice(string $text): ?float
{
$cleaned = preg_replace('/[^\d,.]/', '', str_replace(' ', '', $text));
$cleaned = str_replace(',', '.', $cleaned);
return is_numeric($cleaned) ? (float) $cleaned : null;
}
}
Job with Change Detector
// app/Jobs/MonitorCompetitorPrice.php
class MonitorCompetitorPrice implements ShouldQueue
{
public int $tries = 3;
public array $backoff = [60, 120, 300];
public function __construct(
private int $productId,
private int $competitorId,
private string $competitorUrl
) {}
public function handle(
CompetitorPriceScraper $scraper,
PriceChangeDetector $detector,
RepricingEngine $repricer
): void {
$config = Competitor::find($this->competitorId)->config;
$priceData = $scraper->scrapePrice($this->competitorUrl);
if ($priceData === null) return;
// Save current price
$record = CompetitorPrice::updateOrCreate(
[
'product_id' => $this->productId,
'competitor_id' => $this->competitorId,
],
[
'price' => $priceData->price,
'sale_price' => $priceData->salePrice,
'in_stock' => $priceData->inStock,
'scraped_at' => now(),
]
);
// Save to history (once per day)
CompetitorPriceHistory::firstOrCreate(
[
'product_id' => $this->productId,
'competitor_id' => $this->competitorId,
'recorded_date' => today(),
],
[
'price' => $priceData->price,
'in_stock' => $priceData->inStock,
]
);
// Detect changes
if ($record->wasChanged('price')) {
$change = $detector->analyze($record);
if ($change->isSignificant()) {
Notification::route('mail', config('monitoring.alert_email'))
->notify(new CompetitorPriceChangedNotification($record, $change));
}
// Autoreprice
if (config('repricing.enabled')) {
$repricer->recalculate($this->productId);
}
}
}
}
Setting Up the Monitoring Schedule
The schedule is tailored to your needs. High priority items every 2 hours, bulk items once per night. Laravel's scheduler prevents job overlapping.
For price history aggregation, we run a daily artifact that summarizes data into one record per day—saving space and speeding up chart queries.
The automated parser processes 1000 SKUs in 10 minutes, which is 40 times faster than manual collection.
What Data Is Saved in Price History?
We store price, sale price, stock availability, and date. History is kept daily, enabling trend charts and seasonal analysis. For example, you can see that every winter a competitor drops prices by 10% on certain categories, allowing you to prepare in advance.
Automatic Repricing Strategies
| Strategy | Description | When to Use |
|---|---|---|
| beat_lowest | Set price N rubles below the minimum | Aggressive market share grab if margin allows |
| match_lowest | Match the lowest competitor price | Hold position in highly competitive niches |
| beat_average | Price N rubles below the average | Balance between margin and competitiveness |
| percentile | Price at a given percentile (e.g., 25th) | Maintain positioning (premium or budget) |
Each strategy accounts for a minimum change of 0.5% to avoid price jumps.
Monitoring Dashboard
Key metrics to display:
| Metric | Description |
|---|---|
| Products where we are more expensive | Count and % of catalog |
| Average delta to minimum | Average % difference from lowest price |
| Products with change > 5% | Require manual check |
| Products out of stock at competitors | Potential for price increase |
| The dashboard helps quickly spot anomalies, such as a sudden price change by a major competitor. |
Typical Mistakes When Implementing Price Monitoring
- Ignoring selector changes. If a competitor's site changes layout, the parser fails. Always use fallback selectors and failure alerts.
- Too frequent requests. Cloudflare protection blocks the IP. Maintain delays and use proxy rotation.
- No thresholds in auto-repricing. Without a minimum change percentage, you'll change prices by pennies, annoying customers and losing trust.
How to Configure the Bot in 3 Steps
- Add competitor URLs – Provide URLs for each product or a pattern using SKU. We'll configure selectors and API integrations.
- Set scraping schedule – Define frequency per competitor or product group (e.g., every 2 hours for top sellers, once daily for others).
- Define repricing rules – Choose a strategy (beat, match, percentile) and set minimum change thresholds (default 0.5%). The bot will automatically update your prices.
What's Included in the Work
- Documentation: monitoring architecture description, instructions for adding a new competitor, configuration of repricing rules.
- Access: we provide access to the server, queues, dashboard.
- Training: we conduct a demo session of the dashboard and rules for your team.
- Support: free support for 1 month after launch.
Timeline and Pricing
Development of a basic system (1 competitor + history + alerts) takes 4–6 business days and starts at $1,500. A full system with auto-repricing and dashboard for 5+ competitors takes 10–14 days and starts at $5,000. Savings from reduced manual labor typically exceed $2,000 per month. For a mid-size catalog of 10,000 SKUs, clients report saving 250 hours monthly, boosting net margin by 3% within 3 months. Order a custom price monitoring system—contact us for an assessment of your project.
Example Configuration Detail
For a typical competitor, we configure selectors for price, sale price, and stock status. The bot uses Playwright to render JavaScript if needed. Below is a sample config for a static site:competitor:
name: Example Shop
base_url: ''
selectors:
price: '.price-amount'
in_stock: '.in-stock'
price_regex: '/[\d.,]+/'
url_pattern: '/product/{sku}'
requires_js: false
request_delay: 1000
Playwright — official documentation at playwright.dev