Developing a new arrivals parser bot starts with analyzing the supplier's catalog structure. Parsing web pages allows you to automate data collection. We choose the optimal delta detection strategy: by date added, by the 'New Arrivals' section, or by comparing the SKU list. The latter method is universal and does not depend on layout. It allows finding new products even if the supplier does not highlight new arrivals visually.
The bot saves up to 80% of the team's time on monitoring. Instead of browsing dozens of pages daily, you receive a notification with a list of new SKUs. The cost of manual monitoring errors is high—a missed hot item or a late start to sales. Automation solves this problem.
How Date-Based New Arrivals Detection Works
If a product card has an added date, we parse the catalog pages until we encounter records older than the last snapshot. This saves traffic and time. Example implementation in PHP:
// app/Services/NewArrivals/DateBasedDetector.php
class DateBasedDetector
{
public function detectNew(string $categoryUrl, \DateTimeInterface $since): array
{
$page = 1;
$newProducts = [];
do {
$items = $this->scrapePage($categoryUrl, $page);
$hasOlderItems = false;
foreach ($items as $item) {
$itemDate = $this->parseDate($item['date_added'] ?? '');
if ($itemDate && $itemDate < $since) {
$hasOlderItems = true;
break;
}
if (!$this->existsInDatabase($item['sku'])) {
$newProducts[] = $item;
}
}
$page++;
} while (!$hasOlderItems && count($items) > 0);
return $newProducts;
}
}
This method works well when the supplier clearly indicates dates. If dates are absent, we use other strategies.
What New Arrivals Detection Strategies Exist?
Let's compare three main approaches:
| Strategy | Accuracy | Dependence on Structure | When to Use |
|---|---|---|---|
| By date added | High | Medium—requires date on page | If the supplier consistently provides dates |
| By 'New Arrivals' section | Medium | Low—section URL is enough | If a separate new arrivals page exists |
| By SKU comparison | High | Low—only SKU list needed | Universal method, independent of markup |
The SKU comparison method is 2–3 times more accurate because it does not rely on page metadata. Its implementation:
// app/Services/NewArrivals/SkuDiffDetector.php
class SkuDiffDetector
{
public function detect(int $supplierId, array $currentSkus): array
{
$previousSnapshot = SupplierSnapshot::where('supplier_id', $supplierId)
->latest()
->first();
if (!$previousSnapshot) {
$this->saveSnapshot($supplierId, $currentSkus);
return [];
}
$previousSkus = $previousSnapshot->sku_list;
$newSkus = array_diff($currentSkus, $previousSkus);
$removedSkus = array_diff($previousSkus, $currentSkus);
$this->saveSnapshot($supplierId, $currentSkus);
if (!empty($removedSkus)) {
Log::info("Supplier #{$supplierId}: removed SKUs", ['skus' => $removedSkus]);
SupplierProductsRemoved::dispatch($supplierId, $removedSkus);
}
return $newSkus;
}
private function saveSnapshot(int $supplierId, array $skus): void
{
SupplierSnapshot::create([
'supplier_id' => $supplierId,
'sku_list' => $skus,
'sku_count' => count($skus),
'captured_at' => now(),
]);
}
}
Here, SKU snapshots are stored in the database, and old ones are automatically deleted after 90 days to avoid cluttering storage.
Complete Detection and Processing Cycle
Let's put everything into a single Job that runs on a schedule:
// app/Jobs/CheckSupplierNewArrivals.php
class CheckSupplierNewArrivals implements ShouldQueue
{
public int $tries = 3;
public int $timeout = 600;
public function handle(
SupplierScraper $scraper,
SkuDiffDetector $detector,
NewArrivalsNotifier $notifier
): void {
$supplier = Supplier::findOrFail($this->supplierId);
$allProducts = $scraper->scrapeAllProductSkus($supplier);
$currentSkus = array_column($allProducts, 'sku');
$newSkus = $detector->detect($this->supplierId, $currentSkus);
if (empty($newSkus)) {
Log::info("No new arrivals for supplier #{$this->supplierId}");
return;
}
$newProducts = array_filter(
$allProducts,
fn($p) => in_array($p['sku'], $newSkus)
);
$notifier->notify($supplier, $newProducts);
if ($supplier->auto_import_new_arrivals) {
foreach ($newProducts as $product) {
ImportNewSupplierProduct::dispatch($this->supplierId, $product)
->onQueue('imports');
}
} else {
foreach ($newProducts as $product) {
PendingImport::create([
'supplier_id' => $this->supplierId,
'data' => $product,
'status' => 'pending_review',
]);
}
}
Log::info("Found new arrivals", [
'supplier_id' => $this->supplierId,
'count' => count($newProducts),
]);
}
}
The Job runs in a queue, supports up to 3 retries, and has a timeout of 10 minutes, which is sufficient for catalogs up to 10,000 SKUs.
Notifications and Integrations
You can be instantly informed about found new arrivals. We support email, Slack, Telegram, and webhooks. Example notification by email and Slack:
// app/Notifications/NewSupplierArrivalsNotification.php
class NewSupplierArrivalsNotification extends Notification implements ShouldQueue
{
use Queueable;
public function via($notifiable): array
{
return ['mail', 'slack'];
}
public function toMail($notifiable): MailMessage
{
return (new MailMessage)
->subject("New arrivals: {$this->supplier->name} ({$this->count} products)")
->line("Found {$this->count} new products from supplier **{$this->supplier->name}**")
->line("Detection date: " . now()->format('d.m.Y H:i'))
->action('View new arrivals', route('admin.pending-imports.index', [
'supplier_id' => $this->supplier->id,
]))
->line('Products await review before publishing.');
}
public function toSlack($notifiable): SlackMessage
{
return (new SlackMessage)
->content(
"🆕 *{$this->supplier->name}*: {$this->count} new products\n" .
implode("\n", array_map(
fn($p) => "• {$p['sku']} — {$p['name']}",
array_slice($this->products, 0, 10)
))
);
}
}
The notifications include the first 10 products; the full list is available in the admin panel.
Manual Review Queue
New products often require manual review: category, SEO description, photos. For this, we create a moderation interface with approval or rejection functionality:
// app/Http/Controllers/Admin/PendingImportController.php
class PendingImportController extends Controller
{
public function index(Request $request): Response
{
$pending = PendingImport::query()
->with('supplier')
->when($request->supplier_id, fn($q, $id) => $q->where('supplier_id', $id))
->where('status', 'pending_review')
->orderBy('created_at', 'desc')
->paginate(50);
return Inertia::render('Admin/PendingImports/Index', [
'imports' => $pending,
]);
}
public function approve(PendingImport $import): RedirectResponse
{
ImportNewSupplierProduct::dispatch($import->supplier_id, $import->data);
$import->update(['status' => 'approved']);
return back()->with('success', 'Product sent for import');
}
public function reject(PendingImport $import, Request $request): RedirectResponse
{
$import->update([
'status' => 'rejected',
'reject_reason' => $request->reason,
]);
return back()->with('success', 'Product rejected');
}
}
The moderator sees all new arrivals in one list and can quickly review and send products to the store.
What's Included in the Parser Bot Development?
The scope of work includes:
- Analysis of supplier catalogs and selection of the optimal detection strategy.
- Development of the SKU collection module (support for authentication, pagination, captcha).
- Implementation of delta detection with snapshot storage.
- Setup of notifications (email, Slack, Telegram, webhooks).
- Development of a manual review queue with a moderation interface.
- Creation of operational documentation.
- Testing on real supplier data.
- Staff training on the system.
Timelines and Cost
Development of a detector for one supplier with notifications and queue takes from 4 to 6 working days. The cost is calculated individually—depends on catalog complexity, number of suppliers, and required integrations. We guarantee stable bot operation and provide operational documentation.
Why Order Development from Us?
Our engineers have over 5 years of experience in parsing complex catalogs. During this time, we have implemented over 50 projects for online stores and marketplaces. We guarantee adherence to deadlines, post-launch support, and setup assistance. We will evaluate your project and propose the optimal solution. Contact us for a consultation.
How does the bot handle parsing errors?
For error handling, we use retries with exponential backoff. If a page fails to load, the bot waits 30 seconds and retries. After three failures, an administrator notification is sent. An error log with URL and response code details is also maintained.| Stage | Duration |
|---|---|
| Analysis and design | 1–2 days |
| Detector development | 1–2 days |
| Notification and queue integration | 1 day |
| Testing and debugging | 1 day |
| Documentation and training | 0.5 days |
The parser bot pays off within 2–3 months due to time savings for managers and reduced missed sales. Get a consultation on your project—we will calculate the cost and timeline.