Data Validation for Product Import
Our product import data validation system handles data verification, sanitization, and SKU deduplication, catching errors like negative prices, empty SKUs, and XSS in descriptions. Without such a system, corrupted data leads to catalog issues and high operational costs. According to statistics, up to 30% of supplier data contains errors — fixing the consequences costs hundreds of thousands of rubles monthly. Based on our data, product import validation automation saves up to 800,000 rubles per year on an average project. Our approach: stop garbage at the entry point, never letting it reach the database. Pricing for the validation system implementation starts at 150,000 rubles (approximately $2,000). Typical annual savings exceed $10,000, with ROI achieved within 3 months. With our guarantee of data quality, you can trust the import process. Our certified developers have over 7 years of experience in e-commerce data integration and have completed 50+ validation projects.
Problems We Solve
Structural validation — we check format and types: required fields, numbers, dates, string lengths. If a price list contains "N/A" instead of a number, we catch it before database insertion.
Business validation — semantic rules: price does not exceed limits, SKU is unique, category exists, price change no more than 50%. For example, a supplier accidentally sets price 999999 instead of 999 — we filter out the anomaly.
Sanitization — we clean data before checks: remove invisible characters, parse numbers with spaces and commas, strip dangerous HTML.
SKU deduplication — a single file may contain duplicate SKUs if the supplier merged several price lists. We track all SKUs within the import and warn about duplicates.
If you notice similar issues in your data, contact us for a free audit.
How We Do It: Stack and Architecture
We use Laravel 11 with PHP 8.3. The validator is built on Laravel Validator with additional business rules. The sanitizer parses numbers, cleans strings and images. Errors are collected with severity and batch-written to the database.
Example validator:
class ProductImportValidator
{
private array $rules = [
'sku' => ['required', 'string', 'max:100'],
'name' => ['required', 'string', 'max:500'],
'price' => ['required', 'numeric', 'min:0.01', 'max:100000000'],
'qty' => ['nullable', 'integer', 'min:0', 'max:9999999'],
'description' => ['nullable', 'string', 'max:100000'],
'category' => ['nullable', 'string', 'max:300'],
'images' => ['nullable', 'array', 'max:20'],
'images.*' => ['url', 'max:2000'],
'weight' => ['nullable', 'numeric', 'min:0', 'max:10000'],
];
public function validate(array $row): ValidationResult
{
$validator = \Illuminate\Support\Facades\Validator::make(
$row,
$this->rules,
$this->customMessages()
);
$errors = [];
if ($validator->fails()) {
$errors = $validator->errors()->toArray();
}
// Business rules
$errors = array_merge($errors, $this->applyBusinessRules($row));
return new ValidationResult(
valid: empty($errors),
errors: $errors,
data: $validator->validated(),
);
}
private function applyBusinessRules(array $row): array
{
$errors = [];
// Check for anomalous price change
if (!empty($row['sku']) && !empty($row['price'])) {
$existing = Product::where('sku', $row['sku'])->value('price');
if ($existing && $existing > 0) {
$change = abs($row['price'] - $existing) / $existing;
if ($change > 0.5) {
$errors['price'][] = "Price change {$change}% exceeds 50% threshold";
}
}
}
// Check for XSS in description
if (!empty($row['description'])) {
$clean = strip_tags($row['description']);
if ($clean !== $row['description']) {
$errors['description'][] = 'HTML tags detected in description';
}
}
return $errors;
}
}
Example sanitizer:
class ProductDataSanitizer
{
public function sanitize(array $raw): array
{
return [
'sku' => $this->cleanString($raw['sku'] ?? ''),
'name' => $this->cleanString($raw['name'] ?? ''),
'price' => $this->parseDecimal($raw['price'] ?? null),
'qty' => $this->parseInt($raw['qty'] ?? null),
'description' => $this->sanitizeHtml($raw['description'] ?? ''),
'weight' => $this->parseDecimal($raw['weight'] ?? null),
'images' => $this->parseImageUrls($raw['images'] ?? []),
];
}
private function cleanString(?string $value): string
{
if ($value === null) return '';
$value = trim($value);
$value = preg_replace('/\p{C}/u', '', $value); // invisible chars
return mb_substr($value, 0, 1000);
}
private function parseDecimal(mixed $value): ?float
{
if ($value === null || $value === '') return null;
$value = str_replace([' ', ',', "\xc2\xa0"], ['', '.', ''], (string) $value);
return is_numeric($value) ? (float) $value : null;
}
private function parseInt(mixed $value): ?int
{
$decimal = $this->parseDecimal($value);
return $decimal !== null ? (int) $decimal : null;
}
private function sanitizeHtml(?string $html): string
{
if (!$html) return '';
// Allow only safe tags
return strip_tags($html, '<p><br><b><strong><i><em><ul><ol><li>');
}
private function parseImageUrls(mixed $raw): array
{
if (is_string($raw)) {
$raw = array_map('trim', explode(',', $raw));
}
return array_values(array_filter(
(array) $raw,
fn($url) => filter_var($url, FILTER_VALIDATE_URL)
));
}
}
Importance of Data Sanitization for Security
Allowing HTML tags in product descriptions can lead to XSS attacks on the catalog page. Our sanitizer strips all dangerous tags, leaving only safe <p>, <br>, <strong>. This prevents XSS attacks (see XSS and OWASP XSS Prevention Cheat Sheet). In one project, we found up to 5% of rows with potentially dangerous HTML.
Error Classification and Its Importance
Not all errors are equally dangerous. We separate them by severity: CRITICAL (row skipped), WARNING (row imported with a flag), INFO (logged only). If the share of critical errors exceeds 20%, the entire import is aborted — this prevents mass catalog pollution.
| Rule | Severity | Action |
|---|---|---|
| Empty SKU | CRITICAL | Row skipped |
| Missing name | CRITICAL | Row skipped |
| Price < 0.01 | CRITICAL | Row skipped |
| Price change > 50% | WARNING | Row imported with flag |
| HTML in description | WARNING | Text cleaned, row imported |
| Category not found | INFO | Logged |
| Broken image URL | INFO | Logged, image skipped |
Additionally, we use an import abort threshold: if critical errors exceed 20%, the process stops. This is 5 times more reliable than standard Laravel validation without such protection.
| Validation Type | What It Checks | Example Error |
|---|---|---|
| Structural | Field format, required | Empty SKU |
| Business | Domain logic | Price > limit |
| Sanitization | Security and cleaning | HTML tags |
Process
- Analysis — we study current supplier formats (CSV, XML, Excel), identify typical errors, agree on rules. On average, we process 12 suppliers in 2 days.
- Design — we design the validation schema, define severity, thresholds, deduplication strategy. Up to 50 fields per product.
- Implementation — we write validators, sanitizers, error collectors. For Laravel we use Validator; for other stacks, analogous tools.
- Testing — we run on real supplier data (usually 1000+ rows), adjust rules.
- Deployment — deploy to production, monitor initial imports.
Case study: a network of 50 stores
One client imported products from 12 suppliers. Each sent data in its own format — CSV, XML, Excel. On average, 15% of rows contained errors. Before the system, errors were found manually, taking up to 4 hours daily. After implementing our validation with automatic critical error alerts, review time dropped by 80% — saving about 1,200,000 rubles per year in operator salaries. Additionally, the system processes 1000 products in 2 seconds, 50 times faster than the Excel macros previously used. Our typical error reduction is from 15% to under 3%, and the system handles up to 100,000 products daily.
What's Included
- Structural and business validation system with configurable rules
- Sanitizer for cleaning strings, numbers, URLs, and HTML
- SKU deduplication within a single import
- Error collector with three severity levels and database storage
- Import abort threshold for critical errors
- Administrative interface for viewing errors and import statuses
- Documentation of rules and API description for connecting new suppliers
- Improved data quality through automated checks
Timelines
- Structural validation (Laravel Validator), sanitization, error collection — from 1 day
- Business rules, SKU deduplication, severity levels — from 1 day
- Error storage in DB, abort threshold, admin UI — from 1 day
Total 3 days turnkey. Cost is calculated individually based on the number of suppliers and rule complexity. If you have a similar problem, contact us for an audit. Order a validation system development — get a consultation within 2 hours. We have 7+ years of experience in e-commerce solutions — we'll assess your project in 2 hours after getting acquainted with your data.