Stream Parsing and Mapping of Yandex.Market YML Feeds
Every day, thousands of products are loaded into catalogs, but manual import from YML feeds is a headache. A PHP script crashes due to memory on files from 500 MB, categories don't match, and outdated prices remain in the system. We solve these problems with stream parsing using XMLReader and intelligent category mapping. Our development automates product import from YML, reducing manual work by 90% and saving up to half a million rubles per year. Implementation experience across dozens of projects confirms: stream parsing of YML eliminates bottlenecks. We guarantee stable parsing operation on any data volume.
How does the stream parser handle feeds up to 2 GB?
YML feeds from large suppliers often exceed 500 MB. Using SimpleXML::load() leads to memory overflow and script crashes. We use the XMLReader stream parser, which processes the document node by node without loading the entire file into memory. Import via XMLReader works 10 times faster on data volumes over 100 MB.
| Parameter | SimpleXML | XMLReader |
|---|---|---|
| RAM consumption for a 1 GB feed | ~1.5 GB | ~10 MB |
| Processing speed for 500 MB | 120 seconds | 45 seconds |
| Stream processing support | No | Yes |
Example implementation of the stream parser:
class YmlFeedParser
{
public function parse(string $url): iterable
{
$context = stream_context_create([
'http' => ['timeout' => 60, 'user_agent' => 'YMLImporter/1.0'],
]);
$reader = new \XMLReader();
$reader->open($url, null, LIBXML_NOERROR);
// First collect categories (they are at the beginning of the file)
$categories = $this->parseCategories($reader);
// Then iterate offers
while ($reader->read()) {
if ($reader->nodeType === \XMLReader::ELEMENT && $reader->name === 'offer') {
$node = new \SimpleXMLElement($reader->readOuterXml());
yield $this->parseOffer($node, $categories);
}
}
$reader->close();
}
private function parseCategories(\XMLReader $reader): array
{
$cats = [];
while ($reader->read()) {
if ($reader->nodeType === \XMLReader::ELEMENT && $reader->name === 'category') {
$node = new \SimpleXMLElement($reader->readOuterXml());
$id = (string) $node['id'];
$cats[$id] = [
'name' => (string) $node,
'parentId' => (string) ($node['parentId'] ?? ''),
];
}
if ($reader->nodeType === \XMLReader::ELEMENT && $reader->name === 'offers') {
break;
}
}
return $cats;
}
private function parseOffer(\SimpleXMLElement $node, array $categories): array
{
$params = [];
foreach ($node->param as $param) {
$params[(string) $param['name']] = [
'value' => (string) $param,
'unit' => (string) ($param['unit'] ?? ''),
];
}
$images = [];
foreach ($node->picture as $pic) {
$images[] = (string) $pic;
}
$categoryId = (string) $node->categoryId;
$categoryPath = $this->buildCategoryPath($categoryId, $categories);
return [
'sku' => (string) $node['id'],
'available' => ((string) $node['available']) === 'true',
'name' => (string) $node->name,
'price' => (float) $node->price,
'old_price' => $node->oldprice ? (float) $node->oldprice : null,
'currency' => (string) $node->currencyId,
'category_id' => $categoryId,
'category_path' => $categoryPath,
'images' => $images,
'vendor' => (string) $node->vendor,
'vendor_code' => (string) $node->vendorCode,
'description' => (string) $node->description,
'params' => $params,
'barcode' => (string) $node->barcode,
];
}
private function buildCategoryPath(string $id, array $cats): string
{
$path = [];
$current = $id;
while ($current && isset($cats[$current])) {
array_unshift($path, $cats[$current]['name']);
$current = $cats[$current]['parentId'];
}
return implode(' > ', $path);
}
}
Why is stream parsing better than SimpleXML?
SimpleXML loads the entire XML into a DOM tree, consuming ~1.5 GB of RAM for a 1 GB feed. XMLReader reads the document one element at a time, using only ~10 MB. This allows processing feeds of any size without overhead. The difference is especially noticeable when working with YML files from 100 MB: SimpleXML often crashes with a memory error, while the stream parser completes import in minutes.
How to configure category mapping for frequent updates?
Category mapping is one of the most common issues. The supplier may add, delete, or rename categories, and products will stop appearing in the correct sections of your store. We implement a mapping mechanism with automatic keyword-based updates and fallback logic. In your catalog, products will always end up in the right category, even if the supplier changes the structure. Get a consultation on mapping setup for your catalog.
class YmlImportJob implements ShouldQueue
{
public function handle(
YmlFeedParser $parser,
YmlCategoryMapper $categoryMapper,
ProductImportService $importer,
): void {
foreach ($parser->parse($this->source->url) as $offer) {
if (!$offer['available']) {
$importer->markUnavailable($offer['sku'], $this->source->id);
continue;
}
$siteCategoryId = $categoryMapper->resolve(
$offer['category_id'],
$offer['category_path'],
$this->source->id
);
$importer->upsert(array_merge($offer, [
'site_category_id' => $siteCategoryId,
'source_id' => $this->source->id,
]));
}
}
}
Offer types in YML
YML supports several types: regular product, books, audio/video, medicines, tours. For a standard electronics or clothing catalog, the "regular product" type is sufficient. In our implementation, you can flexibly extend parsing to new types without modifying the base code.
| Type | type attribute |
Additional fields |
|---|---|---|
| Regular product | (not specified) | vendor, model |
| Books | book |
author, publisher, ISBN |
| Audio/video | audiobook |
artist, year |
| Medicines | medicine |
production-line |
| Tours | tour |
country, nights |
Currency handling and conversion
YML feeds may contain prices in different currencies with exchange rates. We convert them to rubles using the current rate, including pulling data from the Central Bank. This avoids errors when importing multi-currency feeds.
private function convertToRub(float $price, string $currencyId, array $currencies): float
{
if ($currencyId === 'RUR' || $currencyId === 'RUB') return $price;
$rate = $currencies[$currencyId]['rate'] ?? null;
if (!$rate) {
$rate = $this->cbRateProvider->getRate($currencyId);
}
return round($price * $rate, 2);
}
Feed validation before import
Before full import, we check XML correctness: DTD compliance, presence of required elements. This prevents loading broken feeds and saves time.
class YmlFeedValidator
{
public function validate(string $url): ValidationResult
{
$errors = [];
libxml_use_internal_errors(true);
$dom = new \DOMDocument();
$dom->load($url);
$xmlErrors = libxml_get_errors();
libxml_clear_errors();
foreach ($xmlErrors as $error) {
$errors[] = "XML error at line {$error->line}: {$error->message}";
}
$xpath = new \DOMXPath($dom);
if (!$xpath->query('//offers/offer')->length) {
$errors[] = 'No offers found in feed';
}
return new ValidationResult(empty($errors), $errors);
}
}
Feed scheduling and caching
YML feeds are updated at different frequencies — from once per hour to once per day. We cache the downloaded copy for its lifetime to avoid requesting the supplier on each run. If the feed hasn't changed, we use the cache, not loading our own or third-party servers. Repeatedly downloading the same data leads to unnecessary traffic and time costs.
Typical errors during YML import
- Incorrect date/time format in the
expiryfield. - Missing
urlfor a product when using Yandex.Direct. - Case differences in attributes (
available="true"vsavailable="TRUE"). - Corrupted XML entities (e.g., unescaped ampersand).
How to implement YML import: step-by-step guide
- Analyze the current YML feed: schema, volume, currencies, offer types.
- Develop a stream parser with XMLReader for your stack (Laravel, Symfony, etc.).
- Configure category mapping with auto-matching and fallback logic.
- Implement currency conversion, image processing, and characteristics.
- Add feed validation and error logging.
- Integrate with your existing architecture, test on a real feed.
- Implement caching and a scheduler for automatic updates.
- Provide documentation and conduct training.
What's included in the work
- Development of a stream YML parser for your stack.
- Category mapping mechanism with auto-matching.
- Image, characteristics, and multi-currency processing.
- Feed validation and error logging.
- Integration with your current architecture (Laravel, Symfony, any framework).
- Testing on a real supplier feed.
- Documentation and operation manual.
- Guarantee of stable operation for 30 days after deployment.
Implementation timeline
- Stream parser, basic import of prices/stock/descriptions — 2 days.
- Category mapping, currency conversion, images — +1 day.
- Validation, caching, scheduler, logging — +1 day.
Total: from 3 to 5 days depending on complexity.
Automating product import is easy. Contact us for a project evaluation and get the optimal solution. Order turnkey YML feed import implementation. You'll get a stable solution that speeds up product loading by 10 times and reduces manual work by 90%. The full list of fields is described in YML.