News parser: where feed collection breaks at scale
On a dozen sources, a news parser seems to work: the feed is read, materials appear. Across hundreds of publications with different publication frequencies, news feed collection fails in ways the team learns about from readers rather than from the system.
Some sources provide publication time in their own time zone, some provide only a date without an hour, and the order of items changes on repeated polling, so the "latest N entries" do not match what actually came out in a day—this is how missed publications appear.
A separate headache is duplicates when the feed is updated. A publisher changes the link structure, appends service parameters to the address, moves the material to another category—and the same news item enters the database a second time.
As long as the record key depends on field order and decorations in the link, deduplication relies on the headline, and it is edited after the material is published.
The bottleneck in collection is usually one: a single polling cycle across all sources. A slow response from one publication delays the others, the interval grows, and at peaks tasks time out—the most active feeds remain unpolled exactly when there is the most news. Then manual backfilling begins: an editor opens sites and copies materials by hand.
We explain how this is addressed: a separate queue per source, its own polling interval instead of a common round, a stable record key, and a second pass over the time window after a failure.
What effect managed news collection provides
The completeness of publication collection no longer depends on how a source serves its feed.
Some materials were lost in pagination and markup—up to 10–15% misses; after moving collection to several channels at once (feed, search results, archive), misses fall to fractions of a percent and are confirmed by reconciliation with the source.
The delay before a material appears is measured in minutes.
The polling interval is adaptive: active sources are checked more often, quiet ones less often, and during a spike in publications the queue does not grow—a new record ends up in the system within a few minutes, not by the next manual run.
The share of manual backfills drops from 10–15% of issues to isolated cases. The editor intervenes only where a source provided incomplete text or a broken link, and such a record arrives with a note about the reason rather than being discovered during layout.
A separate effect is a whole class of failures eliminated when a source changes its layout.
Parsing rules are moved into configuration alongside templates for common platforms such as WordPress: migration to a new theme breaks only one adapter, other sources continue to be collected, and the discrepancy is visible in the parsing log.
In sum, this means stable feed collection: predictable completeness, a short delay before a record appears, and a short list of places for an engineer to check.
Collection formats: scheduled and event-driven news parser
A single source usually has several access points to content, and the choice between them determines both the completeness of the sample and the cost of maintenance. Below are the modes we combine in one collector, and the honest limitations of each.
| Option | When it fits | Limitations |
|---|---|---|
| Parsing an open feed (RSS, Atom, JSON Feed) | The source itself publishes a feed | The feed often contains only a summary and truncated text, dates in different zones, and some publications have a publication delay |
| Crawling source sections | There is no feed or it is incomplete | Markup bindings break during redesign; pagination, mirror crawling, and a request limit are needed |
| Parsing the full article text | The full text is needed, not a summary | The sample includes navigation, ads, and "read also" blocks; cleaning rules and link canonicalization are needed |
| Event-driven polling of sources | Reaction speed matters | Not all sources provide notifications; a fallback scheduled path and protection against duplicate delivery are needed |
| Scheduled collection | The base mode for the whole array | The interval is a compromise between freshness and load; there are daily publication windows and limits on request counts |
How we combine modes
In practice we do not choose one option: scheduled collection covers the entire list of sources, while event-driven polling is enabled where delay is critical.
Mandatory elements of any mode are deduplication by record identifier and text hash, publication window depth, and predictable behavior when a source is unavailable: a skip with a note and retry in the next cycle instead of stopping all collection.
If a source tightens collection restrictions, some sources are switched to less frequent crawling—the sample remains complete, and the load on the publisher decreases.
Stages of parser implementation: from source audit to handover
A news parser rarely breaks in the code—more often the source fails: markup changes, rate limiting is enabled, some publications move to the archive.
Therefore, the work proceeds sequentially, from source audit to handover, and each next step builds on the result of the previous one.
- Source audit. We analyze the list of feeds: which ones provide full text, which only a summary, in what form dates arrive, whether there is an archive and request rate limits. The output is a list of sources with priority and crawling mode.
- Field normalization schema. We bring publications from different feeds to a unified set: headline, date, body, link, author, categories. Separately, we describe bringing dates to a single time zone and handling encodings—otherwise duplicates and records without content appear in the storage.
- Feed crawling rules. We define polling frequency, backfill depth on first run, and deduplication by link and text fingerprint. Here we also describe behavior when a source is unavailable and re-collection of the missed window.
- Validation on a reference sample. We assemble a manually labeled set of publications for a period and compare the result against it: share of correctly extracted fields, number of false duplicates, lost records. Rules and markup parsing are adjusted until the result is stable.
- Handover of the solution to the customer. We provide source configurations, a description of the field schema, instructions for connecting a new feed, and failure alerts that trigger before a reader notices the problem.
Then comes maintenance: updating rules for changed markup, adding feeds, and quality control of the export keep the flow of publications continuous without rewriting the core.
Handover package: configurations, crawling schemas, regulations
The package is assembled so that collection can be run and developed by your own engineers, not only by the authors of the solution.
Each artifact is tied to a specific source and answers the question "what to do if it changes"—this distinguishes a working handover from a folder of documents.
- Source descriptions: a map of feeds and sections—output format, publication frequency, time zone, list of fields collected.
- Feed crawling rules: polling intervals, crawling order, limit of parallel requests, pause under rate limiting, incremental collection by date and identifier.
- Data access policies: where automated collection is permitted, how authorization is stored and rotated, what we do upon a rights holder's request.
- Schemas for bypassing restrictions: backup routes through mirrors and official interfaces, proxy pools, retries with increasing pause.
- Collection logs: for each source—how many records arrived, how many were filtered out during validation, delay relative to publication, error share.
- Data model: deduplication keys, rules for normalizing text and dates, retention and archiving periods.
- Maintenance regulations: alert thresholds, response times for a failed source, checklist for markup changes on the site side.
- Expansion instructions: how to add a source via configuration and test it on a sample.
Artifacts sit next to the code and are covered by checks: a markup change is caught by a test, not by contacting us. Adding a new source comes down to editing the configuration.
Case study: how feed collection stopped failing under peak loads
A story about an industry newsroom: the feed was collected manually, sources were crawled once a day, so announcements and releases came out with a delay, and some materials were backfilled at night in emergency mode.
At publication peaks—a major event, a conference, reporting season—the queue of sources was not processed, and the issue shifted by hours.
What it was before
Collection relied on manual operations: an editor went through sources, copied materials, and compared them with already published ones by eye.
Measurements over a month: up to a third of records were repeats of the same release, 5–7 emergency backfills per week, and up to 15% of publications were lost during a peak wave.
Feeds were polled in full, without accounting for changes, so the load on sources grew along with the archive volume.
What was changed and what the result was
Collection was split into two circuits: partner RSS feeds and Telegram channels, each with its own polling interval and parsing rules.
The record key became a content hash, so a repeated material does not create a new card but updates the existing one; an unavailable source goes into a retry queue with increasing pause instead of bringing down the issue.
Observability was separated out: delay per source, number of new records, share of rejected ones—everything is visible in a dashboard, and a source failure stopped being a surprise for the newsroom.
After the transition, misses at peaks are zero, the duplicate share is below half a percent, a peak wave is processed in minutes, and emergency backfills have disappeared as a class of work.
How RSS collection differs from parsing source pages
RSS collection is work with a contract: the source itself provides a structured document where the headline, link, date, and text or summary are already marked up for each publication.
The parser reads such a stream in full, does not depend on layout, and deduplicates records by identifier and address. Hence the main property: the channel is fast, stable, and almost requires no maintenance—it breaks far less often than markup parsing.
The completeness of an open feed is not a guarantee but a matter of editorial policy.
Some categories are not included in the stream, a summary arrives instead of full text, tags, authors, and sections are lost, and the history is truncated to the latest records: on first run there is nowhere to collect the archive from. This is exactly where the RSS zone ends and parsing source pages begins.
Page parsing proceeds by crawling: sections, pagination, date-based archive feed, sitemap. Selectors are tied to publication blocks, the date is pulled from markup or the address and normalized, and texts are cleaned of navigation.
Completeness is higher—everything published is visible, including materials outside the feed—but fragility appears: a template change or a switch to loading content with scripts breaks the sample, so rendering, failure monitoring, and reconciliation against a reference are needed.
In practice, comparing collection methods ends with a hybrid.
The feed works as the main channel with a short polling interval, parsing backfills the archive and categories closed to the feed, and the intersection of the two sets shows how many unique materials each path provides and what exactly is lost without crawling.
The choice of collection mode is determined by the presence and completeness of the feed, the required archive depth, publication frequency, and the acceptable delay before a material appears in the system. Next we explain how the polling schedule and deduplication rules grow out of this.
Technical concerns worth resolving before the start
The concerns worth resolving before the start are not about whether the parser will read the feed, but about the boundaries of the sources themselves.
RSS and Atom provide a truncated window—only the latest records; closed sources require authorization and request limits, and one news item arrives from several feeds with different headlines and links.
Below are the questions most often asked by the customer's engineers before the start.
The source provides only the latest records—will old news be lost?
Yes, if the first run goes through a truncated feed. Therefore, at the start we perform an initial archive crawl: pagination by sections, index pages, sitemap.
Then we store the identifier of the last processed record and on each cycle read only what is new—a miss is possible only in a failure wider than the feed window, so we schedule crawling more often than the feed has time to overflow.
How are closed sources connected?
Through standard subscription access: credentials are stored in secure storage, we comply with the request limit per source, and if authorization fails we stop crawling that source and raise an alert—instead of retries that could lead to a block.
How are duplicates controlled?
A news item is compared by canonical link, normalized headline, and text fingerprint in a sliding time window. Matches are merged into one record with a list of sources—the database does not contain "three identical news items," and it is visible who published first.
How often is data updated?
The interval is configured per source: from minutes for news feeds to hours for analytical ones. We request with the condition "has it changed since last time," so unchanged pages are not re-downloaded, and parallel requests are limited at the domain level.
What happens if a source is unavailable?
Retries with increasing pause, then temporary disabling of the source. Failures are isolated: one source going down does not stop the others, accumulated data is read after recovery, and the engineer receives a notification with an error history.
Let's discuss the task and come back with a work plan
To make the estimate accurate rather than a market average, we need a practical slice of the task: what we collect, from where, and how regularly. Below is a list of inputs by which we calculate the scope of work, the storage schema, and plan the launch. The more specific the initial data, the fewer assumptions in the estimate.
- List of sources: feed and section addresses, open APIs, regular pages, sitemaps. Note where authorization is needed and where sources provide data unstably.
- Expected update frequency: once a minute, once an hour, or daily—this determines the polling mode, load, and time margin before a material appears in the system.
- Target fields: headline, publication date and time, author, full text or summary, category, link to the original, images. Specify which fields are mandatory and which may be left empty.
- Processing rules: whether translation, summarization, bringing dates to a single time zone, topic tagging, and removal of duplicates of the same material from different sources are needed.
- Storage requirements: record lifetime, archive depth, whether a history of material changes is needed when the source edits it.
- Access rules: who reads the data and how—internal system, file export, separate interface; what roles and restrictions are needed.
- Acceptance criteria: which sources and fields we consider covered, how we check for misses, and how quickly a source must recover after a failure on its side.
Send these inputs in any form—a list, a table, or links to existing exports. We will review them, clarify disputed points, and come back with an estimate, a work plan, and a launch sequence.
