News digest generation is held back by manual curation
News digest generation is held back by manual news selection: an editor keeps dozens of tabs and aggregator tabs open, manually checks headlines, and decides what will make it into the morning or evening digest.
This is not creative work but dispatching: it eats up editorial time and does not scale — as the number of sources grows, publishing does not speed up but simply requires more people.
Duplicates in the feed break this scheme first.
The same news item arrives from eight to twelve reprints, each with its own headline, lede, and truncated context, and source reconciliation turns into a manual check of “has this already been covered or is it new.” At the same time, a second class of errors accumulates — omissions: while the editor sorts through duplicates, a fresh topic goes unnoticed until the next issue.
Publication delay here is not a consequence of slow internet but a direct product of feed size times manual operations. Hours pass from the event to the release of the digest, and by the time it is sent, the first wave of audience attention has already passed.
Thematic selections are assembled unevenly: one rubric gets three news items, another gets none, even though there were just as many events there.
Then editorial overload kicks in. People switch to routine selection, analytics and original materials are postponed, and the quality of the digest begins to depend on who is on shift today.
According to estimates from Reuters Institute for the Study of Journalism, a significant share of editorial time goes specifically to mechanical operations with the feed, not to work with content.
We break down how this is addressed: automatic source reconciliation, deduplication by entities and topics, digest templates for rubrics, and control of selection evenness before publication.
What changes after switching to automatic digest assembly
The digest stops being a manual operation that someone manages to assemble by morning. The system monitors sources continuously, and the release is tied to the appearance of significant publications, not to an editor’s free window. Below is what exactly changes in measurable terms.
Release speed and source coverage
The digest is assembled within minutes after the original source publishes: the system determines significance itself, groups materials by topic, and removes repetitions.
One editor maintains dozens of channels where three or four were reviewed manually — coverage grows without expanding the team. This removes the main bottleneck of manual assembly: while the digest is being prepared, some news items have time to become outdated.
Unified presentation standard and tone
Structure, item length, terminology, and tone are set by rules once and applied to every issue — style does not “float” between authors and shifts. Each line retains a source link and date, so verification takes less time than writing from scratch.
An entire class of errors goes away: duplicates within an issue, lost links, discrepancies in names and dates, accidental tone changes.
Industry practice confirms this: newsrooms that moved routine digests to automatic assembly recorded a reduction in the share of manual edits to single-digit percentages, leaving meaning checks and sensitive topics to humans.
This transition was described, in particular, by Associated Press using the example of financial reports.
Digest preparation modes: batch, streaming, on demand
All three modes rely on one layer of normalization, deduplication, and ranking — only the launch trigger and handling of the freshness window differ.
Therefore, choosing an operating mode is mostly configuration: assembly schedule, aggregation window width, rerun rules, and backfill depth.
| Variant | When suitable | Limitations |
|---|---|---|
| Scheduled batch processing | Regular morning digests, large recipient lists, reproducibility of the issue | Delay equals the schedule window; peak load on sources at launch time |
| Streaming processing | Monitoring urgent topics, minute-level latency matters | More resource-intensive, more complex deduplication and event ordering |
| On-demand processing | Personal digest at the moment of opening, one-off selections | Depends on source availability; without a cache, hits request limits |
Scheduled batch assembly
The run follows a fixed window: the system collects materials for the period, collapses duplicates by publication key, and assembles the issue in full. A repeated run for the same period is idempotent — no second copies appear, and a missed interval is covered by backfill.
Streaming processing and on-demand mode
The stream updates the digest as materials arrive, so latency is measured in minutes, not hours. On-demand mode assembles a digest for a specific recipient at the moment of request, relying on a cache with a limited time to live and already calculated ranks.
In practice, a hybrid is more often chosen: batch covers baseline issues, while streaming and requests cover urgent topics.
How we implement the system: stages of engineering work
The timelines and content of the stages depend on the number of sources and the strictness of requirements for the digest format. Below is the sequence that every project follows: from feed inventory to handing the system over for maintenance.
Source audit and requirements markup
First, we record where the data comes from and what is considered a correct result. Requirements for digest volume, tone, and structure are marked up immediately — the same wording later becomes checks in the control set.
- Source audit. We compile a list of feeds and databases, check access, update frequency, duplicates, and broken channels; filter out everything that does not provide a stable stream.
- Pipeline diagram. We design the path of a material: collection, normalization, duplicate filtering, selection, digest assembly, delivery to the customer’s channel.
- Selection rules setup. We define thematic and priority rules, relevance thresholds, volume limits, and release schedule.
Quality check and acceptance
- Quality check on a control sample. We run manually labeled issues, compare completeness and accuracy of selection, calculate the share of false positives, and adjust rules to acceptable values.
- Production launch. We switch the system to operating mode, enable monitoring of collection failures and delivery delays, and run parallel output for comparison with the manual version.
- Handover for maintenance. We provide documentation on rules and settings, train the customer’s team, and appoint those responsible for editing dictionaries and priorities.
What remains with you: configurations, diagrams, regulations
A digest is not a single request to a model but a pipeline: sources, filtering, generation, verification, delivery.
We describe each node with configuration, not “by agreement within the team,” so the system can be developed and handed over between contractors without the original author.
Pipeline diagrams and filtering rules
- Data flow diagram: the path of a publication from collection from sources to delivery of the finished digest, with control points and places for manual intervention at each transition.
- System configurations: list of sources and their priorities, rubricator, model request templates, summarization parameters, and digest volume limits.
- Filtering rules: what is filtered out as a duplicate, what goes to quarantine, which topics are not published without editor confirmation.
- Access policies: editor, analyst, and administrator roles, data visibility boundaries, and the procedure for granting and revoking access.
Documentation and maintenance regulations
- Maintenance regulations: procedure for updating sources and templates, response to a node failure, person responsible and incident resolution time.
- Instructions for the editorial team: step-by-step scenarios for editing a draft, removing material from publication, and setting up personalization for a reader segment.
- Change log and test run reports on the reference selection — it is visible how rule edits affect digest completeness and accuracy.
All of this remains in your pipeline: using the documentation, a new engineer deploys the system from scratch, and the editorial team works with the digest without a developer.
Case study: reduced issue preparation from six hours to twenty minutes
A media center with a daily industry digest kept the issue on two editors: morning source review, manual duplicate proofreading, text assembly, and newsletter layout.
The feed was closed by midday, some news had time to become outdated by then, and if one editor left, the issue stalled. The task was formulated not as “speed up text writing” but as assembling a pipeline that prepares a draft issue before the start of the workday.
What it was before and what changed
| Metric | Before | After |
|---|---|---|
| Issue preparation time | 5-6 hours of manual collection and layout | 20 minutes to review the draft |
| Share of manual edits | 70-80% of text was rewritten | 15-20%, targeted wording edits |
| Source coverage completeness | about 40 publications, manual review | the entire approved list, continuously |
| Repetitions and duplicates | found during proofreading | filtered out before entering the feed |
| Response to breaking news | 2-3 hours before insertion into the issue | edit takes minutes |
How it works inside
The pipeline is divided into stages: collection from feeds and email newsletters on schedule, bringing materials to a unified format with linkage to entities — companies, persons, topics, then filtering duplicates by entity match and publication proximity.
Then materials are ranked by significance for the specific audience of the issue, and the digest is assembled according to the rubric template defined by the editor in advance.
The editor works not with a blank page but with a draft: the original source is visible for each statement, and the edits made go into the rule set and are taken into account in the next issue. Therefore, the share of intervention decreases from issue to issue. A full breakdown of the solutions is in the project publication.
What to do if you already have your own text processing system deployed
An add-on on top of your own pipeline is almost always possible — the question is not whether it can be done, but what transfers without losses and what needs to be rebuilt.
The boundary is not “in-house or ready-made” but by layers: sources, normalization, selection of what matters, issue assembly.
Preserving existing work is a real asset, not a polite phrase.
The list of sources with weights and polling frequency, glossaries of entities and abbreviations, tone and prohibited wording rules, rubricator, integrations with distribution channels, and issue formats transfer into the new layer as is — they do not need to be relabeled; selection is configured on them.
What will have to be rebuilt is what relied on handwritten heuristics: deduplication of similar publications, linking the same news item from different languages, and significance assessment based on history.
Thresholds in code are replaced by a set of examples and editorial feedback, so quality grows from issue to issue rather than remaining fixed once and for all.
We build phased replacement without stopping the issue. First, shadow comparison on a live stream: news digest generation runs in parallel, discrepancies are analyzed by rubric and source, and the previous system still delivers the issue.
Then we switch part of the rubrics or channels, compare selection completeness and accuracy, and only after stable matching do we move the rest — with a fast return to the previous route.
Compatibility is ensured by the fact that exchange goes through existing points of your pipeline: queues, export files, exchange tables. We do not break what works; we integrate alongside it.
Technical doubts before starting a project
Before an estimate, the same set of questions usually comes up: how noise is filtered out, whether facts in the finished digest can be trusted, what to do when source formats change, how to limit the topic scope, and where data is stored. Below are short answers to each.
How is irrelevant noise filtered out?
A material is assessed by a combination of features: source, rubric, density of thematic matches, duplicates. The cutoff threshold is configured separately for each channel, and discarded items are saved as a list — the rules are visible, and they can be adjusted.
How much can facts in the digest be trusted?
Each statement is tied to a source fragment. Numbers, names, and dates are verified against the original source; anything unconfirmed is either marked as unverified or does not make it into the issue.
What happens when a source format changes?
Parsing is moved to a separate layer: a feed, newsletter, file, or export is brought to a common structure. A new format is connected by configuration, and the digest logic itself is not rewritten.
How to limit the topic scope of issues?
The topic scope is set by a list of topics and stop rules with priorities. The limitation applies before the issue is formed — only what has passed the topic filter gets into the digest.
Where is data stored and for how long?
Source materials, intermediate selections, and finished issues are stored. Retention periods and access levels are configured according to your regulations; export and deletion are performed as standard.
Let’s discuss the task and come back with an estimate and work plan
To ensure the estimate is based on facts rather than assumptions, we start with the composition of the source data.
Describe where the materials come from and in what form you want to see the result — this is enough to outline the architecture for news digest generation and give timelines.
The more concrete the inputs, the fewer assumptions in the plan and the more accurate the first stage of work.
What to send and what we will review at the start:
- list of sources: own feeds, industry media, internal knowledge bases, partner newsletters;
- formats and frequency of arrival: feeds, exports, files, emails — hourly, daily, event-driven;
- selection rules: topics, regions, persons, language, what is considered a duplicate and what is filtered out immediately;
- issue requirements: digest length, tone, links to the original source, whether editor proofreading is needed;
- access: test keys to sources and a staging environment where we deploy the pipeline;
- acceptance criteria: share of relevant materials, number of edits per issue, delay from publication to finished digest.
From this, the first-stage plan takes shape: analysis of data and its quality, a prototype on a real sample from the past period, measurement of metrics on it, and a pipeline diagram with control points.
This way, the customer sees exactly where generation errs and how it is fixed, even before production launch.
Describe the task and sources in free form — we will come back with an estimate and work plan.
