The crypto market reacts to news faster than tradfi: from the publication of an article about regulatory actions to price movement — sometimes seconds. For trading systems, risk monitoring, or sentiment analysis, you need a structured news stream with minimal latency. We are blockchain engineers with extensive production experience — we take on organizing such a stream turnkey. Over our work, we have implemented 7 aggregators for funds and traders, reducing the average delay to 3 seconds.
Why latency is critical
A 30-second delay can cost thousands of dollars in impulse strategy arbitrage. One of our clients lost about $500 on a single trade due to outdated data — after switching to our pipeline, latency dropped from 40 to 2 seconds. News sources vary in speed: CoinDesk RSS feeds update every 5–10 minutes, the CryptoPanic API is real-time, and HTML parsing adds another 10–30 seconds. We design the pipeline so that news reaches your system with minimal latency and guaranteed delivery.
Which sources to use and how?
Comparison of three approaches:
| Method | Speed | Reliability | Complexity | Examples |
|---|---|---|---|---|
| RSS/Atom feeds | Medium | High | Low | CoinDesk, CoinTelegraph, The Block |
| Official API | High | High | Medium | CryptoPanic, Messari, Santiment |
| HTML parsing | Low | Low | High | Blockworks, exchange news |
RSS/Atom feeds (most reliable)
CoinDesk, Cointelegraph, The Block, Decrypt — all have RSS. This is an official, stable channel:
import Parser from "rss-parser" const parser = new Parser({ customFields: { item: [["media:content", "media", { keepArray: false }]], }, }) const feeds: Record<string, string> = { coindesk: "https://www.coindesk.com/arc/outboundfeeds/rss/", cointelegraph: "https://cointelegraph.com/rss", theblock: "https://www.theblock.co/rss.xml", decrypt: "https://decrypt.co/feed", } async function fetchFeed(source: string, url: string): Promise<NewsItem[]> { const feed = await parser.parseURL(url) return feed.items.map((item) => ({ source, title: item.title ?? "", url: item.link ?? "", publishedAt: new Date(item.pubDate ?? ""), summary: item.contentSnippet ?? "", guid: item.guid ?? item.link ?? "", })) } Polling every 5 minutes — a reasonable balance between freshness and load on the source. Deduplication by guid.
Official APIs
CryptoPanic API — news aggregator with sentiment scoring:
GET https://cryptopanic.com/api/v1/posts/?auth_token={key}¤cies=BTC,ETH&kind=news Returns structured data with bullish/bearish community votes. Messari API — quality news with asset tags:
GET https://data.messari.io/api/v1/news?page=1&limit=50 Santiment — news + on-chain data + social metrics in one API.
HTML parsing (when no API)
For sources without RSS — cheerio (Node.js) or BeautifulSoup (Python). Fragile approach: any markup change breaks the parser. For critical sources — monitor parsing success and alert on extraction rate drop below 95%.
import * as cheerio from "cheerio" async function scrapeBlockworks(html: string): Promise<NewsItem[]> { const $ = cheerio.load(html) return $("article.post-card").map((_, el) => ({ title: $(el).find("h2.post-title").text().trim(), url: $(el).find("a").attr("href") ?? "", publishedAt: new Date($(el).find("time").attr("datetime") ?? ""), summary: $(el).find("p.excerpt").text().trim(), })).get() } How to effectively deduplicate news?
Deduplication is critical — the same news may appear in multiple sources. We use a two-level approach: first exact match by external_id (GUID from RSS), then fuzzy match by title (cosine similarity with a threshold of 0.85). This eliminates 99% of duplicates. We use PostgreSQL for storage:
CREATE TABLE news_items ( id BIGSERIAL PRIMARY KEY, source VARCHAR(50) NOT NULL, external_id VARCHAR(255) NOT NULL, title TEXT NOT NULL, url TEXT NOT NULL, published_at TIMESTAMPTZ NOT NULL, summary TEXT, raw_content TEXT, tags TEXT[], UNIQUE (source, external_id) ); CREATE INDEX idx_news_published ON news_items (published_at DESC); CREATE INDEX idx_news_tags ON news_items USING GIN (tags); Asset tagging — determine which crypto assets are mentioned in the news by a list of tickers and names. A simple regex-based approach gives 80–90% accuracy for major assets. For complex cases, we use an NLP model with 95% accuracy.
What matters in production
Monitor freshness: if the latest news from a source is older than 30 minutes — alert. RSS can stall without an explicit error. Extraction rate — percentage of successfully parsed elements: a drop below 90% is a critical alert. Respect robots.txt and rate limiting: avoid generating excessive load. Jitter between requests (uneven intervals). Set a proper User-Agent. Some sources block headless browser user agents.
How to set up news parsing: step-by-step plan
- Source analysis and selection of suitable methods (RSS, API, HTML).
- Data schema and pipeline design.
- Module implementation with unit tests.
- Deduplication and asset tagging setup.
- Deployment in Docker/Kubernetes with monitoring.
- Documentation and integration with the client's system.
Comparison of sentiment analysis tools
| Tool | Type | Accuracy | Speed |
|---|---|---|---|
| CryptoPanic voting | community-based | 70–80% | real-time |
| Santiment social trends | on-chain + social | 85–90% | 5–10 min |
| Vader / TextBlob | lexicon-based | 75–85% | milliseconds |
| FinBERT | NLP model | 90–95% | seconds |
What's included
- Architecture design for data collection (data schema, source selection, load balancing)
- Implementation of parsing modules (RSS, API, HTML) with unit and integration tests
- Deduplication and asset tagging setup
- Deployment in Docker/Kubernetes with monitoring and alerting
- API documentation for data access and instructions for adding new sources
- Post-launch support: we fix parsers within 24 hours if source markup changes
Realistic timeline for an aggregator of 10–15 sources via API + storage: 2–3 weeks. Get a consultation — we will assess your project for free. Order an assessment — we will select the optimal architecture for your budget and requirements.