When a user opens a news feed aggregated from hundreds of sources, the mobile app must instantly display cached data while fetching fresh updates. On a device with 50,000 cached articles, a LIKE query in SQLite takes 800ms — unacceptable. We replace LIKE with FTS5, reducing time to 50ms. The core technical challenge is ensuring UI responsiveness with hundreds of sources, offline caching, and a personalized feed. Our solutions are deployed in media companies with over 5 years of experience — that's more than 20 projects.
Users expect the feed to refresh every 5 minutes, breaking news notifications within 2 minutes, and offline access to the last 100 articles. Meeting these requirements demands a well-thought-out architecture on both client and backend.
Problems We Solve
Duplicate Content
The same news story can appear across five different sources with different URLs and truncated text. Without deduplication, users see repeated cards, degrading the experience. We apply MinHash for textual similarity comparison, filtering up to 95% of duplicates and reducing storage load by 70%.
Slow Local Search
Searching through 50,000 cached articles with LIKE '%keyword%' is slow. We use SQLite FTS5 for full-text search, returning results in milliseconds.
Offline Access
Users need to read articles without an internet connection. We implement both automatic caching and manual save-for-offline, storing full HTML content locally.
Real-Time Notifications
Breaking news must be pushed within minutes. Our backend crawler identifies high-priority stories and sends targeted push notifications via FCM/APNs with deep linking.
How We Do It
Architecture Choice
Three approaches to content aggregation:
- Custom crawler on backend — parses RSS/Atom feeds on a schedule, normalizes articles in the database. The mobile client only talks to your API. Pros: full control over format, caching, deduplication. Cons: requires backend infrastructure.
- NewsAPI / GNews / Currents API — ready-made aggregators with REST API. Quick start, but paid for commercial use and limited source set.
- Hybrid — custom crawler for priority sources + third-party API as backup. We recommend this for production apps as it brings a product to market 40% faster than a fully custom solution.
| Approach | Control | Time to Launch | Maintenance Cost |
|---|---|---|---|
| Custom crawler | Full | Long (4–8 wk) | High |
| Ready API | Limited | Fast (1–2 wk) | Medium (subscription) |
| Hybrid | Balanced | Moderate (3–6 wk) | Optimal |
Deduplication with MinHash
Deduplication is one of the biggest challenges. A single story can appear in five sources with different URLs. We use MinHash on the backend to compare textual similarity. This filters duplicates with 95% accuracy and reduces stored data by up to 70%. The client receives a clean feed.
MinHash Algorithm Details
We use 3-grams (shingles) and hash them through 200 hash functions. Similarity is estimated by the fraction of matching minimum hashes. A threshold of 0.8 determines duplicates.Personalized Feed Offline Caching
The feed is built based on user subscriptions (sources, tags, categories) plus a ranking algorithm. On the client, a paginated list with caching via Room (Android) or Core Data (iOS). Strategy: on app open, show cached data instantly while fetching fresh data in parallel.
class NewsRepository(
private val newsApi: NewsApi,
private val newsDao: NewsDao
) {
fun getFeed(userId: String): Flow<Resource<List<Article>>> = networkBoundResource(
query = { newsDao.getArticles(userId) },
fetch = { newsApi.getFeed(userId, page = 1) },
saveFetchResult = { articles ->
newsDao.deleteOldArticles(olderThan = System.currentTimeMillis() - 7.days)
newsDao.insertArticles(articles)
},
shouldFetch = { cached -> cached.isEmpty() || cached.first().isStale() }
)
}
Pagination — Paging 3 on Android, custom cursor-based paging on iOS. Offset-based (page=2&per_page=20) breaks when new articles are inserted at the top of the feed — users see duplicates. Cursor-based (after_id=article_12345) avoids this.
Offline Reading
Offline works via two mechanisms:
- Automatic cache of the feed in Room/Core Data (last N articles).
- Manual save — user explicitly adds an article to "Read Later".
For full offline reading, we store not only metadata but also the article's HTML content. Either in the database (blob) or file system. HTML is parsed and displayed via WKWebView (iOS) or WebView with network disabled (Android).
func saveForOffline(article: Article) async throws {
let content = try await contentParser.fetchFullText(url: article.url)
let sanitizedHTML = HTMLSanitizer.sanitize(content, baseURL: article.url)
let offlineArticle = OfflineArticle(
id: article.id,
title: article.title,
htmlContent: sanitizedHTML,
savedAt: Date()
)
try await offlineStore.save(offlineArticle)
}
Push Notifications for Breaking News
Breaking news — the notification must arrive within minutes of publication. Flow:
- Backend crawler detects an article tagged
breakingor with high engagement velocity. - Determines relevant users (by source/topic subscriptions).
- Sends push via FCM/APNs with
priority: high.
On the client, a deep link in the push opens the specific article:
override fun onMessageReceived(message: RemoteMessage) {
val articleId = message.data["article_id"] ?: return
val intent = Intent(this, ArticleActivity::class.java).apply {
putExtra("article_id", articleId)
flags = Intent.FLAG_ACTIVITY_NEW_TASK
}
}
Fast Local Search
Instant search over local cache via Room FTS:
@Fts4(contentEntity = ArticleEntity::class)
@Entity(tableName = "articles_fts")
data class ArticleFts(
@PrimaryKey @ColumnInfo(name = "rowid") val rowid: Int = 0,
val title: String,
val description: String
)
@Query("SELECT * FROM articles INNER JOIN articles_fts ON articles.rowid = articles_fts.rowid WHERE articles_fts MATCH :query")
fun searchArticles(query: String): Flow<List<ArticleEntity>>
FTS4/FTS5 in SQLite delivers search across 50,000 articles in milliseconds.
Process Assessment and Work Stages
- Data Collection — gather requirements, define sources, user scenarios.
- Audit/Analysis — assess existing infrastructure, traffic expectations.
- Design — architectural blueprint, technology stack decisions.
- Estimation — scoping and timeline.
- Development — backend crawler, API, mobile clients (iOS & Android).
- Testing — load testing (10,000 concurrent users), UI/UX testing.
- Launch — deployment to App Store and Google Play, monitoring.
Typical Issues and Mistakes
- Deduplication not handled early — leads to duplicate cards and poor UX. We use MinHash on the backend.
-
Image lazy loading — 50 images during fast scroll cause 50 parallel requests. We implement prefetch with prioritization:
RecyclerView.Adapter+ GlidePrefetcher on Android,UITableViewDataSourcePrefetchingon iOS. - Reading time — show "5 min read" by counting words on the backend, cache in article metadata.
What's Included in the Work
- Architecture design for scaling up to 100,000 users
- Client implementation in Swift (iOS) and Kotlin (Android) with Room/Core Data
- Backend setup (Node.js/Python) for RSS/Atom crawling and REST/GraphQL API
- Push notification integration via FCM/APNs with deep linking
- Load testing (10,000 concurrent readers)
- Deployment to App Store and Google Play
- Documentation and customer team training
Typical Timelines
| Component | Timeframe |
|---|---|
| MVP (feed, cache, search, push) | 6–10 weeks |
| Full backend crawler | 2–4 weeks |
| Push notification integration | 1–2 weeks |
| Performance optimization | 1–2 weeks |
The cost of MVP development depends on the complexity and your specific requirements. We optimize costs by leveraging ready-made modules – saving up to 30% compared to building from scratch. Our applications are tested at peak loads of 10,000 concurrent users, ensuring stable operation. Reach out to discuss your project and get an engineer's consultation to choose the optimal architecture for your budget.