Imagine launching a news app with millions of daily publications. Without personalization, users see monotonous posts, quickly lose interest, and leave. Chronological feeds lose up to 60% engagement — an experimentally confirmed figure. Our ranker tackles candidate sorting based on hundreds of features, not generating content but prioritizing it. A typical scenario: a news feed becomes irrelevant after two weeks of use — users complain about sameness. The pipeline accounts for interest dynamics and prevents filter bubbles. Hybrid approach: collaborative filtering complemented by semantic content analysis. This captures both explicit and implicit preferences. The result: each user sees a feed that adapts to their evolving interests in real time. This approach boosts retention by 30% and increases session time by 1.5x. Let's dive into implementation details.
How We Build the Ranking Pipeline
Personalized feed is a two-stage pipeline: retrieval and ranking. In the first stage, we select a few hundred candidates from millions of publications via approximate nearest neighbours (ANN) based on subscriptions and interests. In the second stage, we rank these candidates with a heavier model using hundreds of features. Splitting stages is critical: the ranking model is too slow for the full catalog; the retrieval model is insufficiently accurate for the final order.
Features for Ranking
A good ranking model uses three feature groups:
- User context: time of day, day of week, session type (cold or continuation), activity over 24 hours.
- Content characteristics: post age, engagement rate (likes/views), velocity of views in the first hour, author follower count and historical CTR.
- User-content intersection: semantic similarity to interaction history, topical overlap with top interests, familiarity with the author.
# Feature vector for one candidate
@dataclass
class RankingFeatures:
# Content features
post_age_hours: float
engagement_rate_24h: float
viral_velocity: float # views_per_hour in first 2 hours
# User-content interaction
topic_affinity: float # cosine sim between user profile and post embedding
author_ctr_for_user: float # historical CTR of this author for this user
# Context
hour_of_day: int
is_weekend: bool
session_depth: int # how many posts already viewed in the session
Why LightGBM Instead of a Neural Network?
Neural network rankers offer higher quality, but LightGBM with LambdaRank objective is faster in inference (2–5 ms for 200 candidates) and easier to iterate. For medium scale, this is the optimal choice. For audiences over 10 million, a two-level model with a neural network retriever and LightGBM for ranking may be optimal. The cost of developing a neural solution is higher but often economically justified under high load.
import lightgbm as lgb
model = lgb.LGBMRanker(
objective='lambdarank',
metric='ndcg',
ndcg_eval_at=[5, 10, 20],
n_estimators=500,
learning_rate=0.05,
num_leaves=63
)
model.fit(
X_train, y_train, # y — relevance labels: 0=ignored, 1=viewed, 2=liked, 3=shared
group=train_groups, # group sizes per query
eval_set=[(X_val, y_val)],
eval_group=[val_groups]
)
How to Ensure Seamless Scroll on Mobile?
We implement prefetch: when a user scrolls 70% of the current batch, we load the next 20 posts in the background. On Android, we use Paging 3 with prefetchDistance = 5. Example:
// Android: Paging 3 with prefetch for personalized feed
class FeedPagingSource(
private val feedApi: FeedApi,
private val userId: String
) : PagingSource<String, FeedPost>() {
override suspend fun load(params: LoadParams<String>): LoadResult<String, FeedPost> {
return try {
val response = feedApi.getPersonalizedFeed(
userId = userId,
cursor = params.key,
pageSize = params.loadSize
)
LoadResult.Page(
data = response.posts,
prevKey = null,
nextKey = response.nextCursor
)
} catch (e: Exception) {
LoadResult.Error(e)
}
}
}
// ViewModel
val feed = Pager(
config = PagingConfig(pageSize = 20, prefetchDistance = 5),
pagingSourceFactory = { FeedPagingSource(feedApi, userId) }
).flow.cachedIn(viewModelScope)
Scope of Work and Timelines
| Stage | Duration | Result |
|---|---|---|
| Data and signal audit | 1 week | Document on log quality and feature composition |
| Feature pipeline + model training | 2–3 weeks | Baseline LightGBM ranker with NDCG@10 > 0.45 |
| Serving API and mobile client development | 2 weeks | Integrated prefetch and A/B test in production |
| A/B testing and iterations | 2–4 weeks | Measured CTR@10 improvement of 15–30% |
We guarantee on-time delivery of each stage with full documentation and the trained model. Costs are estimated individually based on your stack and data volume.
Typical Pitfalls
First, lack of diversity. Without MPR (Maximum Marginal Relevance), the feed becomes one-sided. Solution: no more than 2 posts from the same author in the first 10, plus topic heuristics. Second, incorrect labels. Using only views biases toward clickbait. Include likes and shares. Third, ignoring cold start. A new user without history — show popular content and quickly collect signals via bandit algorithms. Resource savings come from using ready-made components and open-source libraries.
Ensuring Content Diversity
We apply MPR post-processing: after ranking, we rescore candidates considering similarity. Additionally, we use global heuristics — for example, each subsequent post must differ in topic from the previous one, and the number of posts from one author is limited.
Timeline Estimates
A LightGBM ranker with basic features and API — 2–3 weeks. A full system with two-stage retrieval+ranking, diversity post-processing, and A/B testing — 6–10 weeks.
| Option | Timeline | Includes |
|---|---|---|
| Basic | 2–3 weeks | LightGBM ranker, basic features, API, no diversity |
| Standard | 4–6 weeks | Basic + data audit, MPR diversity, A/B test |
| Full | 6–10 weeks | Standard + two-level pipeline, deep learning retriever, cold start bandit |
Get a consultation — we will do a quick preliminary assessment based on your data. Contact us to discuss details and optimize the budget.