Training Recommendation Models: Two-Tower, ALS, Negative Sampling

We develop custom recommendation models that work in real conditions, not on overfitted metrics. Our training recommendation models use Two-Tower, matrix factorization, negative sampling, and temporal split. A typical case: an e-commerce client used random split, got NDCG@10 = 0.65, but after deployment the model performed at 0.4. After switching to temporal split, offline metrics dropped, but the A/B test showed a +5% revenue lift. The client noted: "Random split gave inflated metrics — temporal split turned out to be the only honest one." Our stack: PyTorch, Hugging Face Transformers, Faiss for ANN search, ONNX Runtime for inference. The team has 5+ years in ML and 50+ deployed recommender systems in e-commerce, media, and fintech. We use Weights & Biases for experiment tracking and MLflow for model management. Order a preliminary data analysis — we will evaluate your task and propose the optimal solution.

Problems we solve

Incorrect negative example generation

If you take uniform sampling from all items, the model won't learn to distinguish popular from relevant. We use popularity-based and hard negative mining (e.g., items the model already ranks high incorrectly). Our Two-Tower model outperforms ALS by 1.2x in NDCG@10, and hard negative sampling outperforms uniform by 1.3x in Recall@20.

Cold start for new users and items

Without content features, the model outputs zero predictions. We add side information (category, text description, images) via embeddings.

Temporal bias

Real scenarios are sequences of actions. If you shuffle data randomly, the model "sees" the future during training. Temporal split with time-based validation/test sets is mandatory.

How we do it: a detailed case

Among our projects is an online store with 5M interactions and 200K items. We built a Two-Tower model with HNSW index for retrieval. The pipeline included:

• log preprocessing: deduplication, bot filtering, interaction weighting (purchase > view);
• negative sampling ratio 4:1 with popularity weighting; temporal split: 28 days train, 7 val, 7 test;
• Two-Tower: user tower — embedding layer + Dense(256), item tower — same + L2 normalization. Loss — Weighted BCE with log-weight for positive examples;
• training 20 epochs with early stopping on NDCG@10, using AdamW (LR=1e-3). On GPU A100 — 45 minutes;
• deployment via Triton Inference Server with ONNX model. p99 latency — 12 ms.

Result: NDCG@10 lift of 22% over ALS baseline, online A/B test showed +8% in cart additions.

Why proper negative sampling matters

Negative examples define the decision boundary. If all negatives are random tail items, the model easily distinguishes them. But in practice, you need to discern nearly relevant items — e.g., a dark sofa vs a black sofa. Hard negative sampling (from top-100 non-interacted items) yields more robust training. We implement this via an embedding cache and online sampling.

What temporal splitting provides

Evaluating a model on a temporal split is the only way to get honest metrics. Random split can "peek" into the future, inflating results. NDCG@10 on temporal split correlates with online A/B test results: deviation <15%.

Model architecture comparison

Compared to ALS, Two-Tower gives a 1.2x boost in NDCG@10 with comparable inference time.

Negative sampling strategy comparison

In practice, we combine popularity-based and hard negatives: 80% popular, 20% hard negatives. This balances quality and speed.

Timelines and cost

Timelines: from 2 weeks (baseline + modifications) to 2 months (full pipeline with custom architecture). Cost is individual after data analysis. Our clients report up to 40% savings on infrastructure and 30% reduction in maintenance costs due to pipeline optimization. Typical investment ranges from $15,000 to $50,000.

What's included in the work

• Prepared dataset for retraining
• Baseline (ALS or Two-Tower) with metrics report
• Trained final model (PyTorch/ONNX)
• Documentation on pipeline reproduction
• Pipeline code (preprocessing, training, inference)
• Guide on fine-tuning with new data
• Training session for your team
• 30 days of post-deployment support
• Consultation on NDCG and Temporal Split tools

Get a specialist consultation — we will discuss your project and propose the optimal solution.

Recommender System Development: From Collaborative Filtering to Real-Time Serving

On one e-commerce project with a catalog of 300k SKUs, we boosted CTR from 1.8% to 4.4% — a 2.4x increase. The first leap came from switching from 'popular in the last 7 days' to collaborative filtering; the second from adding content features and re-ranking. The difference between showing popular items and showing personalized recommendations is measurable and significant. Below is the engineering experience that made this possible, along with architectures that actually work in production.

Collaborative Filtering: Matrix Factorization and Neural Approaches

Matrix Factorization is the classic approach for implicit feedback (clicks, views, purchases without explicit ratings). ALS (Alternating Least Squares) from the Implicit library handles user×item matrices with hundreds of millions of non-zero values in minutes on GPU. Latent factors 64–256, regularization λ=0.01–0.1 are starting parameters. Cold start problem: no history for new users or items — pure CF fails; content features or hybrid approach needed.

Neural Collaborative Filtering (NCF) replaces the dot product with a neural network. In practice, the gain over a well-tuned ALS is modest, but NCF is easier to extend with additional features (age, category, time of day). Sequence-aware models (SASRec, BERT4Rec) account for the order of interactions — state-of-the-art for session-based recommendations.

How to Choose Recommender System Architecture?

The answer depends on data, load, and cold start requirements. Below are three main approaches with selection criteria.

Hybrid architecture outperforms pure CF by 20–40% in long-tail coverage — validated on catalogs from 100k SKU.

Content-Based Filtering: When Interaction History is Scarce

Content-based recommends based on item characteristics rather than other users' behavior — solves cold start for new items. Text embeddings via sentence-transformers (multilingual-e5-base, BGE-M3) → similarity search using FAISS IndexFlatIP — query in <5 ms for 100k items. Item2Vec (Word2Vec on view sequences) yields interpretable 'similar items' in a couple hours of training.

Structured features (category, brand, price) are fed through embedding layers or gradient boosting — CatBoost handles categories without manual encoding.

Why Hybrid Models Work Better?

Production systems are almost always two-level. Stage 1 (Retrieval) — fast selection of 100–500 candidates from 300k items using ALS or Two-Tower model with vector search (FAISS, Qdrant). Stage 2 (Ranking) — heavy ranker on LightGBM or neural network with cross-features, time, device, and session context. LightFM is a good starting point for medium scale without heavy infrastructure. Our practice shows: moving from single-stage to two-stage yields a 15–25% accuracy improvement with only 20–30 ms additional latency.

Real-Time Serving: Architecture Under Load

Latency SLA — 50–100 ms at thousands of requests per second. Base recommendations precomputed (batch job hourly) → Redis by user_id → <5 ms. Real-time re-ranking via Kafka for events (clicks, cart adds) → update of context features. Feature serving — Redis with TTL (views in 24 hours, last clicked item). At 10k req/s, we deploy Redis Cluster with replication.

A/B testing is the only reliable way to measure improvements. Offline metrics do not always correlate with online. Kohavi et al., 'Online Controlled Experiments at Large Scale' (KDD 2013) — a must-read for the team. Test on 5–10% of traffic, monitor CTR, conversion, revenue per session. One of our client systems after hybridization increased revenue by 18% over a month of A/B.

Recommender System Development Timeline

The stages and typical time frames are in the table below. Costs are calculated individually based on catalog scale and latency requirements.

What's Included in Turnkey Development

1. Data audit — user×item matrix density (typically <0.1%), activity distribution, temporal patterns, cold start statistics.
2. Baseline — 'popular' as a simple threshold that is often hard to beat.
3. Iterative improvement — ALS → content features → two-stage → sequence-aware. Each step with A/B.
4. Serving infrastructure — batch precomputation, Redis, real-time re-ranking, Grafana monitoring.
5. Documentation — model card with metrics, deployment instructions, feature descriptions.
6. Team training — session on interpreting results and model fine-tuning.
7. Support — 1 month post-launch (incident fixes, pipeline tuning).

We are a team with 7+ years of experience in recommender systems, having delivered over 30 projects for e-commerce and media. We guarantee transparent A/B testing and documented metric improvements.

Want to assess the growth potential of your catalog? Contact us for a free data audit. Order recommender system development — first prototype within two weeks.

from implicit.als import AlternatingLeastSquares

model = AlternatingLeastSquares(
    factors=64,
    regularization=0.05,
    iterations=15,
    use_gpu=True
)
model.fit(user_item_matrix)

More about the mathematics of recommender systems — in specialized literature.