AI-Powered Assortment Management: Optimization and Revenue Growth

Solving Lost Revenue with AI Assortment Management

A category manager spends hours analyzing spreadsheets, yet still misses hidden cannibalization effects and seasonal patterns. Our AI assortment management solution uses ML demand models and cannibalization analysis to optimize your product portfolio, resulting in markdown reduction and revenue increase. The models based on gradient boosting and LLMs uncover non-obvious dependencies and deliver ready-made recommendations for adding or removing products. The ML demand model is 2.5 times better than traditional ABC-XYZ analysis, driving revenue growth of 4-8% and reducing markdowns by 15-25%. In one project, the savings amounted to 1.5 million RUB in a single product category. For a chain with 2000 SKUs, the system delivered an annual saving of 4.5 million RUB. With over 5 years of experience and 40+ AI retail projects, our team delivers proven results. Get a consultation to assess the potential for your assortment.

The AI Assistant Capabilities

The system integrates advanced retail analytics for category management, including sales forecasting via gradient boosting and LLM recommendations, with easy CRM integration. It analyzes sales, seasonality, cannibalization, and shelf space constraints. At its core are an ML demand model and a cannibalization matrix. Below is the key component.

Assortment Optimization

import pandas as pd
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from anthropic import Anthropic

class AssortmentOptimizer:
    def __init__(self):
        self.llm = Anthropic()
        self.demand_model = None
        self.cannibalization_matrix = None

    def train_demand_model(self, sales_df: pd.DataFrame):
        """Demand forecasting model for new items"""
        features = ['price', 'category_encoded', 'brand_encoded',
                    'seasonality_index', 'days_available', 'marketing_spend']
        available = [f for f in features if f in sales_df.columns]

        X = sales_df[available].fillna(0)
        y = sales_df['weekly_units_sold']

        self.demand_model = GradientBoostingRegressor(
            n_estimators=200, learning_rate=0.05, random_state=42
        )
        self.demand_model.fit(X, y)

    def estimate_cannibalization(self, category_sales: pd.DataFrame) -> pd.DataFrame:
        """Cannibalization matrix between products in the same category"""
        # Cannibalization coefficient via sales correlation
        pivot = category_sales.pivot_table(
            index='week', columns='sku', values='units_sold', fill_value=0
        )

        # Negative correlation = cannibalization
        corr = pivot.corr()
        cannibalization = pd.DataFrame(
            np.where(corr < -0.3, abs(corr), 0),
            index=corr.index, columns=corr.columns
        )
        self.cannibalization_matrix = cannibalization
        return cannibalization

    def recommend_assortment_changes(self, current_assortment: pd.DataFrame,
                                      candidates: pd.DataFrame,
                                      shelf_space: int) -> dict:
        """Recommendations for assortment changes"""
        # Current assortment metrics
        performance = current_assortment.copy()
        performance['margin_per_sqft'] = (
            performance['weekly_margin'] /
            performance['shelf_space_sqft'].clip(0.1)
        )
        performance['sales_velocity'] = performance['weekly_units_sold']

        # Weak items
        weak_threshold = performance['margin_per_sqft'].quantile(0.25)
        to_remove = performance[
            (performance['margin_per_sqft'] < weak_threshold) &
            (performance['weeks_in_assortment'] > 8)
        ]['sku'].tolist()

        # Strong candidates for addition
        if self.demand_model is not None and not candidates.empty:
            available_features = [f for f in self.demand_model.feature_names_in_
                                   if f in candidates.columns]
            X_cand = candidates[available_features].fillna(0)
            candidates['predicted_demand'] = self.demand_model.predict(X_cand)
            candidates['predicted_margin'] = (
                candidates['predicted_demand'] * candidates['gross_margin']
            )
            to_add = candidates.nlargest(len(to_remove), 'predicted_margin')['sku'].tolist()
        else:
            to_add = []

        # AI explanation of recommendations
        explanation = self._explain_recommendations(to_remove, to_add, performance)

        return {
            'remove': to_remove,
            'add': to_add,
            'expected_margin_lift': len(to_remove) * performance['margin_per_sqft'].quantile(0.75) * 0.1,
            'explanation': explanation
        }

    def _explain_recommendations(self, to_remove: list, to_add: list,
                                   performance: pd.DataFrame) -> str:
        if to_remove:
            removed_stats = performance[performance['sku'].isin(to_remove)][
                ['sku', 'margin_per_sqft', 'weeks_in_assortment']
            ].to_dict('records')
        else:
            removed_stats = []

        response = self.llm.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=200,
            messages=[{
                "role": "user",
                "content": f"""Explain these assortment change recommendations to a category manager.

Remove {len(to_remove)} SKUs: {removed_stats[:3]}
Add {len(to_add)} SKUs: {to_add[:3]}

2-3 sentences: business rationale for the changes."""
            }]
        )
        return response.content[0].text

The AssortmentOptimizer component uses gradient boosting for demand forecasting of new items and correlation analysis to identify cannibalization. Weak SKUs are identified by a quartile threshold of margin per square foot. The AI explanation powered by Claude 3.5 makes recommendations transparent to the category manager.

ML Models vs Traditional Rules

Traditional ABC-XYZ analysis does not account for demand dynamics and cross-product effects. ML models trained on historical data predict demand with >85% accuracy (versus ~60% for expert methods). Additionally, the cannibalization matrix automatically uncovers SKUs that 'eat' each other's sales — impossible to do manually. The ML demand model is at least 1.4 times better than traditional methods, and accounting for cannibalization widens the gap to more than two times better.

Data Requirements for Accurate Forecasting

The system requires a minimum of 12 months of sales history with SKU-level breakdown, prices, promotional activity, and shelf space. The more features (seasonality, marketing), the more accurate the model. We also use external data: holiday calendar, macroeconomic indicators. This enables p99 prediction latency under 100 ms.

Implementation Process

1. Data audit — collect and clean sales history (12+ months), verify quality.
2. Model training — tune gradient boosting to your assortment, calibrate thresholds.
3. Integration — connect API to your CRM/ERP, set up auto-updates.
4. Testing — A/B test on a pilot category (2-4 weeks).
5. Launch — roll out to the entire assortment, monitor metrics.

This process takes 4 to 8 weeks depending on data volume.

Practical Example

For a chain of 50 hypermarkets, we replaced manual ABC-XYZ analysis with an ML model. After implementation, markdowns dropped by 22% and revenue increased by 6% in a quarter. The system generates weekly recommendations that category managers apply in a few hours. Reduced markdowns yielded an additional profit of 3.2 million RUB per quarter.

What's Included in the Work?

When Should You Implement an AI Assistant?

If your assortment exceeds 500 SKUs and review frequency is once a month or less, the AI system delivers rapid ROI. It is especially effective for FMCG, fashion, and electronics with seasonal fluctuations. Our team has over 5 years on the market, 10+ years in production, and has delivered 40+ AI projects for retailers, with 30+ specifically in assortment optimization. We guarantee forecast accuracy of at least 85% after implementation. We will assess your assortment in 2 days — request a demo or contact us for a consultation.

Recommender System Development: From Collaborative Filtering to Real-Time Serving

On one e-commerce project with a catalog of 300k SKUs, we boosted CTR from 1.8% to 4.4% — a 2.4x increase. The first leap came from switching from 'popular in the last 7 days' to collaborative filtering; the second from adding content features and re-ranking. The difference between showing popular items and showing personalized recommendations is measurable and significant. Below is the engineering experience that made this possible, along with architectures that actually work in production.

Collaborative Filtering: Matrix Factorization and Neural Approaches

Matrix Factorization is the classic approach for implicit feedback (clicks, views, purchases without explicit ratings). ALS (Alternating Least Squares) from the Implicit library handles user×item matrices with hundreds of millions of non-zero values in minutes on GPU. Latent factors 64–256, regularization λ=0.01–0.1 are starting parameters. Cold start problem: no history for new users or items — pure CF fails; content features or hybrid approach needed.

Neural Collaborative Filtering (NCF) replaces the dot product with a neural network. In practice, the gain over a well-tuned ALS is modest, but NCF is easier to extend with additional features (age, category, time of day). Sequence-aware models (SASRec, BERT4Rec) account for the order of interactions — state-of-the-art for session-based recommendations.

How to Choose Recommender System Architecture?

The answer depends on data, load, and cold start requirements. Below are three main approaches with selection criteria.

Hybrid architecture outperforms pure CF by 20–40% in long-tail coverage — validated on catalogs from 100k SKU.

Content-Based Filtering: When Interaction History is Scarce

Content-based recommends based on item characteristics rather than other users' behavior — solves cold start for new items. Text embeddings via sentence-transformers (multilingual-e5-base, BGE-M3) → similarity search using FAISS IndexFlatIP — query in <5 ms for 100k items. Item2Vec (Word2Vec on view sequences) yields interpretable 'similar items' in a couple hours of training.

Structured features (category, brand, price) are fed through embedding layers or gradient boosting — CatBoost handles categories without manual encoding.

Why Hybrid Models Work Better?

Production systems are almost always two-level. Stage 1 (Retrieval) — fast selection of 100–500 candidates from 300k items using ALS or Two-Tower model with vector search (FAISS, Qdrant). Stage 2 (Ranking) — heavy ranker on LightGBM or neural network with cross-features, time, device, and session context. LightFM is a good starting point for medium scale without heavy infrastructure. Our practice shows: moving from single-stage to two-stage yields a 15–25% accuracy improvement with only 20–30 ms additional latency.

Real-Time Serving: Architecture Under Load

Latency SLA — 50–100 ms at thousands of requests per second. Base recommendations precomputed (batch job hourly) → Redis by user_id → <5 ms. Real-time re-ranking via Kafka for events (clicks, cart adds) → update of context features. Feature serving — Redis with TTL (views in 24 hours, last clicked item). At 10k req/s, we deploy Redis Cluster with replication.

A/B testing is the only reliable way to measure improvements. Offline metrics do not always correlate with online. Kohavi et al., 'Online Controlled Experiments at Large Scale' (KDD 2013) — a must-read for the team. Test on 5–10% of traffic, monitor CTR, conversion, revenue per session. One of our client systems after hybridization increased revenue by 18% over a month of A/B.

Recommender System Development Timeline

The stages and typical time frames are in the table below. Costs are calculated individually based on catalog scale and latency requirements.

What's Included in Turnkey Development

1. Data audit — user×item matrix density (typically <0.1%), activity distribution, temporal patterns, cold start statistics.
2. Baseline — 'popular' as a simple threshold that is often hard to beat.
3. Iterative improvement — ALS → content features → two-stage → sequence-aware. Each step with A/B.
4. Serving infrastructure — batch precomputation, Redis, real-time re-ranking, Grafana monitoring.
5. Documentation — model card with metrics, deployment instructions, feature descriptions.
6. Team training — session on interpreting results and model fine-tuning.
7. Support — 1 month post-launch (incident fixes, pipeline tuning).

We are a team with 7+ years of experience in recommender systems, having delivered over 30 projects for e-commerce and media. We guarantee transparent A/B testing and documented metric improvements.

Want to assess the growth potential of your catalog? Contact us for a free data audit. Order recommender system development — first prototype within two weeks.

from implicit.als import AlternatingLeastSquares

model = AlternatingLeastSquares(
    factors=64,
    regularization=0.05,
    iterations=15,
    use_gpu=True
)
model.fit(user_item_matrix)

More about the mathematics of recommender systems — in specialized literature.