AI-Powered Market Basket Analysis for Retail Chains

A retail chain of 500 stores was losing 12% of revenue due to the lack of personalized recommendations. Items in the basket didn't cross-sell, promotions didn't leverage synergies, and the analytics department spent weeks on Excel reports. We built an AI basket analysis system that replaced manual work and increased cross-sales by 18% in the first quarter. The key idea is a hybrid of classic association rules and neural networks, providing interpretability for core rules and depth for long-tail.

ML solutions in retail are becoming the standard. Basket segmentation by customer type is another personalization layer we implement through transaction clustering.

What problems does Market Basket Analysis solve?

Data sparsity. In a typical dataset of 1 million transactions and 10,000 SKUs, most item pairs are rarely encountered. Classic algorithms (Apriori, FP-Growth) produce a lot of noise. A neural network model captures weak signals using embeddings and self-attention. We additionally apply data augmentation techniques and weighted sampling for long-tail categories.

Seasonal trends. Rules derived from January data don't work in July. Separate models for each season improve accuracy by 30%. We account for holidays, sales, and weather factors. For each season, we build its own rules and embeddings, allowing recommendations of seasonal product bundles.

Scaling. Processing 1 million transactions with FP-Growth on a single CPU takes 3-8 minutes. For daily real-time recalculations, we use incremental updates and GPU for the neural network. The hybrid architecture allows recalculating rules daily and fine-tuning the neural network weekly — a balance between speed and quality.

Dirty data is another problem. Missing values, duplicates, and outdated prices directly affect rule quality. We automate ETL processes with consistency checks.

Why is a hybrid approach better than classic rules?

According to a McKinsey report, retailers with AI recommendations increase revenue by 10–30%.

Classic FP-Growth + neural network: rules for high-support pairs (lift > 1.2, confidence > 0.3), neural network for long-tail. This yields +15-20% Recall@10 over pure association rules.

import pandas as pd
import numpy as np
from mlxtend.frequent_patterns import fpgrowth, association_rules
from mlxtend.preprocessing import TransactionEncoder
import torch
import torch.nn as nn

class BasketAnalyzer:
    """Classic FP-Growth + neural network extension"""

    def __init__(self, min_support: float = 0.01, min_confidence: float = 0.3):
        self.min_support = min_support
        self.min_confidence = min_confidence
        self.rules = None
        self.te = TransactionEncoder()

    def fit(self, transactions: list[list[str]]) -> pd.DataFrame:
        """
        transactions: [['milk', 'bread', 'butter'], ['milk', 'eggs'], ...]
        """
        te_array = self.te.fit_transform(transactions)
        df = pd.DataFrame(te_array, columns=self.te.columns_)

        # FP-Growth is 10-100x faster than Apriori for large datasets
        frequent_itemsets = fpgrowth(df, min_support=self.min_support, use_colnames=True)

        self.rules = association_rules(
            frequent_itemsets,
            metric='confidence',
            min_threshold=self.min_confidence
        )

        # Add lift and conviction for filtering
        self.rules = self.rules[self.rules['lift'] > 1.2]
        self.rules = self.rules.sort_values('lift', ascending=False)

        return self.rules

    def get_recommendations(self, basket: list[str],
                              top_k: int = 5) -> list[dict]:
        """Recommendations for the current basket"""
        if self.rules is None:
            return []

        basket_set = frozenset(basket)
        matching_rules = self.rules[
            self.rules['antecedents'].apply(lambda x: x.issubset(basket_set))
        ]

        # Remove items already in basket
        recommendations = []
        seen = set()
        for _, rule in matching_rules.iterrows():
            for item in rule['consequents']:
                if item not in basket_set and item not in seen:
                    recommendations.append({
                        'item': item,
                        'confidence': rule['confidence'],
                        'lift': rule['lift'],
                        'support': rule['support']
                    })
                    seen.add(item)
                    if len(recommendations) >= top_k:
                        break
            if len(recommendations) >= top_k:
                break

        return recommendations

    def get_category_affinity(self, transactions: pd.DataFrame) -> pd.DataFrame:
        """Affinity matrix between product categories"""
        # Transactions with categories instead of specific items
        cat_transactions = transactions.groupby('order_id')['category'].apply(list).tolist()
        te_cat = TransactionEncoder()
        cat_array = te_cat.fit_transform(cat_transactions)
        cat_df = pd.DataFrame(cat_array, columns=te_cat.columns_)

        # Category co-occurrence
        co_occurrence = cat_df.T.dot(cat_df)
        np.fill_diagonal(co_occurrence.values, 0)

        # Normalize by support (PMI-like measure)
        totals = cat_df.sum()
        n = len(cat_df)
        affinity = co_occurrence / n / (totals.values[:, None] * totals.values[None, :] / n**2 + 1e-9)

        return affinity


class NeuralBasketPredictor(nn.Module):
    """
    Neural network model for predicting the next item to add to the basket.
    Input: bag-of-items binary vector of the current basket.
    """

    def __init__(self, n_items: int, embedding_dim: int = 64):
        super().__init__()
        self.item_embedding = nn.Embedding(n_items, embedding_dim, padding_idx=0)
        self.attention = nn.MultiheadAttention(embedding_dim, num_heads=4, batch_first=True)
        self.fc = nn.Sequential(
            nn.Linear(embedding_dim, 256),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(256, n_items)
        )

    def forward(self, basket_item_ids: torch.Tensor) -> torch.Tensor:
        """
        basket_item_ids: (batch, seq_len) — indices of items in the basket
        Returns: (batch, n_items) — logits for each item
        """
        emb = self.item_embedding(basket_item_ids)  # (batch, seq, dim)
        attended, _ = self.attention(emb, emb, emb)  # Self-attention over basket
        pooled = attended.mean(dim=1)  # (batch, dim)
        return self.fc(pooled)

How to account for temporal patterns and seasonality?

class TemporalBasketAnalyzer:
    """Analysis of temporal purchase patterns"""

    def find_sequential_patterns(self, orders: pd.DataFrame,
                                   days_window: int = 7) -> pd.DataFrame:
        """Find sequential purchases within an N-day window"""
        orders_sorted = orders.sort_values(['user_id', 'order_date'])

        sequential = []
        for user_id, user_orders in orders_sorted.groupby('user_id'):
            order_list = user_orders.to_dict('records')
            for i, order_i in enumerate(order_list):
                for order_j in order_list[i+1:]:
                    days_diff = (order_j['order_date'] - order_i['order_date']).days
                    if days_diff > days_window:
                        break
                    if days_diff > 0:
                        sequential.append({
                            'item_a': order_i['sku'],
                            'item_b': order_j['sku'],
                            'days_between': days_diff
                        })

        df = pd.DataFrame(sequential)
        if df.empty:
            return df

        # Top sequential pairs
        return (df.groupby(['item_a', 'item_b'])
                  .agg(count=('days_between', 'count'),
                       avg_days=('days_between', 'mean'))
                  .reset_index()
                  .sort_values('count', ascending=False))

    def get_seasonal_baskets(self, transactions: pd.DataFrame) -> dict:
        """Seasonal baskets: what is usually bought together in a specific period"""
        transactions['month'] = transactions['order_date'].dt.month
        transactions['season'] = transactions['month'].map({
            12: 'winter', 1: 'winter', 2: 'winter',
            3: 'spring', 4: 'spring', 5: 'spring',
            6: 'summer', 7: 'summer', 8: 'summer',
            9: 'autumn', 10: 'autumn', 11: 'autumn'
        })

        seasonal_rules = {}
        for season, group in transactions.groupby('season'):
            season_transactions = group.groupby('order_id')['sku'].apply(list).tolist()
            if len(season_transactions) < 100:
                continue

            te = TransactionEncoder()
            arr = te.fit_transform(season_transactions)
            df_season = pd.DataFrame(arr, columns=te.columns_)
            freq = fpgrowth(df_season, min_support=0.02, use_colnames=True)

            if not freq.empty:
                rules = association_rules(freq, metric='lift', min_threshold=1.5)
                seasonal_rules[season] = rules.sort_values('lift', ascending=False).head(20)

        return seasonal_rules

Temporal patterns allow recommendations not only based on a single basket but also on purchase sequences over time. For example, after buying a printer, people often buy cartridges within a week. Seasonal baskets are built separately for each season, increasing recommendation relevance. For each user, the system considers their history — providing personalized temporal links.

Implementation process: from audit to deployment

1. Analytics — data audit (sources, quality, update frequency). Check for missing values, duplicates, transaction distribution.
2. Design — choose architecture (hybrid / rules only / NN only), prototype on a 10k transaction sample.
3. Development — train models, write REST API on FastAPI, integrate with CRM/ERP via webhook.
4. Testing — A/B test on 10% of audience, measure CTR and ATC, analyze lift by category.
5. Deployment — containerization with Docker, deploy in your Kubernetes or cloud, set up monitoring (Grafana, Prometheus, alerting).

What is included in the work (deliverables)

• Training 2-3 models (FP-Growth, Neural, Temporal) with hyperparameter tuning.
• REST API with documentation (OpenAPI 3.0) — endpoints: basket recommendations, seasonal rules, temporal sequences.
• Metrics dashboard (Grafana) — recall@k, precision@k, CTR, latency p99.
• Training for your engineers (2 workshops, 4 hours each) — architecture, result interpretation, fine-tuning.
• Warranty support for 12 months with response time up to 4 hours.

Approach comparison: metrics and speeds

The hybrid approach gives 45% more recall than pure rules while maintaining interpretability for top pairs. We use it for retail chains with 50+ stores — experience with over 10 projects.

Get a preliminary estimate of your data — we will analyze the structure and propose an optimal architecture in 2 days.

Contact us to discuss your task. Order a pilot project — get the first recommendations in 2 weeks. Get a consultation on assessing your data.

Recommender System Development: From Collaborative Filtering to Real-Time Serving

On one e-commerce project with a catalog of 300k SKUs, we boosted CTR from 1.8% to 4.4% — a 2.4x increase. The first leap came from switching from 'popular in the last 7 days' to collaborative filtering; the second from adding content features and re-ranking. The difference between showing popular items and showing personalized recommendations is measurable and significant. Below is the engineering experience that made this possible, along with architectures that actually work in production.

Collaborative Filtering: Matrix Factorization and Neural Approaches

Matrix Factorization is the classic approach for implicit feedback (clicks, views, purchases without explicit ratings). ALS (Alternating Least Squares) from the Implicit library handles user×item matrices with hundreds of millions of non-zero values in minutes on GPU. Latent factors 64–256, regularization λ=0.01–0.1 are starting parameters. Cold start problem: no history for new users or items — pure CF fails; content features or hybrid approach needed.

Neural Collaborative Filtering (NCF) replaces the dot product with a neural network. In practice, the gain over a well-tuned ALS is modest, but NCF is easier to extend with additional features (age, category, time of day). Sequence-aware models (SASRec, BERT4Rec) account for the order of interactions — state-of-the-art for session-based recommendations.

How to Choose Recommender System Architecture?

The answer depends on data, load, and cold start requirements. Below are three main approaches with selection criteria.

Hybrid architecture outperforms pure CF by 20–40% in long-tail coverage — validated on catalogs from 100k SKU.

Content-Based Filtering: When Interaction History is Scarce

Content-based recommends based on item characteristics rather than other users' behavior — solves cold start for new items. Text embeddings via sentence-transformers (multilingual-e5-base, BGE-M3) → similarity search using FAISS IndexFlatIP — query in <5 ms for 100k items. Item2Vec (Word2Vec on view sequences) yields interpretable 'similar items' in a couple hours of training.

Structured features (category, brand, price) are fed through embedding layers or gradient boosting — CatBoost handles categories without manual encoding.

Why Hybrid Models Work Better?

Production systems are almost always two-level. Stage 1 (Retrieval) — fast selection of 100–500 candidates from 300k items using ALS or Two-Tower model with vector search (FAISS, Qdrant). Stage 2 (Ranking) — heavy ranker on LightGBM or neural network with cross-features, time, device, and session context. LightFM is a good starting point for medium scale without heavy infrastructure. Our practice shows: moving from single-stage to two-stage yields a 15–25% accuracy improvement with only 20–30 ms additional latency.

Real-Time Serving: Architecture Under Load

Latency SLA — 50–100 ms at thousands of requests per second. Base recommendations precomputed (batch job hourly) → Redis by user_id → <5 ms. Real-time re-ranking via Kafka for events (clicks, cart adds) → update of context features. Feature serving — Redis with TTL (views in 24 hours, last clicked item). At 10k req/s, we deploy Redis Cluster with replication.

A/B testing is the only reliable way to measure improvements. Offline metrics do not always correlate with online. Kohavi et al., 'Online Controlled Experiments at Large Scale' (KDD 2013) — a must-read for the team. Test on 5–10% of traffic, monitor CTR, conversion, revenue per session. One of our client systems after hybridization increased revenue by 18% over a month of A/B.

Recommender System Development Timeline

The stages and typical time frames are in the table below. Costs are calculated individually based on catalog scale and latency requirements.

What's Included in Turnkey Development

1. Data audit — user×item matrix density (typically <0.1%), activity distribution, temporal patterns, cold start statistics.
2. Baseline — 'popular' as a simple threshold that is often hard to beat.
3. Iterative improvement — ALS → content features → two-stage → sequence-aware. Each step with A/B.
4. Serving infrastructure — batch precomputation, Redis, real-time re-ranking, Grafana monitoring.
5. Documentation — model card with metrics, deployment instructions, feature descriptions.
6. Team training — session on interpreting results and model fine-tuning.
7. Support — 1 month post-launch (incident fixes, pipeline tuning).

We are a team with 7+ years of experience in recommender systems, having delivered over 30 projects for e-commerce and media. We guarantee transparent A/B testing and documented metric improvements.

Want to assess the growth potential of your catalog? Contact us for a free data audit. Order recommender system development — first prototype within two weeks.

from implicit.als import AlternatingLeastSquares

model = AlternatingLeastSquares(
    factors=64,
    regularization=0.05,
    iterations=15,
    use_gpu=True
)
model.fit(user_item_matrix)

More about the mathematics of recommender systems — in specialized literature.