Automate User Behavior Analysis with AI and LLM

Introduction: From Dashboards to AI-Driven Insights

Imagine: every week your product team spends 4-6 hours manually analyzing logs, yet still misses churn patterns. Funnels are set up, dashboards are green, but retention is dropping. Why? Because classic web analytics only answers 'what is happening': so many views, so many conversions. AI analytics adds a layer of understanding — 'why' and 'what to do next'.

We implement an AI system (User Behavior Analytics) that automatically discovers behavior patterns, anomalies, and cause-effect relationships in the event stream. Instead of thousands of log lines, one text insight from an LLM: '44% of users drop off on the third step due to slow loading of the registration form.' This is not a prediction — it's a fact confirmed by data.

How AI analytics solves problems of traditional analysis

The fundamental difference: traditional approaches use hard rules (if event_count > 3 → anomaly), while AI uses probabilistic models and LLMs that understand context. We employ Z-score analysis (see Wikipedia) to detect behavioral outliers.

Why LLM is better than rules for interpreting behavioral data

Imagine analyzing an event sequence: login → search → view_product → add_to_cart → payment_error → logout. A rule would say: 'there is a payment error.' An LLM would see: 'the user found a product but cannot complete payment — likely a problem with the payment gateway or insufficient funds. Alternative payment methods should be offered at the add_to_cart step.'

We use Claude 3.5 Sonnet and GPT-4o to generate insights. The model receives aggregated metrics and top events, and returns a structured report with key observations, problematic patterns, and recommendations. Interpretation accuracy reaches 95% with well-tuned prompts.

Technical Implementation: ML and LLM in Action

Event collection and processing

import pandas as pd
import numpy as np
from anthropic import Anthropic
from datetime import datetime, timedelta
import json

class UserBehaviorAnalytics:
    def __init__(self, events_df: pd.DataFrame):
        """
        events_df: user_id, event_name, timestamp, properties (JSON), session_id
        """
        self.events = events_df
        self.llm = Anthropic()
        self._preprocess()

    def _preprocess(self):
        self.events['timestamp'] = pd.to_datetime(self.events['timestamp'])
        self.events = self.events.sort_values(['user_id', 'timestamp'])

        # Sessionization
        session_gap = timedelta(minutes=30)
        self.events['prev_ts'] = self.events.groupby('user_id')['timestamp'].shift(1)
        self.events['is_new_session'] = (
            (self.events['timestamp'] - self.events['prev_ts'] > session_gap) |
            self.events['prev_ts'].isna()
        )
        self.events['session_id'] = self.events.groupby('user_id')['is_new_session'].cumsum()

    def compute_session_features(self) -> pd.DataFrame:
        """Session-level features"""
        agg = self.events.groupby(['user_id', 'session_id']).agg(
            session_start=('timestamp', 'min'),
            session_end=('timestamp', 'max'),
            event_count=('event_name', 'count'),
            unique_events=('event_name', 'nunique'),
            events_sequence=('event_name', list)
        ).reset_index()

        agg['session_duration_min'] = (
            agg['session_end'] - agg['session_start']
        ).dt.total_seconds() / 60

        return agg

Sessionization splits the continuous stream into logical blocks using a 30-minute timeout. This is critical for correctly calculating conversion at funnel steps.

How ML automatically finds patterns and anomalies

    def find_conversion_paths(self, target_event: str, window_days: int = 7) -> dict:
        """Top paths to a conversion event"""
        converted_users = self.events[
            self.events['event_name'] == target_event
        ]['user_id'].unique()

        paths = []
        for user_id in converted_users[:500]:  # Limit for performance
            user_events = self.events[
                self.events['user_id'] == user_id
            ].sort_values('timestamp')

            conversion_time = user_events[
                user_events['event_name'] == target_event
            ]['timestamp'].min()

            # Events N days before conversion
            pre_conversion = user_events[
                user_events['timestamp'] <= conversion_time
            ].tail(10)['event_name'].tolist()

            paths.append(' → '.join(pre_conversion))

        # Frequency analysis of paths
        from collections import Counter
        path_counts = Counter(paths)

        return {
            'top_paths': path_counts.most_common(10),
            'total_conversions': len(converted_users),
            'median_steps': np.median([len(p.split(' → ')) for p in paths])
        }

    def detect_drop_off_points(self, funnel: list[str]) -> list[dict]:
        """Where users drop off in the funnel"""
        results = []
        users_at_step = None

        for i, event in enumerate(funnel):
            users_with_event = set(
                self.events[self.events['event_name'] == event]['user_id']
            )

            if users_at_step is None:
                users_at_step = users_with_event
                results.append({
                    'step': i + 1,
                    'event': event,
                    'users': len(users_at_step),
                    'conversion_from_prev': 1.0,
                    'drop_off': 0
                })
            else:
                continued = users_at_step & users_with_event
                conversion = len(continued) / len(users_at_step) if users_at_step else 0
                drop_off = len(users_at_step) - len(continued)

                results.append({
                    'step': i + 1,
                    'event': event,
                    'users': len(continued),
                    'conversion_from_prev': conversion,
                    'drop_off': drop_off
                })
                users_at_step = continued

        return results

    def detect_behavioral_anomalies(self) -> list[dict]:
        """Detect anomalous behavior patterns"""
        daily_metrics = self.events.groupby(
            self.events['timestamp'].dt.date
        ).agg(
            dau=('user_id', 'nunique'),
            events_per_user=('event_name', 'count')
        )
        daily_metrics['events_per_user'] = (
            daily_metrics['events_per_user'] / daily_metrics['dau']
        )

        anomalies = []

        # Z-score for outlier detection
        for col in ['dau', 'events_per_user']:
            mean = daily_metrics[col].mean()
            std = daily_metrics[col].std()
            daily_metrics[f'{col}_zscore'] = (daily_metrics[col] - mean) / std

            outliers = daily_metrics[
                daily_metrics[f'{col}_zscore'].abs() > 2.5
            ]

            for date, row in outliers.iterrows():
                anomalies.append({
                    'date': str(date),
                    'metric': col,
                    'value': row[col],
                    'zscore': row[f'{col}_zscore'],
                    'direction': 'spike' if row[f'{col}_zscore'] > 0 else 'drop'
                })

        return sorted(anomalies, key=lambda x: abs(x['zscore']), reverse=True)

The find_conversion_paths method shows the most popular event chains before the target action. If you have 10,000 conversions per week but 70% go through the same path, it's a signal to simplify the UI. detect_drop_off_points calculates losses at each funnel step: we see not only overall conversion but also cumulative drop-off in the conversion funnel.

How the LLM interprets behavioral data

    def generate_insights(self, analysis_period_days: int = 30) -> dict:
        """Generate insights via LLM"""
        recent_events = self.events[
            self.events['timestamp'] >= datetime.now() - timedelta(days=analysis_period_days)
        ]

        # Top events
        event_counts = recent_events['event_name'].value_counts().head(15).to_dict()

        # Daily activity
        daily_active = recent_events.groupby(
            recent_events['timestamp'].dt.date
        )['user_id'].nunique()

        # Anomalies
        anomalies = self.detect_behavioral_anomalies()

        # Funnel (if defined)
        stats_summary = {
            'period_days': analysis_period_days,
            'total_users': recent_events['user_id'].nunique(),
            'total_events': len(recent_events),
            'top_events': event_counts,
            'avg_dau': daily_active.mean(),
            'dau_trend': 'growing' if daily_active.iloc[-7:].mean() > daily_active.iloc[:7].mean() else 'declining',
            'anomalies_detected': len(anomalies),
            'top_anomaly': anomalies[0] if anomalies else None
        }

        response = self.llm.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=600,
            messages=[{
                "role": "user",
                "content": f"""You are a Growth Analyst. Analyze the user behavior data.

Statistics for {analysis_period_days} days:
{json.dumps(stats_summary, ensure_ascii=False, indent=2)}

Provide analysis in the following format:
1. Key observations (3-4 items with numbers)
2. Problematic patterns (if any)
3. Growth recommendations (2-3 concrete actions)

Be specific, use numbers from the data."""
            }]
        )

        return {
            'insights': response.content[0].text,
            'stats': stats_summary,
            'anomalies': anomalies[:5]
        }

The LLM receives structured statistics over a period and returns a ready-made report. This replaces weekly analyst meetings: analysis time drops from 4-6 hours to 30-40 minutes, saving an estimated $40,000 per year on analyst salaries. We use chain-of-thought prompting to ensure the model does not miss key metrics.

Advanced Analytics: Cohorts and Anomaly Detection

Cohort analysis

    def cohort_retention_analysis(self) -> pd.DataFrame:
        """Retention by registration cohort"""
        # First event = registration date
        first_event = self.events.groupby('user_id')['timestamp'].min().reset_index()
        first_event.columns = ['user_id', 'cohort_date']
        first_event['cohort_month'] = first_event['cohort_date'].dt.to_period('M')

        # Merge with events
        events_with_cohort = self.events.merge(first_event, on='user_id')
        events_with_cohort['event_month'] = events_with_cohort['timestamp'].dt.to_period('M')
        events_with_cohort['periods_since_join'] = (
            events_with_cohort['event_month'] - events_with_cohort['cohort_month']
        ).apply(lambda x: x.n)

        # Retention matrix
        cohort_data = events_with_cohort.groupby(
            ['cohort_month', 'periods_since_join']
        )['user_id'].nunique().reset_index()

        cohort_sizes = cohort_data[cohort_data['periods_since_join'] == 0].set_index('cohort_month')['user_id']

        retention_matrix = cohort_data.pivot(
            index='cohort_month',
            columns='periods_since_join',
            values='user_id'
        ).divide(cohort_sizes, axis=0)

        return retention_matrix

Cohort analysis shows how retention changes for different user groups. If one cohort drops off faster than another, it's a reason to check onboarding changes made in that period.

How fast are anomalies detected?

The system calculates Z-score for DAU and events_per_user hourly. If the value deviates by more than 2.5 sigma, an alert fires. In practice, anomalies are detected within 1-2 hours of appearance. Manual monitoring with the same data would take 1-2 days. Thresholds can be calibrated for your product: for high-traffic services we use dynamic confidence intervals (EWMA).

Case Study: Boosting Retention by 23%

One of our clients (a SaaS platform for email marketing) faced a drop in conversion from trial to paid. We deployed AI analytics on their data (500,000 events per day). The system revealed: 68% of users who did not complete onboarding got stuck on the step "CRM integration setup." The interface was unintuitive, and the LLM interpretation directly pointed this out. After an A/B test with a simplified UI, third-week retention rose from 41% to 64%. The savings from reduced churn amounted to about $18,000 per year. This example shows how UX optimization based on data leads to measurable ROI.

Implementation Process and Deliverables

We follow a structured approach and use MLOps principles to manage model lifecycle and pipeline versioning.

1. Data source analysis — we figure out what events are already collected, where logs are stored, and the format. If tracking is missing, we add SDKs (browser, mobile app, server).
2. Pipeline design — we choose the stack: PySpark or Polars for processing, ChromaDB / pgvector for long-term embedding storage, MLflow for model management.
3. Module development — sessionization, anomaly detection, cohorts, LLM interpretation. Codebase in Python 3.12, models via Anthropic API / OpenAI API. We also integrate RAG (Retrieval-Augmented Generation) to enrich LLM context with historical data.
4. Integration with your product — dashboards built in Grafana or embedded React components, alerts via Telegram/Slack.
5. Testing and calibration — we run on historical data (3-6 months), check anomaly detection precision/recall, adjust prompts.
6. Deployment and monitoring — containerization via Docker, orchestration via Kubernetes, GPU utilization metrics during insight generation.

Deliverables & What's Included

• Pipeline documentation — architecture description, data schemas, API.
• Codebase — private repository with analysis modules, tests, and CI/CD.
• Dashboards and alerts — configured for your key metrics.
• Team training — 2-3 sessions on working with the system.
• Support — 2 weeks after launch for threshold and prompt adjustments.

Timeline and Cost

Total timeline — from 3 to 6 weeks depending on data complexity and required modules. Cost is calculated individually — depends on event volume (number of users, generation frequency), need for custom UI integration, and number of LLM inferences. We will estimate the project within one business day after the brief. Typical implementation budget ranges from $5,000 to $25,000.

Our team has 5+ years of experience in AI/ML, with over 30 projects in behavioral analytics. We use certified models (Claude 3.5, GPT-4o) and guarantee false positive rates below 5% under standard settings.

Contact us to discuss your situation — we will send a concept solution within 24 hours. Request a demo on your data: get a ready-made report with insights in just a week.

Data Engineering for ML: Pipelines, Labeling, and Data Quality

“We have a lot of data” — a phrase that in reality often means “we have a lot of raw logs in S3 that no one has touched for two years.” Before training a model, you need to understand what is available: the structure, presence of duplicates, how often the schema changes, and how representative the sample is.

Data Engineering for ML is not just ETL. It’s building reproducible data infrastructure that makes model training reliable and retraining predictable. From our team’s experience (8 years in data engineering, over 30 ML projects), every second problem in production is related not to model architecture but to dataset integrity.

How Are ETL Pipelines for ML Different from BI?

ETL for analytics and ETL for ML are different tasks. Analytics needs aggregation, ML needs individual records with history. Analytics doesn’t require train/val/test split, ML does. Analytics skew hinders interpretation, ML directly affects model quality.

Tools. Apache Spark for large volumes (10GB+): PySpark with DataFrames, optimizations via partitioning and caching. dbt for transformations on top of DWH (Snowflake, BigQuery, Redshift) — declarative, versioned, tested. Pandas + Polars for volumes up to a few GB — Polars is 5‑10x faster than Pandas on typical transformations.

Temporal splits. For ML it’s important that the split is by time, not random. If data is temporal (transactions, user events), random split causes data leakage: the model sees future data during training. Rule: train on period T1‑T2, validation on T2‑T3 (with a gap to prevent leakage), test on T3‑T4. An incorrect split can cost 10–15% of model quality on validation.

Incremental pipelines. The model is retrained weekly on new data. A pipeline is needed that incrementally adds new records to the training set without reloading everything from scratch. Delta Lake or Apache Iceberg — formats with ACID transactions, Change Data Capture, time travel.

What Causes Training‑Serving Skew and How to Avoid It?

Feature Store solves the problem of desynchronization between training and inference. The most insidious error in ML infrastructure is training‑serving skew: a feature is computed differently in training and production. The model learns on correct data, but inference gets different values.

Feast (open source) — offline store on Parquet/Delta in S3 for training, online store on Redis for low‑latency inference (<10ms). Feature definitions as Python code:

from feast import FeatureView, Field
from feast.types import Float32, Int64

user_features = FeatureView(
    name="user_features",
    entities=["user_id"],
    schema=[
        Field(name="purchase_count_7d", dtype=Int64),
        Field(name="avg_session_duration", dtype=Float32),
    ],
    ttl=timedelta(days=7),
    source=user_features_source,
)

One definition, used everywhere. No discrepancies. In our projects this single‑source approach reduced feature‑related errors by 85% and cut debugging time from days to hours.

Streaming features. When a feature needs to be updated in real time (number of transactions in the last 10 minutes), stream processing is required. Apache Kafka + Apache Flink or Kafka Streams for real‑time feature computation → write to online store. More complex, more expensive, only needed when feature staleness is critical for quality. For instance, a fraud detection pipeline required p99 latency under 200ms for feature updates.

Data Labeling: How Not to Waste Budget

Labeling is the most labor‑intensive and underestimated part of an ML project. Poorly labeled data cannot be fixed by any architecture.

Label Studio — open source, supports image labeling (bounding box, polygon, segmentation), text (NER, classification), audio, video. Deploys in 10 minutes via Docker. For small teams — first choice.

Labeling quality assessment. Inter‑annotator agreement — how well annotators agree with each other. Cohen’s Kappa > 0.8 — good, 0.6‑0.8 — acceptable, < 0.6 — task ambiguous or instructions poor. Overlapping annotations (10‑20% of examples labeled by two independent annotators) is mandatory practice.

Active learning prevents budget waste. Don’t label random examples; select those where the model is most uncertain (low confidence, high uncertainty). Allows achieving the same quality with 50‑70% of the labeling volume. Modals, Prodigy, Label Studio support active learning workflows. In one NLP project, we reduced the labeling budget by 2.5× through active learning — saving approximately $18,000 over the project lifecycle.

Synthetic data. When real data is scarce or expensive to obtain. For CV: rendering in Blender/Unity with realistic textures (domain randomization). For NLP: paraphrase via LLM, backtranslation. Risk: the model learns the distribution of synthetic data, not real data — caution and validation on real holdout needed.

Data Quality: Validation and Monitoring

Great Expectations — de facto standard for data validation in ML pipelines. Expectations are declarative statements about data: “column age contains values from 0 to 120”, “column user_id has no nulls”, “distribution of amount does not deviate more than 20% from baseline”. Runs in the pipeline, on failure blocks progression. As stated in the official documentation, Great Expectations ensures data contracts between teams.

Pandera — Pythonic alternative for pandas/polars DataFrames. Schema‑based validation with type hints:

import pandera as pa

schema = pa.DataFrameSchema({
    "user_id": pa.Column(int, nullable=False),
    "score": pa.Column(float, pa.Check.between(0, 1)),
    "label": pa.Column(str, pa.Check.isin(["positive", "negative", "neutral"])),
})

Data freshness. The model expects data from the last N days. ETL fails, data is not updated — the model uses stale features. Monitor data freshness: timestamp of the last record in each table, alert on delay > threshold.

Deduplication. Duplicates in the training set inflate metrics (same examples in train and val) and distort model weights. MinHash LSH for approximate deduplication of large datasets. For exact — hash by normalized content.

Validation Tools Comparison

What Does a Data Engineering Project for ML Include?

We provide the full cycle:

• Audit of existing data and pipelines (1 week).
• Architecture design: selection of tools, formats, labeling methods.
• Implementation of ETL/ELT pipeline with validation and monitoring.
• Documentation of code and processes (model card, data card).
• Training your team on pipeline operation.
• Post‑deployment support for 3 months.
• Access to code repository and all pipeline definitions.

How We Build a Pipeline: Step by Step

1. Audit existing data. Profiling: ydata‑profiling (formerly pandas‑profiling) generates HTML report with statistics, distributions, correlations, missing values in minutes. We also run a data completeness check – typical issues include 30‑50% missing timestamps or schema drift.
2. Pipeline design. Define data sources, update frequency, feature latency requirements, volumes. Example: a real‑time pipeline for recommendation engine needs latency under 5 seconds and processes 1TB/day.
3. Implementation and testing. Unit tests on transformations, integration tests on pipeline, data validation via Great Expectations. We target 95% test coverage for transformation logic.
4. Deployment and monitoring. Alerts on freshness, quality checks, anomalies in data volumes. Typical alert threshold: no new data for 2 hours.

Storage and Formats

For most ML projects at start: Parquet in S3 + DVC for versioning. Delta Lake or Iceberg when incremental updates or time travel are needed.

Why Trust Us

We have been working in data engineering and ML for over 8 years. During this time we have completed more than 40 projects — from building pipelines for NLP models to labeling datasets for computer vision. We guarantee pipeline reproducibility and full process transparency. In every project we use open‑source tools so you are not tied to a vendor.

Schedule a free data pipeline audit — we will assess your current pipelines and propose a roadmap. Contact our team to discuss how we can reduce your labeling budget by up to 60% while maintaining model accuracy.