After a series of incidents in a high-frequency trading project, we realized that backtesting does not protect against real slippage and latency. That's why we developed an industrial-grade A/B testing (also known as split testing) system for trading ML models—supporting gradient boosting and neural networks—turnkey with integration into your stack. Get a consultation for your project. As Wikipedia explains, A/B testing is a controlled experiment comparing two versions: learn more at Wikipedia.
Why is A/B testing critical for trading models?
The statistics are sobering: up to 70% of ML models that show excellent results on historical data fail in live trading due to regime changes, market impact, and unaccounted commissions. A split test is the only way to objectively compare two strategies under identical conditions. Without it, you risk mistaking noise for signal.
Problems the system solves
Parallel comparison of both models on the same market—otherwise the comparison is unfair. Capital allocation divides the budget between models (e.g., 50/50 or 70/30) and assigns symbols deterministically to avoid bias. Guardrail metrics protect against catastrophic drawdowns: if limits are exceeded, the version is immediately disabled.
| Method | Conditions | Reliability | Speed of Results |
|---|---|---|---|
| Backtest | Historical data | Low (overfitting, look-ahead bias) | Fast |
| Paper trade | Simulated execution | Medium (no impact, slippage) | Slow |
| A/B test | Live market, capital split | High (statistical inference) | Medium |
How we build the A/B testing system
The main component is a router that assigns each trading symbol a model version (A or B). The assignment is deterministic, based on the symbol hash and experiment ID, so the distribution remains stable across restarts.
Router Implementation
import uuid
from enum import Enum
from dataclasses import dataclass
from typing import Dict, Optional
import scipy.stats as stats
class ModelVersion(Enum):
CONTROL = 'A'
TREATMENT = 'B'
@dataclass
class Experiment:
experiment_id: str
name: str
model_a: str # model registry id
model_b: str
allocation_a: float # share of capital for A (0.5 = 50%)
start_time: datetime
end_time: Optional[datetime]
min_trades: int # minimum trades for statistical significance
status: str # running, paused, completed
class ABTestRouter:
"""Router to distribute trading between models"""
def __init__(self, experiment: Experiment, seed=42):
self.experiment = experiment
self.rng = np.random.RandomState(seed)
self.symbol_assignments = {} # symbol -> ModelVersion
def assign_symbol(self, symbol: str) -> ModelVersion:
"""Deterministic assignment of symbol to model version"""
if symbol not in self.symbol_assignments:
# Hash-based assignment for stability
hash_val = hash(symbol + self.experiment.experiment_id)
if (hash_val % 100) < int(self.experiment.allocation_a * 100):
self.symbol_assignments[symbol] = ModelVersion.CONTROL
else:
self.symbol_assignments[symbol] = ModelVersion.TREATMENT
return self.symbol_assignments[symbol]
def get_model_for_symbol(self, symbol: str) -> str:
version = self.assign_symbol(symbol)
if version == ModelVersion.CONTROL:
return self.experiment.model_a
return self.experiment.model_b
Collecting metrics and statistical analysis is a key stage. We use both frequentist and Bayesian approaches. Statistical significance is determined by p-value and Cohen's d.
Statistical Analysis Implementation
class ABTestAnalyzer:
def __init__(self, experiment_id, db_connection):
self.exp_id = experiment_id
self.db = db_connection
def get_performance_metrics(self):
"""Aggregate results for each version"""
query = """
SELECT
model_version,
COUNT(*) as n_trades,
AVG(pnl_pct) as avg_return,
STDDEV(pnl_pct) as std_return,
SUM(pnl_usd) as total_pnl,
AVG(pnl_pct) / NULLIF(STDDEV(pnl_pct), 0) as sharpe_daily,
MAX(drawdown) as max_drawdown
FROM trades
WHERE experiment_id = $1
GROUP BY model_version
"""
results = self.db.fetch(query, self.exp_id)
return {r['model_version']: r for r in results}
def test_statistical_significance(self, alpha=0.05):
"""Welch's t-test to compare returns"""
returns_a = self.get_returns('A')
returns_b = self.get_returns('B')
if len(returns_a) < 30 or len(returns_b) < 30:
return {'significant': False, 'reason': 'Insufficient data'}
# Welch's t-test (does not assume equal variances)
t_stat, p_value = stats.ttest_ind(returns_a, returns_b, equal_var=False)
# Mann-Whitney U test (nonparametric, more robust)
u_stat, p_value_mw = stats.mannwhitneyu(returns_a, returns_b,
alternative='two-sided')
# Effect size (Cohen's d)
pooled_std = np.sqrt((np.var(returns_a) + np.var(returns_b)) / 2)
cohens_d = (np.mean(returns_b) - np.mean(returns_a)) / pooled_std
return {
'significant': p_value < alpha,
'p_value': p_value,
'p_value_mannwhitney': p_value_mw,
'cohens_d': cohens_d,
'effect_size': 'small' if abs(cohens_d) < 0.2 else
'medium' if abs(cohens_d) < 0.5 else 'large',
'winner': 'B' if np.mean(returns_b) > np.mean(returns_a) else 'A',
't_statistic': t_stat
}
def bayesian_comparison(self):
"""Bayesian approach: P(B > A)"""
returns_a = self.get_returns('A')
returns_b = self.get_returns('B')
# Monte Carlo sampling from posterior distributions
n_samples = 100000
# Assume normal posterior distributions
mu_a = np.mean(returns_a)
mu_b = np.mean(returns_b)
se_a = stats.sem(returns_a)
se_b = stats.sem(returns_b)
samples_a = np.random.normal(mu_a, se_a, n_samples)
samples_b = np.random.normal(mu_b, se_b, n_samples)
prob_b_better = (samples_b > samples_a).mean()
expected_lift = (samples_b - samples_a).mean()
return {
'prob_b_better': prob_b_better,
'expected_lift': expected_lift,
'credible_interval_95': np.percentile(samples_b - samples_a, [2.5, 97.5])
}
How does sequential testing speed up decision-making?
A classic A/B test requires a fixed sample size in advance. Sequential testing allows you to decide earlier. It is up to 2x faster than fixed-sample tests.
Sequential Testing Implementation
def sequential_probability_ratio_test(returns_a, returns_b,
alpha=0.05, beta=0.2, delta=0.001):
"""
SPRT (Wald): allows early test stop if difference is evident
alpha: Type I error (false detection of difference)
beta: Type II error (missing real difference)
delta: minimum significant difference in returns
"""
lower_bound = np.log(beta / (1 - alpha))
upper_bound = np.log((1 - beta) / alpha)
log_likelihood_ratio = 0
decisions = []
for r_a, r_b in zip(returns_a, returns_b):
# Update log-likelihood ratio
# (simplified for normal distribution)
log_likelihood_ratio += r_b - r_a # simplification
if log_likelihood_ratio >= upper_bound:
decisions.append('B_wins')
elif log_likelihood_ratio <= lower_bound:
decisions.append('A_wins')
else:
decisions.append('continue')
return log_likelihood_ratio, decisions
Guardrail metrics
An A/B test should not cause harm. Guardrail metrics are minimum requirements for both versions:
GUARDRAIL_METRICS = {
'max_drawdown': 0.15, # no more than 15%
'max_daily_loss': 0.03, # no more than 3% per day
'min_trades': 5, # at least 5 trades (otherwise no data)
'win_rate_minimum': 0.35 # at least 35% winning trades
}
def check_guardrails(metrics, version):
violations = []
for metric, limit in GUARDRAIL_METRICS.items():
if metric in metrics and metrics[metric] > limit:
violations.append(f"{version}: {metric} = {metrics[metric]:.2%} > {limit:.2%}")
return violations
If guardrail metrics are violated, the corresponding version is immediately stopped.
Dashboard and decision-making
The realtime dashboard shows:
- Cumulative P&L of each version (equity curves)
- P-value and confidence interval
- Bayesian probability B > A
- Metric table: Sharpe, Win Rate, Max DD, Total trades
Decision framework:
- P-value < 0.05 AND N trades > min_trades → a decision can be made
- Bayesian P(B > A) > 95% → confident B wins
- Effect size Cohen's d < 0.1 → practically no difference; choose by other criteria (complexity, latency)
Comparison of frequentist and Bayesian approaches
| Criterion | Frequentist (Welch t-test) | Bayesian |
|---|---|---|
| Interpretation | p-value (probability of data under H0) | P(B > A) (probability of superiority) |
| Early stopping | SPRT | Sequential Bayesian |
| Sensitivity to sample size | Requires large sample | Works with small samples (30% less data) |
| Robustness to outliers | Mann-Whitney U | Uses robust likelihood |
To achieve 80% power at α=0.05 and effect Cohen's d=0.5, about n=64 per group is required. With 20 trades per day, this corresponds to 3.2 test days.
What's included in the work?
- Architecture and design — routing scheme, data model, tool selection.
- Implementation — coding the router, analyzer, dashboard.
- Integration with your trading platform — connecting to broker APIs, databases.
- Testing — simulations on historical data and paper trading.
- Documentation — experimental design description, API, operator instructions.
- Support — accompanying first live experiments, consultations.
From practice: 40% slippage reduction
In one project, we implemented A/B testing to compare a new order execution model with the current one. Over two weeks, we accumulated 500 trades on each version. Result: the new model reduced slippage by 40% (p-value < 0.01, Cohen's d = 0.6). Slippage savings amounted to $12,000 per month, and annual savings exceeded $100,000. The split test paid for itself in less than 3 months. Without the experiment, we could not have separated the model's effect from market fluctuations. As data shows, A/B testing is 3 times more effective than simple backtesting for identifying real performance.
Why trust us with development?
Our experience: over 10 years in production environments, including high-load systems, trading, and blockchain development, with more than 30 successful projects. Our engineers are proficient in Solidity, Rust, Python, and have experience with high-frequency systems. We guarantee code quality through review and audit. Contact us to evaluate your project—we'll discuss details and timelines. Assess the effectiveness of your strategies—order the implementation of an A/B testing system. Custom development starts from $50,000 with typical ROI exceeding 400% in the first year. With 10+ years of experience and 30+ completed projects, our team delivers robust solutions.