You redesigned a page: new headline, image, and button. Conversion improved, but you don't know which element made the difference. What if you could test all combinations and find the perfect mix? That's exactly what multivariate testing (MVT) does, helping you optimize conversion rate.
We are a team with 8+ years of conversion rate optimization (CRO) experience. We have run over 30 MVT testing experiments for e-commerce, SaaS, and landing pages. Over 30 projects, we've achieved an average conversion lift of 22%. Our methodology guarantees statistically significant results. For example, in one project we tested four elements on a product page: headline, image, description, and CTA. There were 48 combinations, but using a fractional factorial design we reduced it to 8. The result: a 23% lift in conversion over the original version, generating an additional $12,500 per month. In another project for a SaaS service, MVT testing on the registration page increased conversion by 31% by changing the form and testimonial block, adding $8,000 in monthly revenue. The average conversion lift across our projects is 15–30%, which at your current traffic level can mean significant additional revenue. For a typical project, clients see an average increase of $10,000 in monthly revenue, and our services start at $1,500, often yielding a 10x return.
We conduct web experiments analyzing all element combinations to optimize your site.
What is Multivariate Testing?
Wikipedia defines multivariate testing as a technique for testing a hypothesis in which multiple variables are modified. MVT testing is an experimental optimization method where multiple page elements are changed simultaneously: headline, image, CTA, form, or other blocks. Unlike an A/B test that compares only two versions, MVT creates all possible combinations of each element's variants. This not only identifies the winning combination but also reveals how elements interact — so-called interaction effects.
Why MVT testing is better than A/B testing?
| Criteria | A/B test | MVT testing |
|---|---|---|
| Number of changes | 1 (2 variants) | 2+ elements × N variants |
| Traffic volume | Lower | Significantly higher |
| Speed | Faster | Slower |
| Interaction effects | Not detected | Detected |
| Goal | Single winner | Best combination |
MVT testing is essential when changes are interdependent, you have enough traffic, and you need to understand how headline + CTA + image interplay. An A/B test won't show that a new headline amplifies the effect of a new button; MVT will.
How to choose elements for MVT?
Element selection is critical. We typically start with heatmaps and analytics: where users hover, which blocks are clickable. Typical candidates are headline, price, image, CTA, form. It's important not to test more than 3–4 elements simultaneously, or the number of combinations explodes. If traffic is low, we apply fractional factorial design or a full factorial experiment with partial fractioning.
How to calculate required traffic for MVT?
Example: testing on a product page:
- Headline: 2 variants (A, B)
- Image: 3 variants (A, B, C)
- CTA button: 2 variants (A, B)
Total: 2 × 3 × 2 = 12 combinations. Each combination is a separate variant.
Click to see code for traffic calculation
def mvt_sample_size(n_combinations, baseline_cr, mde=0.05, alpha=0.05, power=0.8):
"""Each combination requires the same traffic as an AB test"""
from scipy import stats
import math
p1 = baseline_cr
p2 = baseline_cr * (1 + mde)
p_avg = (p1 + p2) / 2
z_a = stats.norm.ppf(1 - alpha / 2)
z_b = stats.norm.ppf(power)
n_per_combo = ((z_a * math.sqrt(2 * p_avg * (1-p_avg)) +
z_b * math.sqrt(p1*(1-p1) + p2*(1-p2))) / (p2-p1)) ** 2
total = n_per_combo * n_combinations
print(f"Per combination: {math.ceil(n_per_combo):,}")
print(f"Total needed: {math.ceil(total):,}")
print(f"At 1000 daily visitors: {math.ceil(total/1000)} days")
mvt_sample_size(n_combinations=12, baseline_cr=0.04, mde=0.15)
# Per combination: 4,519
# Total needed: 54,228
# At 1000 daily visitors: 55 days
If traffic is insufficient, use fractional factorial design. For instance, with a Latin Square you test 4 combinations instead of 12, sacrificing some interaction information but conserving resources.
How we implement MVT testing on your site?
We use Optimizely or our own JavaScript solver. Example programmatic approach:
// Optimizely Snippet (add to <head>)
<script src="https://cdn.optimizely.com/js/PROJECT_ID.js"></script>
// Programmatic variant
const optimizely = window.optimizely || []
// Get variation for a specific experiment
const variationKey = optimizely.get('state').getVariationMap()['mvt_homepage_elements']
// variationKey = "headline_B_image_C_cta_A"
// Implementation without Optimizely
function getMVTVariant(userId, elements) {
const variants = {}
for (const [element, options] of Object.entries(elements)) {
const hash = cyrb53(`${userId}_${element}`)
variants[element] = options[hash % options.length]
}
return variants
}
const elements = {
headline: ['Buy today', '20% off for new customers'],
image: ['lifestyle', 'product-white', 'product-action'],
cta: ['Add to cart', 'Buy now'],
}
const variants = getMVTVariant(userId, elements)
// variants = { headline: 'Buy today', image: 'product-action', cta: 'Buy now' }
// Apply variants to DOM
applyVariants(variants)
// Log to analytics
gtag('event', 'mvt_assignment', {
experiment: 'product_page_mvt',
headline: variants.headline,
image: variants.image,
cta: variants.cta,
combination: Object.values(variants).join('_')
})
A custom implementation gives you full control over the experiment and avoids licensing fees. We also integrate the test with your analytics system — Google Analytics, Yandex.Metrica, or Amplitude. All our engineers are Optimizely certified and experienced with React, Vue, Next.js.
MVT result analysis
import pandas as pd
from scipy import stats
def analyze_mvt(data):
"""data: DataFrame with columns combination, visitors, conversions"""
data['cvr'] = data['conversions'] / data['visitors']
data = data.sort_values('cvr', ascending=False)
# Find best combination
best = data.iloc[0]
control = data[data['combination'] == 'baseline'].iloc[0]
print(f"\nTop combinations:")
print(data.head(5).to_string())
# Statistical significance of best vs control
from scipy.stats import proportions_ztest
_, p_value = proportions_ztest(
[best['conversions'], control['conversions']],
[best['visitors'], control['visitors']]
)
print(f"\nBest: {best['combination']} CVR={best['cvr']:.2%}")
print(f"Control: CVR={control['cvr']:.2%}")
print(f"P-value: {p_value:.4f}")
# Main effects analysis
for element in ['headline', 'image', 'cta']:
effect = data.groupby(element)['cvr'].mean()
print(f"\nMain effect of {element}:")
print(effect.sort_values(ascending=False))
We always analyze main effects — this helps understand which element contributes the most. In our experience, often the image has a larger effect than the headline. For instance, in one project changing only the image gave +8% conversion, while simultaneously changing headline and CTA gave +12%.
Step-by-step process for running MVT
- Analysis (1-2 days): Collect current conversion data, identify elements to test.
- Design (1 day): Design combinations, applying fractional factorial if needed.
- Implementation (1-2 days): Set up tool (Optimizely/JS), integrate with analytics.
- Validation (1 day): Verify correct traffic distribution, absence of errors.
- Launch (1 day): Start the test, monitor for deviations.
- Analysis (1-2 days): After completion, collect results, compute winner and main effects.
Typical MVT mistakes
- Testing too many elements (≥5) — exponential growth of combinations.
- Prematurely stopping the test before reaching statistical significance.
- Using identical variants for different elements (e.g., two identical headlines).
- Ignoring interaction effects — interpreting main effects without checking interactions.
- Ignoring traffic segments: effects may differ on mobile users.
We guarantee that each test undergoes validation for correct traffic distribution and absence of conflicts with other experiments.
What's included in our work
- Documentation: hypothesis description, experiment design, analysis plan.
- Code implementation on your site (with repository access).
- Integration with Google Analytics, Yandex.Metrica, or any other system.
- Weekly progress reports.
- Final report with recommendations for implementing the winning combination.
- One month of support after test completion.
Timeline and cost
Planning and implementation of an MVT with full analysis takes from 5 to 7 business days. Cost is calculated individually, depending on the experiment complexity and number of elements. For a typical MVT with 3 elements, costs start from $1,500. We guarantee transparency at every stage.
If you're considering MVT testing, order a consultation for a preliminary assessment of traffic and hypotheses. Contact us to get an evaluation of your project. Describe your task, and we'll propose an optimal testing plan.