Recreating Real Traffic: From Logs to k6 Scenarios
A typical mistake in load testing is using uniform request pacing with a fixed number of virtual users. In reality, traffic has peaks (morning and evening), different user types (mobile browsers, API clients), random pauses, and an 80/20 distribution. For example, 80% of requests hit 20% of pages. Synthetic tests often miss issues with caching, session state, and concurrency.
We offer an approach based on analyzing real traffic from Nginx logs or Google Analytics, and generating k6 scenarios that accurately reproduce real user behavior. Realistic testing is 3 times better than uniform load in identifying performance issues. Clients achieve significant savings on debugging by catching issues early. Contact us to discuss your project.
How Log Analysis Works
# Extract patterns from nginx access log
import re
from collections import Counter, defaultdict
import json
def analyze_access_log(log_file: str):
pattern = re.compile(
r'(?P<ip>\S+) .+ \[(?P<time>[^\]]+)\] '
r'"(?P<method>\w+) (?P<path>[^"]+) HTTP/\d+" '
r'(?P<status>\d+) (?P<bytes>\d+)'
)
endpoint_counts = Counter()
method_counts = Counter()
hourly_traffic = defaultdict(int)
with open(log_file) as f:
for line in f:
m = pattern.match(line)
if not m:
continue
# Normalize path (remove IDs)
path = re.sub(r'/\d+', '/{id}', m.group('path').split('?')[0])
endpoint_counts[f"{m.group('method')} {path}"] += 1
method_counts[m.group('method')] += 1
# Hourly distribution
hour = m.group('time').split(':')[1]
hourly_traffic[hour] += 1
total = sum(endpoint_counts.values())
print("=== Top Endpoints (% of traffic) ===")
for endpoint, count in endpoint_counts.most_common(20):
pct = count / total * 100
print(f" {pct:.1f}% {endpoint}")
print("\n=== Hourly Distribution ===")
for hour in sorted(hourly_traffic):
bar = '█' * (hourly_traffic[hour] // 100)
print(f" {hour}:00 {bar} {hourly_traffic[hour]}")
# Export for k6 scenario
weights = {ep: round(cnt/total, 3) for ep, cnt in endpoint_counts.most_common(20)}
return weights
The extracted weights are exported to JSON and used to generate k6 scenarios. Analyzing access logs lets us identify the 20% of endpoints generating 80% of traffic, following the Pareto principle.
Limitations of Uniform Traffic
Uniform load doesn't create the "crowd" effect: when 1000 users simultaneously navigate to a product after a social media post. It doesn't test session caching, database locks under concurrent writes, or degradation under sustained peaks. Realistic simulation with a Pareto (80/20) distribution and session behavior reproduces such scenarios, uncovering bottlenecks before production deployment.
How to Build a Scenario Based on Logs
From the extracted weights we generate k6 scenarios. For each user type we create a separate executor with different intensity. For example, 40% traffic – anonymous browsers, 50% – logged-in users, 10% – API clients. Scenarios include random pauses, branching, and probabilistic transitions.
// tests/realistic/user-journey.js
import http from 'k6/http'
import { check, sleep } from 'k6'
import { SharedArray } from 'k6/data'
import { randomItem, randomIntBetween } from 'https://jslib.k6.io/k6-utils/1.4.0/index.js'
// Load test data from CSV
const users = new SharedArray('users', function() {
return open('./data/test-users.csv').split('\n')
.slice(1)
.map(row => {
const [email, token, userId] = row.split(',')
return { email, token, userId }
})
})
const searchTerms = new SharedArray('searches', function() {
return open('./data/popular-searches.txt').split('\n').filter(Boolean)
})
export const options = {
scenarios: {
// Anonymous browsers (40% of traffic)
anonymous_browse: {
executor: 'ramping-vus',
startVUs: 0,
stages: [
{ duration: '5m', target: 40 },
{ duration: '30m', target: 40 },
{ duration: '5m', target: 0 }
],
exec: 'anonymousBrowse'
},
// Logged-in users (50% of traffic)
logged_in_users: {
executor: 'ramping-vus',
startVUs: 0,
stages: [
{ duration: '5m', target: 50 },
{ duration: '30m', target: 50 },
{ duration: '5m', target: 0 }
],
exec: 'loggedInJourney'
},
// API clients (10% of traffic)
api_clients: {
executor: 'constant-arrival-rate',
rate: 10,
timeUnit: '1s',
duration: '40m',
preAllocatedVUs: 20,
exec: 'apiClient'
}
},
thresholds: {
http_req_duration: ['p(95)<800'],
http_req_failed: ['rate<0.01'],
}
}
const BASE = __ENV.BASE_URL || 'https://staging.example.com'
// Scenario: anonymous browser
export function anonymousBrowse() {
// Landing → catalog → product → exit
http.get(`${BASE}/`)
sleep(randomIntBetween(1, 4))
const category = randomItem(['electronics', 'clothing', 'books', 'sports'])
http.get(`${BASE}/api/products?category=${category}&limit=20`)
sleep(randomIntBetween(2, 8))
// 30% leave immediately, 70% view product
if (Math.random() > 0.3) {
const productId = randomIntBetween(1, 500)
http.get(`${BASE}/api/products/${productId}`)
sleep(randomIntBetween(3, 15))
}
// 20% perform a search
if (Math.random() < 0.2) {
const term = randomItem(searchTerms)
http.get(`${BASE}/api/search?q=${encodeURIComponent(term)}`)
sleep(randomIntBetween(1, 5))
}
}
// Scenario: logged-in user
export function loggedInJourney() {
const user = randomItem(users)
const headers = {
'Authorization': `Bearer ${user.token}`,
'Content-Type': 'application/json'
}
// Profile
http.get(`${BASE}/api/me`, { headers })
sleep(randomIntBetween(1, 3))
// Browse products
for (let i = 0; i < randomIntBetween(2, 8); i++) {
const productId = randomIntBetween(1, 500)
http.get(`${BASE}/api/products/${productId}`, { headers })
sleep(randomIntBetween(2, 10))
}
// 40% add to cart
if (Math.random() < 0.4) {
http.post(`${BASE}/api/cart/items`, JSON.stringify({
productId: randomIntBetween(1, 500),
quantity: randomIntBetween(1, 3)
}), { headers })
sleep(randomIntBetween(1, 3))
// 60% of those who added — checkout
if (Math.random() < 0.6) {
http.get(`${BASE}/api/cart`, { headers })
sleep(randomIntBetween(2, 5))
const checkout = http.post(`${BASE}/api/orders`, JSON.stringify({
paymentMethod: 'saved_card',
shippingAddressId: 1
}), { headers })
check(checkout, { 'order created': (r) => r.status === 201 })
}
}
}
// Scenario: API client (integration)
export function apiClient() {
const apiKey = __ENV.API_KEY
const headers = {
'X-API-Key': apiKey,
'Content-Type': 'application/json'
}
// Sync products
const r = http.get(`${BASE}/api/v1/products?since=${Date.now() - 3600000}`,
{ headers })
check(r, { 'api: 200': (r) => r.status === 200 })
}
A realistic scenario provides 1.5 times more accurate modeling compared to uniform load.
Pareto Distribution in k6
Real traffic: 20% of pages receive 80% of traffic. This is modeled in k6 with a function that generates IDs according to a power law:
// Pareto distribution ID generator
function paretoId(maxId, shape = 1.5) {
const u = Math.random()
return Math.ceil(maxId * Math.pow(1 - u, 1 / shape))
}
// Usage
const productId = paretoId(10000) // mostly IDs 1-200, rarely ID 9000+
Comparison: Synthetic vs Realistic Testing
| Characteristic | Synthetic Testing | Realistic Testing |
|---|---|---|
| Traffic pattern | Uniform, manually set | Reproduces real patterns (peaks, sessions) |
| User behavior | Same scenario for all VUs | Different scenarios (anonymous, logged-in, API) |
| User journey | Linear (home → product → cart) | Branching with probabilistic transitions |
| Bottleneck detection | Only throughput | Caching, slow endpoints, concurrency |
| Preparation time | Hours | Days (requires log analysis) |
How We Work
- Log analysis: Collect Nginx access logs or Google Analytics data, extract patterns (endpoints, statuses, hourly distribution).
- Scenario design: Determine user types (anonymous, logged-in, API), build probabilistic behavior models.
- k6 implementation: Write JavaScript scenarios with executors, random pauses, and branching.
- Test run: Execute test on staging environment, collect metrics (LCP, CLS, TTFB, errors).
- Analysis and report: Identify bottlenecks, compare with baseline, provide optimization recommendations.
| Stage | Duration | Result |
|---|---|---|
| Log analysis | 0.5–1 day | JSON profile of endpoints and distributions |
| Scenario design | 0.5–1 day | Probabilistic models for each user type |
| k6 implementation | 1–2 days | Working k6 scripts with executors |
| Test run | 1 day | Metrics, graphs, thresholds |
| Report and recommendations | 0.5 day | Document with analysis and optimization plan |
Example hourly load profile
Traffic is distributed unevenly: peak at 10-11 AM and 6-7 PM. Other times decline. We assign weights for each hour and generate k6 stages to match the real daily cycle.What's Included
- Scenario documentation describing behavior of each user type.
- k6 configurations (options, thresholds, threshold values).
- Test results report: load graphs, response time percentiles, errors.
- Performance optimization recommendations (database indexes, caching, async queues).
- Support during first run and result interpretation.
Company Metrics
With 5+ years of experience in performance testing and 200+ projects delivered, we guarantee all scenarios are verified on our test benches. Our clients report 15–30% improvement in response times after implementing our recommendations.
Pricing and Timelines
Development of a realistic load test scenario based on real traffic analysis takes 2 to 5 working days. Pricing starts from $2500 for a basic scenario and goes up to $7500 for complex multi-user tests. Typical savings: $15000 on early bug detection.
Order a realistic load testing scenario turnkey. Get a consultation from engineers.