MLOps Infrastructure for Trading Models: Development and Automation
Situation: an algorithmic trader spends 8 hours manually extracting data, training a model, and deploying an inference server. One config error — a lost day. With 50 trades per day, each minute of deployment delay costs $500, and with an average trade ticket of $5000, losing one is a tangible blow to P&L. This is not hypothetical — we have encountered it many times. The MLOps infrastructure we developed for trading models automates the entire pipeline: from receiving market ticks to issuing trading signals. On one project, we reduced rollout time from 4 hours to 15 minutes — 16 times faster than the manual process. This approach allows teams to focus on strategy development rather than infrastructure dances.
Why MLOps Is Critical for Trading Algorithms
In trading, every millisecond of deployment delay or inference server downtime costs real money. MLOps infrastructure solves three main problems:
- Reproducibility: a model trained today must produce the same result tomorrow. Without data and code versioning, this is impossible.
- Speed: manual deployment takes hours, automated takes minutes. We cut rollout time from 4 hours to 15 minutes on one project (94% reduction).
- Monitoring: feature drift or metric degradation go unnoticed without an alerting system. Our Grafana dashboards show accuracy (target threshold 95%), latency, and volume in real time.
In practice, this means the difference between a profitable trade and a loss. For example, in high-frequency trading, a 100 ms delay can cost $10,000 per month. That's why we use proven tools and best practices.
How We Build the MLOps Pipeline
We use a proven stack: ClickHouse for tick data, PostgreSQL for trades, S3/MinIO for raw data. Orchestration — Prefect, model versioning — MLflow, data versioning — DVC. Inference on Kubernetes with autoscaling. All steps are described in the MLflow Documentation.
Experiments with MLflow
import mlflow
import mlflow.sklearn
import mlflow.pytorch
from mlflow.models.signature import infer_signature
def train_with_mlflow_tracking(experiment_name, config, X_train, y_train, X_val, y_val, X_test, y_test):
mlflow.set_experiment(experiment_name)
with mlflow.start_run(run_name=f"{config['model_type']}_{config['version']}"):
mlflow.log_params({
'model_type': config['model_type'],
'n_features': X_train.shape[1],
'train_size': len(X_train),
'val_size': len(X_val),
**config.get('hyperparams', {})
})
model = train_model(config, X_train, y_train, X_val, y_val)
val_metrics = evaluate_model(model, X_val, y_val)
test_metrics = evaluate_model(model, X_test, y_test)
mlflow.log_metrics({f'val_{k}': v for k, v in val_metrics.items()})
mlflow.log_metrics({f'test_{k}': v for k, v in test_metrics.items()})
signature = infer_signature(X_train[:10], model.predict_proba(X_train[:10]))
mlflow.sklearn.log_model(model, 'model', signature=signature, registered_model_name=f"crypto_{config['symbol']}_predictor")
import matplotlib.pyplot as plt
fig = plot_feature_importance(model, X_train.columns)
mlflow.log_figure(fig, 'feature_importance.png')
run_id = mlflow.active_run().info.run_id
return run_id, test_metrics
Data Versioning with DVC
# dvc.yaml — pipeline definition
stages:
fetch_data:
cmd: python src/data/fetch_ohlcv.py --symbol BTC --days 730
deps:
- src/data/fetch_ohlcv.py
outs:
- data/raw/btc_ohlcv.parquet
feature_engineering:
cmd: python src/features/engineer.py
deps:
- src/features/engineer.py
- data/raw/btc_ohlcv.parquet
outs:
- data/features/btc_features.parquet
params:
- params.yaml:
- feature_engineering
train:
cmd: python src/train.py
deps:
- src/train.py
- data/features/btc_features.parquet
outs:
- models/btc_predictor.pkl
metrics:
- metrics/train_metrics.json
params:
- params.yaml:
- training
CI/CD for ML with GitHub Actions
# .github/workflows/ml_pipeline.yml
name: ML Training Pipeline
on:
schedule:
- cron: '0 1 * * 0'
workflow_dispatch:
inputs:
symbol:
description: 'Trading symbol'
default: 'BTC'
jobs:
train:
runs-on: [self-hosted, gpu]
steps:
- uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install dependencies
run: pip install -r requirements.txt
- name: Pull data with DVC
run: dvc pull data/
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_KEY }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET }}
- name: Run training pipeline
run: dvc repro
env:
MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_URI }}
- name: Validate model
run: python src/validate_model.py --min-accuracy 0.54 --min-sharpe 1.0
- name: Deploy to production
if: success()
run: python src/deploy_model.py
env:
TRADING_API_KEY: ${{ secrets.TRADING_API }}
Deploy on Kubernetes
# k8s/ml-inference-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: crypto-ml-inference
spec:
replicas: 3
selector:
matchLabels:
app: ml-inference
template:
spec:
containers:
- name: inference
image: crypto-ml-inference:latest
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
env:
- name: MLFLOW_TRACKING_URI
valueFrom:
secretKeyRef:
name: ml-secrets
key: mlflow_uri
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 8000
initialDelaySeconds: 10
Feature Store: Single Registry for Features
A feature store is a single point of access for features used in training and inference. We use Feast: define entities and feature views, online serving returns up-to-date values in milliseconds. Without a feature store, features are recalculated in each pipeline, leading to train/serve skew and errors. In practice, this saves up to 40% of time when developing new features.
Tool Comparison: MLflow, DVC, Prefect
The choice of orchestrator depends on scale. MLflow is ideal for experiments — it logs hyperparameters, metrics, and models with minimal code. DVC complements it with data versioning on top of Git, convenient for small teams. Prefect handles complex DAGs with retries and monitoring. In trading, where step order is critical (first fetch_data, then train), Prefect is more reliable than Airflow due to built-in retry policies. We do not use paid tools — the entire stack is open-source.
| Criterion | MLflow | DVC | Prefect |
|---|---|---|---|
| Focus | Experiments | Data | Orchestration |
| Storage | MLflow Tracking Server | Git + S3 | Prefect Server / Cloud |
| Language | Python, R, Java | Python | Python |
| When to choose | 1-3 models | 1-5 models | 5+ models |
| Retry | No | No | Built-in |
How MLOps Reduces Deployment Time
In one project, we automated the pipeline for a crypto fund. Previously, a data scientist spent 4 hours preparing a release: data extraction, training, metric validation, manual deployment. After implementing MLOps, the same cycle takes 15 minutes. All steps are fixed in a DVC pipeline and run with a single command. CI/CD validates model quality (minimum accuracy 0.54, Sharpe ratio 1.0) and, if successful, automatically rolls out a new container to Kubernetes. Inference server downtime dropped from 2 hours to 30 seconds, uptime reached 99.9%. Savings from downtime losses amounted to $5000 per month.
What's Included
- Audit of the current pipeline and infrastructure
- MLOps architecture design
- Deployment and configuration of MLflow, DVC, Prefect
- CI/CD pipeline (GitHub Actions / GitLab CI)
- Kubernetes manifests for inference
- Monitoring (Prometheus, Grafana, alerts)
- Documentation and team training
- 1 month of support after launch
Work Stages
- Analysis — review your stack, latency and model update frequency requirements. Define SLAs.
- Design — outline architecture, select tools, create a proof of concept on a small model.
- Implementation — set up infrastructure, write pipelines, integrate CI/CD.
- Testing — load testing, reproducibility checks, inference stress test.
- Deploy — deploy to production, configure monitoring.
- Handover — documentation, training, Q&A session.
Estimated Timelines
| Scope | Timeline |
|---|---|
| Basic MLOps (1 model, versioning, CI/CD) | 4-6 weeks |
| MLOps with real-time features, multiple models | 8-12 weeks |
| Full cycle + monitoring + support | 10-14 weeks |
3 Common Mistakes When Implementing MLOps
- Ignoring data versioning — the model trains on different slices, results are unpredictable. DVC solves this.
- No drift alerts — when the feature distribution changes, model accuracy drops. We deploy a PSI counter in Prometheus.
- One binary for all models — different models require different environments. We use Docker with tagging.
Experience and expertise: we have completed 50+ projects in fintech and crypto. Certified AWS and Kubernetes engineers. Contact us for a consultation on your project. Order turnkey MLOps infrastructure development. Get a consultation — we'll explain how to adapt best practices to your stack.