Custom Gym Environment for RL Trading Strategy

Standard environments from Gymnasium (CartPole, LunarLander) are not suitable for trading. They ignore slippage, commissions, and market impact. The result: Sharpe 2 on backtest, but in reality—a deposit blowout. Imagine: you trained an agent on a standard environment, got Sharpe 2.5, run it on a live account—and within a month drawdown is 40%. Sound familiar? The problem is that standard environments don't account for market microstructure: slippage on execution, broker commissions (0.05–0.2% per trade), market impact of large orders (up to 5% of volume). The agent learns to trade on perfect data, but reality is different. We solve this by creating custom Gym environments that include all key factors: spreads, commissions, order book depth for HFT, and even risk management. This allows the agent to train in conditions as close to real as possible and deliver consistent results on live markets. We will assess your project in one day—get in touch with us.

Why a custom Gym environment is critical for a trading strategy?

Standard environments produce inflated results by 20–40%. For example, a strategy with limit orders on multiple exchanges will show excellent results in a standard environment, but in reality slippage of 0.2% and commission of 0.1% eat 30% of profit. The agent learns to trade incorrectly, treating noise as signal. In monetary terms, for a $100,000 portfolio, this means a loss of $30,000 per year—just from costs. A custom environment models these effects, and the agent learns to avoid them.

How we develop a custom environment

We build the environment based on Gymnasium with precise modeling of all trading aspects.

Class structure CustomTradingEnv

import gymnasium as gym
from gymnasium import spaces
import numpy as np

class CustomTradingEnv(gym.Env):
    metadata = {'render_modes': ['human', 'rgb_array']}

    def __init__(self, df, config):
        super().__init__()
        self.df = df
        self.config = config

        # observation space: OHLCV + indicators + portfolio state
        n_features = config['n_features']
        self.observation_space = spaces.Box(
            low=-np.inf, high=np.inf,
            shape=(n_features,), dtype=np.float32
        )

        # action space: position size [-1, 1] per asset
        n_assets = config['n_assets']
        self.action_space = spaces.Box(
            low=-1.0, high=1.0,
            shape=(n_assets,), dtype=np.float32
        )

        self._reset_portfolio()

    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        self._reset_portfolio()
        self.current_step = self.config['window_size']
        obs = self._get_observation()
        return obs, {}

    def step(self, action):
        # 1. Execute action (with realistic modeling)
        executed_action = self._execute_order(action)

        # 2. Move to next step
        self.current_step += 1

        # 3. Update portfolio with new prices
        self._update_portfolio()

        # 4. Calculate reward
        reward = self._compute_reward()

        # 5. Observation
        obs = self._get_observation()

        # 6. Termination condition
        terminated = self.current_step >= len(self.df) - 1
        truncated = self.portfolio_value < self.config['min_capital']

        info = {
            'portfolio_value': self.portfolio_value,
            'positions': self.positions.copy(),
            'total_trades': self.total_trades
        }

        return obs, reward, terminated, truncated, info

Realistic order execution

Large orders move the market—ignoring this gives false signals. We model slippage and commissions:

def _execute_order(self, target_weights):
    current_prices = self.df.iloc[self.current_step][['open', 'high', 'low', 'close']]

    # slippage: depends on order size and spread
    order_size = np.abs(target_weights - self.current_weights)
    slippage = order_size * self.config['slippage_factor']
    execution_price = current_prices['open'] * (1 + slippage)

    # commission
    trade_value = np.abs(order_size) * execution_price * self.portfolio_value
    commission = trade_value * self.config['commission_rate']

    self.portfolio_value -= commission.sum()
    self.current_weights = target_weights.copy()
    self.total_trades += (order_size > 0.01).sum()

    return target_weights

For HFT strategies, we add order book simulation:

class LOBSimulator:
    """Level-2 order book simulation"""
    def __init__(self, spread_bps=5, depth_levels=10):
        self.spread_bps = spread_bps
        self.depth_levels = depth_levels

    def get_fill_price(self, mid_price, order_size_usd):
        # fill price depends on order book depth
        spread = mid_price * self.spread_bps / 10000
        market_impact = np.sqrt(order_size_usd / 1e6) * spread
        return mid_price + spread/2 + market_impact

How to choose a reward function for a trading strategy?

Choosing the reward is the most important part of a custom environment. We offer several options:

def _compute_reward(self):
    daily_return = (self.portfolio_value / self.prev_portfolio_value) - 1

    # option 1: simple return
    reward = daily_return

    # option 2: Sharpe-adjusted (rolling window)
    self.returns_history.append(daily_return)
    if len(self.returns_history) >= 20:
        sharpe = np.mean(self.returns_history[-20:]) / (np.std(self.returns_history[-20:]) + 1e-8)
        reward = daily_return * (1 + sharpe)

    # option 3: penalized drawdown
    current_dd = (self.peak_value - self.portfolio_value) / self.peak_value
    reward = daily_return - self.config['dd_penalty'] * current_dd

    # penalty for excessive trading
    reward -= self.config['turnover_penalty'] * self.daily_turnover

    return float(reward)

The turnover penalty is especially important for HFT—it teaches the agent not to trade when there's no edge.

Observation Engineering

def _get_observation(self):
    window = self.df.iloc[self.current_step - self.window_size:self.current_step]

    features = []
    # price returns (normalized)
    returns = window['close'].pct_change().fillna(0).values
    features.extend(returns[-self.window_size:])

    # technical indicators
    features.extend([
        window['rsi'].iloc[-1] / 100,
        window['macd_norm'].iloc[-1],
        window['bb_position'].iloc[-1]  # (price - lower) / (upper - lower)
    ])

    # portfolio state
    features.extend(self.current_weights)
    features.append(self.portfolio_value / self.initial_capital - 1)

    return np.array(features, dtype=np.float32)

We add normalized features—this accelerates agent convergence.

Comparison of standard vs custom environment

Process and timelines

Total time: from 3–5 days for a single asset to 3–4 weeks for multi-asset with order book. Cost is calculated individually—leave a request for a consultation, and we will assess the project.

What you get as a result

• Source code of the custom environment in Python (Gymnasium) with support for your data.
• Configuration files for single-asset, multi-asset, and HFT modes.
• Documentation on API, reward functions, and execution modeling.
• Jupyter notebook with agent training and result visualization.
• Support for integration into the pipeline (MLflow, Weights & Biases).
• Guaranteed to pass gymnasium.utils.env_checker checks.

Get in touch with us to discuss your strategy—and receive a consultation within a day.

Typical problems when developing an environment

• Look-ahead bias: normalization on the entire dataset. We check temporal splitting and prevent future data leakage.
• Ignoring commissions in reward: the agent learns to trade frequently, reducing returns by 20–30%.
• Too simple reward (only return): leads to high volatility and drawdowns. We use combined rewards with penalties.

For environment verification, we use gymnasium.utils.env_checker and sanity checks:

• Random policy loses capital due to commissions.
• Buy & Hold is reproduced with constant weights.
• No look-ahead: observation does not contain future prices.

Our engineers with certifications in machine learning and trading guarantee quality. Get a consultation on your strategy—get in touch with us.

Reinforcement Learning: PPO, SAC, DQN and Industrial Applications

We see projects every day that fail not because of a weak algorithm, but because of incorrect rewards. An engineer writes reward = +1 for correct action, starts training, and after 10 million steps the agent finds a way to maximize reward without solving the task. This is reward hacking — a systemic pain of industrial RL. Our experience shows: proper reward accounts for 70% of success.

Why is RL harder than supervised learning?

In supervised learning, there is a dataset with correct answers. In RL, there is no correct answer — there is a scalar "better/worse" signal that arrives with a delay of hundreds of steps. The agent explores the space and finds a strategy on its own.

Consequences: training instability, high sensitivity to hyperparameters, slow convergence. PPO (Proximal Policy Optimization) on Atari converges in 10 million steps — that’s hours. On robotic tasks with real physics — days or weeks in simulation.

Algorithm selection by task:

How to tune PPO and avoid common problems?

PPO is the workhorse of RL. The main idea: limit policy updates via ratio clipping clip_range=0.2. This provides stability compared to vanilla policy gradient. But without proper tuning, the agent does not converge.

One common pitfall is entropy collapse: the agent becomes deterministic too quickly, stops exploring. Symptom — entropy coefficient drops to zero. Cure — ent_coef=0.01–0.05 and do not lower below 0.001. Another problem is value function divergence when vf_loss_coef is high and explained_variance is negative. We recommend vf_coef=0.5 and gradient clipping max_grad_norm=0.5.

Incorrect n_steps also breaks training. n_steps=2048 is Stable-Baselines3 default. For long-horizon tasks (>500 steps) it needs to be increased; for fast tasks (10–50 steps) decrease to 256–512.

For quick start, use stable-baselines3 + sb3-contrib. For research and custom algorithms — tianshou or CleanRL.

SAC for continuous control

SAC (Soft Actor-Critic) adds entropy maximization to the objective — the agent learns to be both efficient and diverse. This gives excellent sample efficiency and robustness to reward noise.

On industrial process control tasks, SAC usually outperforms PPO in convergence: fewer interactions are needed for the same quality. The key parameter is target_entropy. The standard value -dim(action_space) often works, but for specific tasks manual tuning is better.

How to transfer a trained agent to a real device?

Training RL on a real robot is expensive and dangerous. Standard approach: train in simulation → transfer to real hardware. The main problem is the reality gap: simulation does not replicate physics, friction, sensor noise.

The primary tool is domain randomization. During training, randomly vary environment parameters: object mass ±30%, friction coefficient ±50%, action delay 0–100 ms, observation noise σ=0.01–0.1. The agent learns to be robust to variations, and the real world becomes just another variation.

Comparison of popular simulators:

RLHF: training LLMs from human feedback

RLHF became the standard after InstructGPT. Classic scheme: supervised fine-tuning → reward model → PPO.

Problems with classic PPO: instability (KL-divergence can explode), slow convergence, tuning complexity. Hence popular alternatives:

• DPO — bypasses reward model, learns from preference pairs. Simpler, more stable, but less flexible.
• GRPO — used in DeepSeek-R1, good for reasoning tasks.
• ORPO — combines SFT and alignment into one stage.

The trl library from Hugging Face is the standard. Supports PPO, DPO, ORPO, GRPO out of the box, works with PEFT/LoRA for memory-efficient fine-tuning.

"Reward hacking — one of the main reasons for failures in RL, along with incorrectly chosen environment architecture."

What is included in the work

• Architectural solution and justification of algorithm selection
• Development and documentation of the reward function
• Creating a simulator or configuring an existing one
• Training, hyperparameter sweep (Optuna / Ray Tune)
• Transfer to real hardware or integration into product
• Documentation, access to code and simulators
• Team training and 3-month support after deployment

Work process

1. Task audit — define goals, resources, constraints.
2. Reward engineering — formalize desired behavior, check for reward hacking.
3. Environment and algorithm selection — baseline, first runs.
4. Systematic hyperparameter sweep — use Optuna.
5. Training in simulation with domain randomization.
6. Testing on real equipment (if necessary).
7. Deployment, monitoring, support.

Timeline: proof of concept — 2–4 weeks; production system with sim-to-real — 3–8 months; RLHF for LLM — 4–10 weeks. Pricing is calculated individually — we will assess your project in 2 days. Contact us for a consultation.

Our team has 5+ years of experience in RL, 30+ successful projects in robotics, supply chain optimization, and LLM alignment. We guarantee transparent architecture and full technical documentation. Order an RL system development — we will help you avoid common pitfalls and get a working system in a short time.