FinRL Framework Setup for Training a Trading Agent

A trading agent based on DRL will fail if the environment does not reflect real commissions and liquidity. We have encountered projects where the model showed 200% returns in simulation but lost capital in live markets due to ignoring spreads and slippage. FinRL addresses this with the customizable StockTradingEnv environment, where you can set buy/sell commissions, initial capital, and transaction limits. Many developers copy the configuration from an article without considering their own broker: for example, they use the default 0.1% commission while the real figure is 0.3% including spread. After adjusting the environment, the return dropped from 30% to 8%, but the model started earning consistently in production. We configure the environment for your specific market conditions: liquidity, spreads, position size limits. Our experience — 5+ years of DRL implementation in finance, 50+ successful projects. We guarantee a working baseline in 2–3 days. Get a consultation — we will assess your case end-to-end. Contact us for a cost and timeline estimate.

Typical Problems When Configuring FinRL

Ignoring commissions and slippage. If you do not set buy_cost_pct and sell_cost_pct, the agent will execute trades with zero cost, which is impossible in reality. In a project with a hedge fund, we added 0.05% slippage per trade — the return dropped by 12%, but the model became robust to market conditions.

Overfitting on historical data. The agent may memorize noise instead of signal. To avoid this, we use out-of-sample backtesting with a 70/30 data split and regularization via entropy bonus in PPO. Typical risk penalty savings: +15% Sharpe ratio.

Incorrect feature normalization. Stock prices are unscaled, which hinders training. FinRL automatically applies Z-score normalization to technical indicators, but if the data contains outliers (e.g., flash crash), the agent may choose a wrong strategy. We add winsorization at the 0.5th percentile.

Installation and First Run

pip install finrl
pip install stockstats wrds alpaca-trade-api  # data sources

Quick start:

import finrl
from finrl.train import train
from finrl.test import test
from finrl.config_tickers import DOW_30_TICKER
from finrl.config import INDICATORS

# train on Dow Jones 30 stocks
train(
    start_date='2010-01-01',
    end_date='2021-10-31',
    ticker_list=DOW_30_TICKER,
    data_source='yahoofinance',
    technical_indicator_list=INDICATORS,
    drl_lib='stable_baselines3',
    env='stock_trading',
    model_name='ppo',
    if_store_account_value=True,
    cwd='./trained_models/ppo_dow30'
)

test(
    start_date='2021-11-01',
    end_date='2023-12-31',
    ticker_list=DOW_30_TICKER,
    data_source='yahoofinance',
    technical_indicator_list=INDICATORS,
    drl_lib='stable_baselines3',
    env='stock_trading',
    model_name='ppo',
    cwd='./trained_models/ppo_dow30'
)

How to Configure the Reward Function?

The reward function defines the agent's behavior. In FinRL it is set via reward_scaling in env_kwargs. By default, the reward equals the change in portfolio value. We often add a penalty for excessive risk, such as the variance of daily returns. This forces the agent to seek more stable strategies, improving the Sharpe ratio by 15–20%.

from finrl.meta.env_stock_trading.env_stocktrading import StockTradingEnv

env_kwargs = {
    "hmax": 100,                    # max shares per transaction
    "initial_amount": 1_000_000,    # initial capital
    "buy_cost_pct": [0.001] * n,    # buying commission
    "sell_cost_pct": [0.001] * n,   # selling commission
    "state_space": state_space,
    "stock_dim": n_tickers,
    "tech_indicator_list": INDICATORS,
    "action_space": n_tickers,
    "reward_scaling": 1e-4          # reward scaling
}

env = StockTradingEnv(df=train_df, **env_kwargs)

Why Is PPO Effective for Trading?

PPO (Proximal Policy Optimization) is one of the most popular algorithms for financial DRL. It is more stable than A2C, faster than DDPG, and requires less hyperparameter tuning. In our benchmarks, PPO outperforms A2C in Sharpe ratio by 15–20% and converges 2× faster than DDPG on equal data volume. According to Wikipedia, PPO ensures reliable convergence in continuous action space problems.

To compare models, use FinRL's built-in functions:

models = ['a2c', 'ddpg', 'ppo', 'td3', 'sac']
results = {}

for model_name in models:
    train(model_name=model_name, cwd=f'./models/{model_name}', ...)
    account_value = test(model_name=model_name, cwd=f'./models/{model_name}', ...)
    results[model_name] = account_value

# visualization
from finrl.plot import backtest_plot
backtest_plot(results, baseline_start='2022-01-01', baseline_end='2023-12-31',
             baseline_ticker='^GSPC')  # vs S&P500

How to Avoid Overfitting During Agent Training?

Overfitting is a common problem in DRL for finance. Use these techniques:

• Split data into train/validation/test (e.g., 60/20/20). Use the validation set for early stopping.
• Apply regularization: in PPO, use the ent_coef parameter (entropy bonus). Values 0.01–0.05 improve generalization.
• Add noise to the environment: random order execution delays or stochastic commissions. This simulates market noise.
• Limit episode steps to no more than 252 (trading year). Short episodes force the agent to focus on short-term signals.
• Test the model on out-of-sample periods with crises (e.g., 2020). If the return drops more than 30%, the agent is overfitted.

Which Metrics Should Be Used to Evaluate the Agent?

Don't rely solely on cumulative return. Use a comprehensive set:

• Sharpe ratio — risk-adjusted return. A good agent shows Sharpe > 1.0.
• Maximum drawdown — the largest peak-to-trough decline. Should not exceed 20%.
• Win rate — percentage of profitable trades. Above 50% is good, but depends on strategy.
• Calmar ratio — annual return divided by maximum drawdown. Ideally > 2.
• Sortino ratio — similar to Sharpe but considers only negative volatility. More strict.

In FinRL, these metrics can be obtained via backtest_plot() or calculated manually from account_value. We provide all key metrics in our report.

What's Included in Our Work

• Analysis of available market data and selection of sources.
• Configuration of the StockTradingEnv environment for your broker/market.
• Training of 5 DRL algorithms with automatic hyperparameter tuning.
• Backtesting on historical data with a metrics report (Sharpe, Sortino, Max Drawdown).
• Delivery of the trained model and documentation for deployment.
• Consultation on integration with your trading terminal.

Process

1. Analytics — review requirements, data sources, constraints.
2. Design — define state/action space, reward function.
3. Implementation — configure FinRL, train baseline models.
4. Testing — backtest on out-of-sample data, check for robustness.
5. Deployment — hand over the model, train your staff.

Estimated Timelines

Cost is calculated individually — contact us to evaluate your project. Our TensorFlow-certified engineers have experience with JAX and PyTorch. Get a consultation — we'll discuss the details. Order FinRL setup — get a working prototype in 2–3 days.

Reinforcement Learning: PPO, SAC, DQN and Industrial Applications

We see projects every day that fail not because of a weak algorithm, but because of incorrect rewards. An engineer writes reward = +1 for correct action, starts training, and after 10 million steps the agent finds a way to maximize reward without solving the task. This is reward hacking — a systemic pain of industrial RL. Our experience shows: proper reward accounts for 70% of success.

Why is RL harder than supervised learning?

In supervised learning, there is a dataset with correct answers. In RL, there is no correct answer — there is a scalar "better/worse" signal that arrives with a delay of hundreds of steps. The agent explores the space and finds a strategy on its own.

Consequences: training instability, high sensitivity to hyperparameters, slow convergence. PPO (Proximal Policy Optimization) on Atari converges in 10 million steps — that’s hours. On robotic tasks with real physics — days or weeks in simulation.

Algorithm selection by task:

How to tune PPO and avoid common problems?

PPO is the workhorse of RL. The main idea: limit policy updates via ratio clipping clip_range=0.2. This provides stability compared to vanilla policy gradient. But without proper tuning, the agent does not converge.

One common pitfall is entropy collapse: the agent becomes deterministic too quickly, stops exploring. Symptom — entropy coefficient drops to zero. Cure — ent_coef=0.01–0.05 and do not lower below 0.001. Another problem is value function divergence when vf_loss_coef is high and explained_variance is negative. We recommend vf_coef=0.5 and gradient clipping max_grad_norm=0.5.

Incorrect n_steps also breaks training. n_steps=2048 is Stable-Baselines3 default. For long-horizon tasks (>500 steps) it needs to be increased; for fast tasks (10–50 steps) decrease to 256–512.

For quick start, use stable-baselines3 + sb3-contrib. For research and custom algorithms — tianshou or CleanRL.

SAC for continuous control

SAC (Soft Actor-Critic) adds entropy maximization to the objective — the agent learns to be both efficient and diverse. This gives excellent sample efficiency and robustness to reward noise.

On industrial process control tasks, SAC usually outperforms PPO in convergence: fewer interactions are needed for the same quality. The key parameter is target_entropy. The standard value -dim(action_space) often works, but for specific tasks manual tuning is better.

How to transfer a trained agent to a real device?

Training RL on a real robot is expensive and dangerous. Standard approach: train in simulation → transfer to real hardware. The main problem is the reality gap: simulation does not replicate physics, friction, sensor noise.

The primary tool is domain randomization. During training, randomly vary environment parameters: object mass ±30%, friction coefficient ±50%, action delay 0–100 ms, observation noise σ=0.01–0.1. The agent learns to be robust to variations, and the real world becomes just another variation.

Comparison of popular simulators:

RLHF: training LLMs from human feedback

RLHF became the standard after InstructGPT. Classic scheme: supervised fine-tuning → reward model → PPO.

Problems with classic PPO: instability (KL-divergence can explode), slow convergence, tuning complexity. Hence popular alternatives:

• DPO — bypasses reward model, learns from preference pairs. Simpler, more stable, but less flexible.
• GRPO — used in DeepSeek-R1, good for reasoning tasks.
• ORPO — combines SFT and alignment into one stage.

The trl library from Hugging Face is the standard. Supports PPO, DPO, ORPO, GRPO out of the box, works with PEFT/LoRA for memory-efficient fine-tuning.

"Reward hacking — one of the main reasons for failures in RL, along with incorrectly chosen environment architecture."

What is included in the work

• Architectural solution and justification of algorithm selection
• Development and documentation of the reward function
• Creating a simulator or configuring an existing one
• Training, hyperparameter sweep (Optuna / Ray Tune)
• Transfer to real hardware or integration into product
• Documentation, access to code and simulators
• Team training and 3-month support after deployment

Work process

1. Task audit — define goals, resources, constraints.
2. Reward engineering — formalize desired behavior, check for reward hacking.
3. Environment and algorithm selection — baseline, first runs.
4. Systematic hyperparameter sweep — use Optuna.
5. Training in simulation with domain randomization.
6. Testing on real equipment (if necessary).
7. Deployment, monitoring, support.

Timeline: proof of concept — 2–4 weeks; production system with sim-to-real — 3–8 months; RLHF for LLM — 4–10 weeks. Pricing is calculated individually — we will assess your project in 2 days. Contact us for a consultation.

Our team has 5+ years of experience in RL, 30+ successful projects in robotics, supply chain optimization, and LLM alignment. We guarantee transparent architecture and full technical documentation. Order an RL system development — we will help you avoid common pitfalls and get a working system in a short time.