Developing RL Trading Agents Based on TD3

Overestimation of the Q-function is a systemic error in DDPG that breaks stable learning in trading tasks. The TD3 algorithm (Twin Delayed Deep Deterministic Policy Gradient, proposed by Scott Fujimoto in Addressing Function Approximation Error in Actor-Critic Methods Addressing Function Approximation Error in Actor-Critic Methods) solves this problem with three mechanisms: twin critics, delayed policy updates, and target smoothing. We apply TD3 to build RL agent trading systems with continuous position sizing, ensuring stable training and reproducible results on real markets. This approach applies reinforcement learning to finance, delivering robust trading strategies.

In one project, switching from DDPG to TD3 raised the Sharpe ratio from 0.8 to 1.6, demonstrating that TD3 is 2x better than DDPG in Sharpe ratio. The key factor was suppressing the overestimation bias, which in DDPG led to overtrading and frequent drawdowns. The client reduced annual trading costs by $200,000 after implementing TD3.

Problems Solved by TD3

Overestimation bias. Standard DDPG inflates value estimates, and the policy optimizes toward unrealistic targets. TD3 uses two critics with target = min(Q₁, Q₂), providing a conservative approximation.

Unstable training. Fast policy updates relative to critics cause oscillation. TD3 updates the actor every d steps (d=2), allowing the Q-functions to stabilize.

Hyperactive trading. Without regularization, the agent makes excessive trades. Adding target policy smoothing with Gaussian noise (σ=0.2) and a transaction cost penalty (0.1% per trade) solves the problem.

How TD3 Solves Overestimation

Two independent critics Q₁ and Q₂ act as mutual verification. The target value:

y = r + γ · min(Q₁_target(s', π(s')), Q₂_target(s', π(s')))

This estimate is systematically lower than the true value but never inflated. In practice, this reduces update variance and improves final returns.

TD3 vs SAC: Comparison

TD3 uses a deterministic policy, SAC a stochastic one. TD3 provides reproducible signals, which is critical for live trading. SAC performs better under high uncertainty, but for trending markets, TD3 shows a more stable Sharpe ratio. Development of a TD3 trading agent typically costs between $15,000 and $30,000 depending on complexity.

Architecture for Trading

class TD3Actor(nn.Module):
    def __init__(self, state_dim, action_dim, max_action):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(state_dim, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
            nn.Linear(256, action_dim), nn.Tanh()
        )
        self.max_action = max_action

    def forward(self, state):
        return self.net(state) * self.max_action

class TD3Critic(nn.Module):
    def __init__(self, state_dim, action_dim):
        super().__init__()
        # Q1
        self.q1 = nn.Sequential(
            nn.Linear(state_dim + action_dim, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
            nn.Linear(256, 1)
        )
        # Q2
        self.q2 = nn.Sequential(
            nn.Linear(state_dim + action_dim, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
            nn.Linear(256, 1)
        )

    def forward(self, state, action):
        sa = torch.cat([state, action], dim=1)
        return self.q1(sa), self.q2(sa)

    def q1_forward(self, state, action):
        sa = torch.cat([state, action], dim=1)
        return self.q1(sa)

Continuous Position Sizing and Reward

Action space: [-1, 1] for each asset, with budget constraint Σ|wᵢ| ≤ 1. Reward includes a Sharpe-like metric and a penalty for position changes:

def reward_fn(returns_series):
    if len(returns_series) < 20:
        return returns_series[-1]
    mean_r = np.mean(returns_series[-20:])
    std_r = np.std(returns_series[-20:]) + 1e-8
    sharpe = mean_r / std_r
    return sharpe * returns_series[-1]

position_change = np.abs(new_position - old_position)
transaction_cost = position_change * 0.001
reward -= transaction_cost

Importantly, adding the transaction cost penalty (0.1% per trade) significantly reduces the number of trades and improves net returns. In one project, this cut trade frequency by 40% while preserving overall profit. For example, one client reduced annual trading costs by $200,000 after implementing TD3.

Hyperparameter Guidelines

For multi-asset scenarios, the action space is normalized so that the sum of absolute weights does not exceed 1. This can be implemented via a softmax-like transformation or projection onto a simplex.

Why Choose TD3 for Your Trading Project?

TD3 ensures deterministic actions for the same state, which is critical for backtesting and live trading. You get a reproducible strategy that can be tested on historical data without stochastic fluctuations. Additionally, the target smoothing mechanism makes the agent robust to market data noise. The agent is integrated with broker APIs for live trading, providing a complete solution that applies reinforcement learning to finance.

What's Included in the Work?

• Full implementation of the TD3 agent (actor/critic networks, buffer, training)
• Data collection pipeline and simulation environment
• Hyperparameter tuning and cross-validation
• Integration with broker API (Interactive Brokers, Alpaca, or yours)
• Reproducibility and modification documentation
• Team training (1 session) and 3-month warranty support

End-to-End Development Process

1. Analysis — historical data, metrics, benchmarks
2. Design — state/action/reward, simulation environment
3. Training — hyperparameter tuning, exploration decay, monitoring
4. Testing — out-of-sample backtest, stress test on crisis periods
5. Deployment — broker API integration, monitoring and alert setup

Our team's experience in RL trading spans over 5 years, with 10+ projects delivered for funds and private traders. Contact us for a preliminary consultation — we will evaluate your project within one business day. Order a consultation on RL trading to discuss the possibilities of implementing TD3 in your strategy.

Reinforcement Learning: PPO, SAC, DQN and Industrial Applications

We see projects every day that fail not because of a weak algorithm, but because of incorrect rewards. An engineer writes reward = +1 for correct action, starts training, and after 10 million steps the agent finds a way to maximize reward without solving the task. This is reward hacking — a systemic pain of industrial RL. Our experience shows: proper reward accounts for 70% of success.

Why is RL harder than supervised learning?

In supervised learning, there is a dataset with correct answers. In RL, there is no correct answer — there is a scalar "better/worse" signal that arrives with a delay of hundreds of steps. The agent explores the space and finds a strategy on its own.

Consequences: training instability, high sensitivity to hyperparameters, slow convergence. PPO (Proximal Policy Optimization) on Atari converges in 10 million steps — that’s hours. On robotic tasks with real physics — days or weeks in simulation.

Algorithm selection by task:

How to tune PPO and avoid common problems?

PPO is the workhorse of RL. The main idea: limit policy updates via ratio clipping clip_range=0.2. This provides stability compared to vanilla policy gradient. But without proper tuning, the agent does not converge.

One common pitfall is entropy collapse: the agent becomes deterministic too quickly, stops exploring. Symptom — entropy coefficient drops to zero. Cure — ent_coef=0.01–0.05 and do not lower below 0.001. Another problem is value function divergence when vf_loss_coef is high and explained_variance is negative. We recommend vf_coef=0.5 and gradient clipping max_grad_norm=0.5.

Incorrect n_steps also breaks training. n_steps=2048 is Stable-Baselines3 default. For long-horizon tasks (>500 steps) it needs to be increased; for fast tasks (10–50 steps) decrease to 256–512.

For quick start, use stable-baselines3 + sb3-contrib. For research and custom algorithms — tianshou or CleanRL.

SAC for continuous control

SAC (Soft Actor-Critic) adds entropy maximization to the objective — the agent learns to be both efficient and diverse. This gives excellent sample efficiency and robustness to reward noise.

On industrial process control tasks, SAC usually outperforms PPO in convergence: fewer interactions are needed for the same quality. The key parameter is target_entropy. The standard value -dim(action_space) often works, but for specific tasks manual tuning is better.

How to transfer a trained agent to a real device?

Training RL on a real robot is expensive and dangerous. Standard approach: train in simulation → transfer to real hardware. The main problem is the reality gap: simulation does not replicate physics, friction, sensor noise.

The primary tool is domain randomization. During training, randomly vary environment parameters: object mass ±30%, friction coefficient ±50%, action delay 0–100 ms, observation noise σ=0.01–0.1. The agent learns to be robust to variations, and the real world becomes just another variation.

Comparison of popular simulators:

RLHF: training LLMs from human feedback

RLHF became the standard after InstructGPT. Classic scheme: supervised fine-tuning → reward model → PPO.

Problems with classic PPO: instability (KL-divergence can explode), slow convergence, tuning complexity. Hence popular alternatives:

• DPO — bypasses reward model, learns from preference pairs. Simpler, more stable, but less flexible.
• GRPO — used in DeepSeek-R1, good for reasoning tasks.
• ORPO — combines SFT and alignment into one stage.

The trl library from Hugging Face is the standard. Supports PPO, DPO, ORPO, GRPO out of the box, works with PEFT/LoRA for memory-efficient fine-tuning.

"Reward hacking — one of the main reasons for failures in RL, along with incorrectly chosen environment architecture."

What is included in the work

• Architectural solution and justification of algorithm selection
• Development and documentation of the reward function
• Creating a simulator or configuring an existing one
• Training, hyperparameter sweep (Optuna / Ray Tune)
• Transfer to real hardware or integration into product
• Documentation, access to code and simulators
• Team training and 3-month support after deployment

Work process

1. Task audit — define goals, resources, constraints.
2. Reward engineering — formalize desired behavior, check for reward hacking.
3. Environment and algorithm selection — baseline, first runs.
4. Systematic hyperparameter sweep — use Optuna.
5. Training in simulation with domain randomization.
6. Testing on real equipment (if necessary).
7. Deployment, monitoring, support.

Timeline: proof of concept — 2–4 weeks; production system with sim-to-real — 3–8 months; RLHF for LLM — 4–10 weeks. Pricing is calculated individually — we will assess your project in 2 days. Contact us for a consultation.

Our team has 5+ years of experience in RL, 30+ successful projects in robotics, supply chain optimization, and LLM alignment. We guarantee transparent architecture and full technical documentation. Order an RL system development — we will help you avoid common pitfalls and get a working system in a short time.