Industrial Robots and AGVs: Reduce Downtime by 30% with RL

From Magnetic Tape to RL: How We Transformed a Warehouse into a Dynamic System

Imagine: a 20,000 m² warehouse, a fleet of 30 AGVs, but every morning — a bottleneck. AGVs collide in narrow aisles, idle waiting, and magnetic strips prevent quick route changes when blocked. We removed the fixed layout and replaced it with dynamic routing via multi-agent reinforcement learning (MARL). This same technology allowed cobots on the assembly line to pick parts of any shape without manual programming.

Implementing RL is a practical tool, not an experiment. We have completed 10+ projects for warehouses and assembly lines. Contact us for a free audit of your production.

How RL Solves Production Problems?

Grasping Irregularly Shaped Objects (Bin Picking)

The classical approach requires CAD models and hand-coded trajectories. RL solves this without that: the cobot learns to grasp solely from simulation. State space: point cloud from Intel RealSense D435 + joint angles. Action: 7-DoF continuous gripper movement. The reward is constructed as follows:

def grasp_reward(self):
    if self._check_grasp_success():
        return 1.0 + lift_height_bonus
    elif self._check_collision():
        return -1.0
    else:
        dist = np.linalg.norm(self.gripper_pos - self.target_pos)
        return -0.01 * dist

Sim-to-real transfer is done via Isaac Sim with domain randomization (random mass, friction, lighting). This yields 90+% successful grasps on a real UR10 without fine-tuning.

Multi-Agent Path Finding for AGV Fleet

Note: when there are many AGVs, a centralized planner cannot recalculate routes quickly. Decentralized RL — each agent chooses a direction (N/S/E/W/Stay) based on a local occupancy map. Reward:

rewards = np.zeros(self.n_agv)
for i, agv in enumerate(self.agvs):
    if agv.reached_goal():
        rewards[i] += 10.0
    if self._collision(i):
        rewards[i] -= 5.0
    rewards[i] -= 0.01  # time penalty
    rewards[i] -= 0.1 * agv.steps_waiting

Result: warehouse throughput 20-30% higher than with fixed routes. The RL approach handles up to 200 vehicles — twice as many as a classical centralized planner.

Job Shop Scheduling with RL

Scheduling N machines and M jobs is NP-hard. An RL agent outperforms the best dispatching rules (SPT, EDD) by 5–15% on makespan. We use a hybrid: RL for fast local decisions, Google OR-Tools for periodic global optimization. This reduced idle time by 18%.

Predictive Maintenance

RL is also applied to predicting equipment failures. By analyzing sensor telemetry (vibration, temperature, current), the agent estimates remaining useful life. This allows maintenance to be planned without production stoppages, reducing unplanned downtime.

Why RL Wins Over Classical Algorithms?

Research in the Journal of Manufacturing Systems shows RL reduces downtime by 30%. Compare: the classical approach requires reprogramming when the object changes, while RL adapts through reward tuning. The table below provides a clear comparison.

The RL approach is twice as performant as the classical centralized planner for AGV fleets.

How We Do It: Stack and Case Study

Stack: ROS2 (Humble), MoveIt 2 with OMPL, Isaac Sim + Isaac Lab, PyTorch, Gymnasium-Robotics. For AGVs — custom simulator based on ROS2 Navigation2.

Case study from our practice: automation of an automotive component assembly line. 5 UR10e cobots, 20 AGVs, 12 machines. We implemented:

• RL grasping of parts from bins (6 part types, up to 10 cycles per minute);
• MARL for AGV fleet — delivery time reduced by 35%;
• RL job queue scheduler for machines — idle time reduced by 18%.

The work process is broken into stages:

What's Included

• RL model for your task (grasping / AGV routing / scheduling) with trained policy
• Integration into ROS2 infrastructure: node-controller, topics, services
• Simulation environment (Isaac Sim / Gazebo) for future fine-tuning
• Documentation: state/action space description, reward design, metric card model
• Training for your team (2-3 sessions of 4 hours each)
• Deployment guarantee: policy adjustment for condition changes within 3 months

How to Set Up RL Grasping for a Cobot

1. Collect data with an Intel RealSense D435 camera to create 3D models of objects.
2. Set up the simulation environment in Isaac Sim with domain randomization.
3. Define state/action space and start training with PPO.
4. Validate the policy in simulation over 1000 episodes, then transfer to the real robot.
5. Calibrate reward functions based on real test results.

Approximate Timelines

• AGV fleet routing: 10 to 14 weeks
• Robotic grasping (sim-to-real): 16 to 24 weeks
• Production line RL scheduler: 20 to 32 weeks

Cost is calculated individually based on your KPIs. Get a consultation — our engineers will analyze your production task and propose a solution.

Our Advantages

We have 10+ successful RL deployments on industrial sites. Certified integrators for ROS2 and Isaac Sim. We exclusively use open-source tools — you are not locked into vendor software. We provide full documentation and a policy guarantee. Request a consultation to discuss your project.

Reinforcement Learning: PPO, SAC, DQN and Industrial Applications

We see projects every day that fail not because of a weak algorithm, but because of incorrect rewards. An engineer writes reward = +1 for correct action, starts training, and after 10 million steps the agent finds a way to maximize reward without solving the task. This is reward hacking — a systemic pain of industrial RL. Our experience shows: proper reward accounts for 70% of success.

Why is RL harder than supervised learning?

In supervised learning, there is a dataset with correct answers. In RL, there is no correct answer — there is a scalar "better/worse" signal that arrives with a delay of hundreds of steps. The agent explores the space and finds a strategy on its own.

Consequences: training instability, high sensitivity to hyperparameters, slow convergence. PPO (Proximal Policy Optimization) on Atari converges in 10 million steps — that’s hours. On robotic tasks with real physics — days or weeks in simulation.

Algorithm selection by task:

How to tune PPO and avoid common problems?

PPO is the workhorse of RL. The main idea: limit policy updates via ratio clipping clip_range=0.2. This provides stability compared to vanilla policy gradient. But without proper tuning, the agent does not converge.

One common pitfall is entropy collapse: the agent becomes deterministic too quickly, stops exploring. Symptom — entropy coefficient drops to zero. Cure — ent_coef=0.01–0.05 and do not lower below 0.001. Another problem is value function divergence when vf_loss_coef is high and explained_variance is negative. We recommend vf_coef=0.5 and gradient clipping max_grad_norm=0.5.

Incorrect n_steps also breaks training. n_steps=2048 is Stable-Baselines3 default. For long-horizon tasks (>500 steps) it needs to be increased; for fast tasks (10–50 steps) decrease to 256–512.

For quick start, use stable-baselines3 + sb3-contrib. For research and custom algorithms — tianshou or CleanRL.

SAC for continuous control

SAC (Soft Actor-Critic) adds entropy maximization to the objective — the agent learns to be both efficient and diverse. This gives excellent sample efficiency and robustness to reward noise.

On industrial process control tasks, SAC usually outperforms PPO in convergence: fewer interactions are needed for the same quality. The key parameter is target_entropy. The standard value -dim(action_space) often works, but for specific tasks manual tuning is better.

How to transfer a trained agent to a real device?

Training RL on a real robot is expensive and dangerous. Standard approach: train in simulation → transfer to real hardware. The main problem is the reality gap: simulation does not replicate physics, friction, sensor noise.

The primary tool is domain randomization. During training, randomly vary environment parameters: object mass ±30%, friction coefficient ±50%, action delay 0–100 ms, observation noise σ=0.01–0.1. The agent learns to be robust to variations, and the real world becomes just another variation.

Comparison of popular simulators:

RLHF: training LLMs from human feedback

RLHF became the standard after InstructGPT. Classic scheme: supervised fine-tuning → reward model → PPO.

Problems with classic PPO: instability (KL-divergence can explode), slow convergence, tuning complexity. Hence popular alternatives:

• DPO — bypasses reward model, learns from preference pairs. Simpler, more stable, but less flexible.
• GRPO — used in DeepSeek-R1, good for reasoning tasks.
• ORPO — combines SFT and alignment into one stage.

The trl library from Hugging Face is the standard. Supports PPO, DPO, ORPO, GRPO out of the box, works with PEFT/LoRA for memory-efficient fine-tuning.

"Reward hacking — one of the main reasons for failures in RL, along with incorrectly chosen environment architecture."

What is included in the work

• Architectural solution and justification of algorithm selection
• Development and documentation of the reward function
• Creating a simulator or configuring an existing one
• Training, hyperparameter sweep (Optuna / Ray Tune)
• Transfer to real hardware or integration into product
• Documentation, access to code and simulators
• Team training and 3-month support after deployment

Work process

1. Task audit — define goals, resources, constraints.
2. Reward engineering — formalize desired behavior, check for reward hacking.
3. Environment and algorithm selection — baseline, first runs.
4. Systematic hyperparameter sweep — use Optuna.
5. Training in simulation with domain randomization.
6. Testing on real equipment (if necessary).
7. Deployment, monitoring, support.

Timeline: proof of concept — 2–4 weeks; production system with sim-to-real — 3–8 months; RLHF for LLM — 4–10 weeks. Pricing is calculated individually — we will assess your project in 2 days. Contact us for a consultation.

Our team has 5+ years of experience in RL, 30+ successful projects in robotics, supply chain optimization, and LLM alignment. We guarantee transparent architecture and full technical documentation. Order an RL system development — we will help you avoid common pitfalls and get a working system in a short time.