Adaptive Traffic Signal Control with MARL

Adaptive AI Traffic Signal Control and Optimization

Fixed-cycle traffic lights cause 30% of urban traffic delays. Every day, millions of drivers lose hours in congestion, and cities bear economic losses. We develop adaptive systems based on Multi-Agent Reinforcement Learning (MARL) that reduce average waiting time by 15-30%. For instance, in Hangzhou, such a system cut congestion by 22%, and in Saudi Arabia by 18%. Our approach uses MARL: each intersection is an agent with local observations optimizing phases in real time. A city with a population of 500,000 can save up to 10 million rubles per year in reduced fuel consumption and travel time.

Why RL and MARL Are the Best Approach for Traffic Signal Control?

Fixed cycles work well under predictable flow, but real traffic is unpredictable. Adaptive systems like SCOOT and SCATS require manual calibration and handle anomalies poorly. An RL agent directly optimizes phases for current traffic without manual tuning and adapts to accidents, rain, and events. Training is done in a simulator, then transferred to real controllers. In our tests, MARL outperforms SCOOT by 20-30% in average travel time.

The problem: N intersections, each with K phases (directions). The agent selects the current phase and duration to minimize total waiting time.

Simulation and Training Environment

CityFlow Simulation Environment

import cityflow

# City configuration from JSON (roads, intersections, traffic)
eng = cityflow.Engine('config.json', thread_num=4)

# Intersection state
lane_vehicles = eng.get_lane_vehicle_count()  # vehicles per lane
lane_waiting = eng.get_lane_waiting_vehicle_count()
current_phase = eng.get_current_phase(intersection_id)

# Action: change phase
eng.set_tl_phase(intersection_id, new_phase)
eng.next_step()

For detailed microsimulation, we also use SUMO—it models individual car behavior more accurately.

Gym-Compatible Environment

class TrafficEnv(gym.Env):
    def __init__(self, n_intersections, config_path):
        self.n = n_intersections
        self.eng = cityflow.Engine(config_path)

        # observation per intersection: vehicle counts + waiting + phase
        obs_dim = 12 + 12 + 8  # 12 approaching lanes, 8 phases
        self.observation_space = spaces.Box(
            low=0, high=100, shape=(n_intersections, obs_dim))

        # action: select phase for each intersection
        self.action_space = spaces.MultiDiscrete([8] * n_intersections)

    def step(self, actions):
        for i, action in enumerate(actions):
            self.eng.set_tl_phase(f'intersection_{i}', int(action))

        # multiple simulation steps per agent decision
        for _ in range(self.control_interval):  # 10–30 sec
            self.eng.next_step()

        obs = self._get_obs()
        reward = -self.eng.get_average_travel_time()  # minimize ATT
        return obs, reward, False, False, {}

Building the MARL System

Cooperation Algorithms

Independent DQN (InDQN) is a simple baseline: each intersection trains independently, treating others as part of the environment. CoLight uses attention to account for neighboring intersections:

class CoLightAttention(nn.Module):
    def forward(self, own_obs, neighbor_obs):
        # own_obs: [batch, obs_dim]
        # neighbor_obs: [batch, n_neighbors, obs_dim]
        query = self.q_proj(own_obs).unsqueeze(1)
        keys = self.k_proj(neighbor_obs)
        values = self.v_proj(neighbor_obs)

        attention = F.softmax(
            torch.bmm(query, keys.transpose(1,2)) / math.sqrt(self.d_k), dim=-1
        )
        context = torch.bmm(attention, values).squeeze(1)
        return self.output_proj(torch.cat([own_obs, context], dim=-1))

MPLight combines pressure-based feature engineering (queue pressure) with MARL, showing superior results on CityFlow benchmarks.

Reward Design

Pressure-based reward yields the best results in multi-agent settings.

Integration with Real Controllers

Most cities use SCATS (Australia) or SCOOT (UK). Our RL solution outputs signals in these systems' format via the NTCIP protocol (national ITS standard). For compatibility, we implement an SNMP adapter. In case of RL failure (network outage), the system automatically falls back to actuated control with vehicle detectors. NTCIP 1202 v03 is the key standard for interoperability.

What Data Is Needed?

Data sources: video detectors (real-time queues), inductive loops (exact counts), GPS fleet (travel time feedback), Connected Vehicles (V2X/C-V2X) for proactive optimization. An MLOps pipeline automatically updates the model with new data. For training, 1-3 months of detector recordings suffice. If V2X data is available, accuracy improves by 10-15%.

KPIs Tracked

Contact us for a personalized KPI assessment.

Work Process

1. Analysis: Collect traffic data, calibrate simulator. Build a digital twin of the district. Set up a predictive model.
2. Design: Choose agent topology (Independent / CoLight / MPLight), design reward.
3. Implementation: Train in CityFlow (1000+ episodes), test on real scenarios.
4. Testing: A/B test on a dedicated intersection, verify KPIs.
5. Deployment: Integrate with NTCIP controllers, fallback modes, MLOps monitoring.

What's Included

• MARL model with CoLight/MPLight support
• CityFlow simulator configured for your district
• Reward engineering tailored to your KPIs
• NTCIP adapter (SCATS/SCOOT)
• Documentation, operator training, 3 months of support

Deployment Timeline

A single intersection in simulation: 4 weeks. A district (20-50 intersections) with coordinated agents: 12 weeks. Full deployment with city infrastructure: 20-24 weeks.

We can evaluate your project in 2 days. Contact us for a consultation—our engineers have experience in cities with populations over 5 million and guarantee congestion reduction.

Reinforcement Learning: PPO, SAC, DQN and Industrial Applications

We see projects every day that fail not because of a weak algorithm, but because of incorrect rewards. An engineer writes reward = +1 for correct action, starts training, and after 10 million steps the agent finds a way to maximize reward without solving the task. This is reward hacking — a systemic pain of industrial RL. Our experience shows: proper reward accounts for 70% of success.

Why is RL harder than supervised learning?

In supervised learning, there is a dataset with correct answers. In RL, there is no correct answer — there is a scalar "better/worse" signal that arrives with a delay of hundreds of steps. The agent explores the space and finds a strategy on its own.

Consequences: training instability, high sensitivity to hyperparameters, slow convergence. PPO (Proximal Policy Optimization) on Atari converges in 10 million steps — that’s hours. On robotic tasks with real physics — days or weeks in simulation.

Algorithm selection by task:

How to tune PPO and avoid common problems?

PPO is the workhorse of RL. The main idea: limit policy updates via ratio clipping clip_range=0.2. This provides stability compared to vanilla policy gradient. But without proper tuning, the agent does not converge.

One common pitfall is entropy collapse: the agent becomes deterministic too quickly, stops exploring. Symptom — entropy coefficient drops to zero. Cure — ent_coef=0.01–0.05 and do not lower below 0.001. Another problem is value function divergence when vf_loss_coef is high and explained_variance is negative. We recommend vf_coef=0.5 and gradient clipping max_grad_norm=0.5.

Incorrect n_steps also breaks training. n_steps=2048 is Stable-Baselines3 default. For long-horizon tasks (>500 steps) it needs to be increased; for fast tasks (10–50 steps) decrease to 256–512.

For quick start, use stable-baselines3 + sb3-contrib. For research and custom algorithms — tianshou or CleanRL.

SAC for continuous control

SAC (Soft Actor-Critic) adds entropy maximization to the objective — the agent learns to be both efficient and diverse. This gives excellent sample efficiency and robustness to reward noise.

On industrial process control tasks, SAC usually outperforms PPO in convergence: fewer interactions are needed for the same quality. The key parameter is target_entropy. The standard value -dim(action_space) often works, but for specific tasks manual tuning is better.

How to transfer a trained agent to a real device?

Training RL on a real robot is expensive and dangerous. Standard approach: train in simulation → transfer to real hardware. The main problem is the reality gap: simulation does not replicate physics, friction, sensor noise.

The primary tool is domain randomization. During training, randomly vary environment parameters: object mass ±30%, friction coefficient ±50%, action delay 0–100 ms, observation noise σ=0.01–0.1. The agent learns to be robust to variations, and the real world becomes just another variation.

Comparison of popular simulators:

RLHF: training LLMs from human feedback

RLHF became the standard after InstructGPT. Classic scheme: supervised fine-tuning → reward model → PPO.

Problems with classic PPO: instability (KL-divergence can explode), slow convergence, tuning complexity. Hence popular alternatives:

• DPO — bypasses reward model, learns from preference pairs. Simpler, more stable, but less flexible.
• GRPO — used in DeepSeek-R1, good for reasoning tasks.
• ORPO — combines SFT and alignment into one stage.

The trl library from Hugging Face is the standard. Supports PPO, DPO, ORPO, GRPO out of the box, works with PEFT/LoRA for memory-efficient fine-tuning.

"Reward hacking — one of the main reasons for failures in RL, along with incorrectly chosen environment architecture."

What is included in the work

• Architectural solution and justification of algorithm selection
• Development and documentation of the reward function
• Creating a simulator or configuring an existing one
• Training, hyperparameter sweep (Optuna / Ray Tune)
• Transfer to real hardware or integration into product
• Documentation, access to code and simulators
• Team training and 3-month support after deployment

Work process

1. Task audit — define goals, resources, constraints.
2. Reward engineering — formalize desired behavior, check for reward hacking.
3. Environment and algorithm selection — baseline, first runs.
4. Systematic hyperparameter sweep — use Optuna.
5. Training in simulation with domain randomization.
6. Testing on real equipment (if necessary).
7. Deployment, monitoring, support.

Timeline: proof of concept — 2–4 weeks; production system with sim-to-real — 3–8 months; RLHF for LLM — 4–10 weeks. Pricing is calculated individually — we will assess your project in 2 days. Contact us for a consultation.

Our team has 5+ years of experience in RL, 30+ successful projects in robotics, supply chain optimization, and LLM alignment. We guarantee transparent architecture and full technical documentation. Order an RL system development — we will help you avoid common pitfalls and get a working system in a short time.