AI-Driven Game Testing and QA System: Turnkey Development

Manual QA testing doesn't scale: an open world with 1000+ quests and 100+ mechanics cannot be manually tested in reasonable time. A team of 5 testers covers about 2000 unique states per week, while our RL agent covers 10 million in a day. We developed a turnkey system that autonomously explores the game world, finds crashes, softlocks, exploits, and balance issues — working 24/7 and increasing coverage to 90% in hours.

Why Manual Testing Falls Short

Even a large QA team misses edge cases: rare action combinations, race conditions, quests in non-standard order. Our RL agent with curiosity-driven exploration (ICM) purposefully seeks unknown states, while the Go-Explore algorithm remembers interesting points and returns to them. In 10 million steps, the agent visits more unique states than a team of 5 in a month. As a result, we find 3x more bugs at launch, and QA costs drop by 10x.

How We Build the AI Testing System

Crash/Softlock Testing

The agent randomly explores the entire action space, triggering edge cases that crash the game or get stuck in a loop. Agent code with ICM:

class GameTestingAgent:
    """Agent for coverage-based testing"""

    def __init__(self, game_env):
        self.env = game_env
        self.visited_states = set()
        self.crashes = []
        self.softlocks = []

        # Curiosity-driven exploration
        # ICM (Intrinsic Curiosity Module): reward for new states
        self.icm = ICM(obs_dim=game_env.obs_dim,
                       action_dim=game_env.action_dim)

        self.policy = PPO("MlpPolicy", game_env,
                          ent_coef=0.05)  # high entropy for exploration

    def collect_coverage_data(self, n_steps=1_000_000):
        obs = self.env.reset()
        for step in range(n_steps):
            try:
                action, _ = self.policy.predict(obs)
                obs, _, done, _, info = self.env.step(action)

                # log new states
                state_hash = self._hash_state(obs)
                self.visited_states.add(state_hash)

                # detect softlock (agent goes in circles)
                if self._detect_softlock():
                    self.softlocks.append(self.env.get_state_dump())

                if done: obs = self.env.reset()

            except Exception as e:
                self.crashes.append({
                    'error': str(e),
                    'state': self.env.get_state_dump(),
                    'action_sequence': self.recent_actions[-100:]
                })
                obs = self.env.reset()

Content Coverage Testing

We check if there are zones, items, or achievements never reached by normal gameplay. The agent with ICM is rewarded for visiting new states — hidden areas are detected automatically. On one project, we found 23 unimplemented quests and 5 hidden achievements that developers had forgotten about.

Exploit Detection

A special agent trained to maximize score without constraints:

# exploit reward: ONLY score, ignore all "normal" paths
def exploit_reward(info):
    return info['score']  # + info['gold'] + info['level']

# train on minimal steps (fast exploit)
model = PPO("MlpPolicy", env,
            gamma=0.5,        # short horizon = wants fast rewards
            ent_coef=0.1)     # lots of exploration

# if agent finds a way to get 10x average score in 1 minute
# → this is an exploit for QA team

What Is Intrinsic Curiosity Module (ICM)?

class ICM(nn.Module):
    """Intrinsic Curiosity: reward = prediction error for new states"""
    def __init__(self, obs_dim, action_dim, feature_dim=256):
        super().__init__()
        # feature encoder
        self.phi = nn.Sequential(
            nn.Linear(obs_dim, feature_dim), nn.ELU(),
            nn.Linear(feature_dim, feature_dim)
        )
        # forward model: predict next state features
        self.forward_model = nn.Sequential(
            nn.Linear(feature_dim + action_dim, feature_dim), nn.ELU(),
            nn.Linear(feature_dim, feature_dim)
        )

    def intrinsic_reward(self, obs, action, next_obs):
        phi_obs = self.phi(obs)
        phi_next = self.phi(next_obs)

        # prediction error = how new this state is
        a_onehot = F.one_hot(action, self.action_dim).float()
        predicted_next = self.forward_model(torch.cat([phi_obs, a_onehot], dim=1))
        curiosity = F.mse_loss(predicted_next, phi_next.detach(), reduction='none').mean(-1)
        return curiosity  # high for new, low for known

How Go-Explore Solves the Sparse Reward Problem

Classic RL gets stuck in games with sparse rewards. Go-Explore (Adept AI):

1. Stores an archive of interesting states (by diversity).
2. Randomly selects a state from the archive.
3. Returns to it (deterministic replay).
4. Continues exploration from there.

class GoExploreAgent:
    def __init__(self, game):
        self.game = game
        self.archive = {}  # cell -> (score, state_snapshot)

    def cell_key(self, state):
        """Discretize state into cell (simplified for storage)"""
        # for 2D game: (x//50, y//50, level_id)
        return (state['x'] // 50, state['y'] // 50, state['level'])

    def run(self, n_iterations):
        for _ in range(n_iterations):
            # select a state from archive (rarely visited)
            cell = self._select_cell()
            state = self.archive[cell]['snapshot']

            # restore state (savestate)
            self.game.load_state(state)

            # randomly explore N steps
            for _ in range(np.random.randint(5, 100)):
                action = self.game.action_space.sample()
                new_state, _, done, _, _ = self.game.step(action)
                new_cell = self.cell_key(new_state)
                if new_cell not in self.archive:
                    self.archive[new_cell] = {
                        'snapshot': self.game.save_state(),
                        'visits': 0
                    }
                if done: break

Approach Comparison: ICM vs Random vs Expert

What's Included in the Turnkey Package?

Total: 12–16 weeks for a full system. Cost is calculated individually after auditing your project. Contact us for a consultation — we'll evaluate your game and propose a solution.

Efficiency Metrics

• State space coverage: % of game states visited (goal ≥90%).
• Crash count per build: average 3 crashes per build before optimization, 0 after.
• Softlock incidents found: 50+ per project.
• New exploits detected per patch: average 8.
• Time to 90% coverage: 4 to 8 hours (manually — a month).

Regression Testing

After a patch — launch agents to ensure previously passing tests are not broken. CI/CD pipeline automatically runs tests on each pull request:

# GitHub Actions / Jenkins
name: Game QA Tests
on: [push, pull_request]

jobs:
  ai-qa:
    runs-on: ubuntu-latest
    steps:
      - name: Run Coverage Agent (10 min budget)
        run: python run_coverage_agent.py --budget 600 --headless

      - name: Check Coverage Metrics
        run: |
          python check_coverage.py \
            --min-level-coverage 0.85 \
            --max-new-crashes 0

Our engineers have 10+ years of experience in AI game development, with over 50 successful projects. We guarantee the system will find at least 3x more bugs than manual testing at launch. Get a consultation — we'll evaluate your game and propose a solution.

Reinforcement Learning: PPO, SAC, DQN and Industrial Applications

We see projects every day that fail not because of a weak algorithm, but because of incorrect rewards. An engineer writes reward = +1 for correct action, starts training, and after 10 million steps the agent finds a way to maximize reward without solving the task. This is reward hacking — a systemic pain of industrial RL. Our experience shows: proper reward accounts for 70% of success.

Why is RL harder than supervised learning?

In supervised learning, there is a dataset with correct answers. In RL, there is no correct answer — there is a scalar "better/worse" signal that arrives with a delay of hundreds of steps. The agent explores the space and finds a strategy on its own.

Consequences: training instability, high sensitivity to hyperparameters, slow convergence. PPO (Proximal Policy Optimization) on Atari converges in 10 million steps — that’s hours. On robotic tasks with real physics — days or weeks in simulation.

Algorithm selection by task:

How to tune PPO and avoid common problems?

PPO is the workhorse of RL. The main idea: limit policy updates via ratio clipping clip_range=0.2. This provides stability compared to vanilla policy gradient. But without proper tuning, the agent does not converge.

One common pitfall is entropy collapse: the agent becomes deterministic too quickly, stops exploring. Symptom — entropy coefficient drops to zero. Cure — ent_coef=0.01–0.05 and do not lower below 0.001. Another problem is value function divergence when vf_loss_coef is high and explained_variance is negative. We recommend vf_coef=0.5 and gradient clipping max_grad_norm=0.5.

Incorrect n_steps also breaks training. n_steps=2048 is Stable-Baselines3 default. For long-horizon tasks (>500 steps) it needs to be increased; for fast tasks (10–50 steps) decrease to 256–512.

For quick start, use stable-baselines3 + sb3-contrib. For research and custom algorithms — tianshou or CleanRL.

SAC for continuous control

SAC (Soft Actor-Critic) adds entropy maximization to the objective — the agent learns to be both efficient and diverse. This gives excellent sample efficiency and robustness to reward noise.

On industrial process control tasks, SAC usually outperforms PPO in convergence: fewer interactions are needed for the same quality. The key parameter is target_entropy. The standard value -dim(action_space) often works, but for specific tasks manual tuning is better.

How to transfer a trained agent to a real device?

Training RL on a real robot is expensive and dangerous. Standard approach: train in simulation → transfer to real hardware. The main problem is the reality gap: simulation does not replicate physics, friction, sensor noise.

The primary tool is domain randomization. During training, randomly vary environment parameters: object mass ±30%, friction coefficient ±50%, action delay 0–100 ms, observation noise σ=0.01–0.1. The agent learns to be robust to variations, and the real world becomes just another variation.

Comparison of popular simulators:

RLHF: training LLMs from human feedback

RLHF became the standard after InstructGPT. Classic scheme: supervised fine-tuning → reward model → PPO.

Problems with classic PPO: instability (KL-divergence can explode), slow convergence, tuning complexity. Hence popular alternatives:

• DPO — bypasses reward model, learns from preference pairs. Simpler, more stable, but less flexible.
• GRPO — used in DeepSeek-R1, good for reasoning tasks.
• ORPO — combines SFT and alignment into one stage.

The trl library from Hugging Face is the standard. Supports PPO, DPO, ORPO, GRPO out of the box, works with PEFT/LoRA for memory-efficient fine-tuning.

"Reward hacking — one of the main reasons for failures in RL, along with incorrectly chosen environment architecture."

What is included in the work

• Architectural solution and justification of algorithm selection
• Development and documentation of the reward function
• Creating a simulator or configuring an existing one
• Training, hyperparameter sweep (Optuna / Ray Tune)
• Transfer to real hardware or integration into product
• Documentation, access to code and simulators
• Team training and 3-month support after deployment

Work process

1. Task audit — define goals, resources, constraints.
2. Reward engineering — formalize desired behavior, check for reward hacking.
3. Environment and algorithm selection — baseline, first runs.
4. Systematic hyperparameter sweep — use Optuna.
5. Training in simulation with domain randomization.
6. Testing on real equipment (if necessary).
7. Deployment, monitoring, support.

Timeline: proof of concept — 2–4 weeks; production system with sim-to-real — 3–8 months; RLHF for LLM — 4–10 weeks. Pricing is calculated individually — we will assess your project in 2 days. Contact us for a consultation.

Our team has 5+ years of experience in RL, 30+ successful projects in robotics, supply chain optimization, and LLM alignment. We guarantee transparent architecture and full technical documentation. Order an RL system development — we will help you avoid common pitfalls and get a working system in a short time.