AI development for games: where the live production loop breaks
In a game with long-term support, most decisions are made manually: writers write lines, game designers configure enemy behavior with spreadsheets, support handles tickets using scripts, moderators manually review chat reports. While the audience is small, this works.
As soon as the number of sessions and events grows many times over, the queue of edits becomes longer than the window between updates, and each new mechanic adds rules that no one monitors in full.
The limit of manual decisions shows up in several places at once. Matchmaking stops holding interest at high ranks — the rules were written for the average level. User-generated content and chat are not checked before players see the violation.
The living world reacts to actions in the same way: the supply of reactions runs out by the third hour of play.
A prototype of an intelligent service usually proves quality on a prepared dataset. In the live production loop, the distribution is different: players behave differently than in testing, a new season launches — and the metrics drop.
Next to it is a question that is almost never closed at the start: how to roll back a model if it starts making mistakes, and who will notice it first.
That is why the project often exists as separate scripts and an analyst's notebooks: in a demo the benefit is visible, but it cannot withstand load, data changes, or handover to the team.
We break down how this is closed — from the evaluation loop and the fallback mode of the model to the separation of responsibility between editable rules and learned logic.
How product metrics change after service implementation
We measure the metrics of intelligent services not by "model quality" but by the state of the product.
We take the baseline before implementation: how much time is spent preparing a seasonal event, what share of requests is closed by people, how many incidents occur during peak hours. Without these "before" measurements, any effect later cannot be proven.
Content release speed grows through generation and preliminary validation: a pack of events that the team used to assemble in 4–5 weeks ships in 8–10 days.
Manual reviews of materials and texts fall to 10–15% — the rest is filtered out by automatic rules, and ambiguous cases go to a human for review. The share of automatic decisions on typical requests (complaints, match log analysis, signals of unfair play) reaches 70%.
A separate line is eliminated classes of failures. Balance degradation after a patch goes away, as do false bans and queues in player matchmaking at peak.
Downtime on peak evenings is the most expensive: an hour of unavailability means unfinished sessions, lost purchases, and a surge in support requests, so we aim for recovery in minutes, not hours.
We carry out integration through the engine's standard extension points — integration with Unity does not require rewriting game code. That is why metrics are taken on the live product in production, not on a test bench.
Engagement formats: in-house loop, hybrid, or ready-made service
Three formats are not levels of difficulty but different deployment points for the model and different areas of responsibility.
The choice is dictated by three questions: where player data lives, how strict the response latency requirements are, and who updates the model when balance or meta changes.
Below are the alternatives that are actually chosen between at the start, with the honest limitations of each. We have operated all three formats in game project production, including integration into Unreal Engine.
| Format | When it fits | Limitations |
|---|---|---|
| Ready-made industry service | Fast start: chat moderation, rating-based matchmaking, basic analytics | The model is shared and does not know your entities; payment by request volume; fine-tuning for your domain is unavailable |
| In-house loop for the task | Non-standard mechanics, sensitive data, offline operation or on the player side | Longer rollout: data, labeling, model update policy, on-call duty |
| Hybrid scheme | Core is in-house, some tasks use a ready-made service: NPC behavior, content generation, balancing | Observability is more complex: a unified telemetry and versioning layer is needed |
The format is not final: starting on a ready-made service closes the first hypotheses, and when your own telemetry accumulates, the core moves into an in-house loop without rewriting logic.
We fix the model's input-output contract upfront, so moving between formats is an adapter replacement, not a new project.
Implementation stages: from data audit to loop handover
We fix the work order before the start: each stage ends with an artifact that the client can verify. Otherwise, model fine-tuning turns into endless tweaking, and acceptance turns into a debate about taste.
- Data audit. We review game telemetry, session logs, replays, moderation history: volume, labeling, gaps, duplicates, usage rights. The output is a suitability report and a list of what needs to be collected or labeled.
- Integration scheme. We determine where the model lives and how it is called from the game loop: synchronously on an event or asynchronously via a queue, what is cached, how the system degrades when the service is unavailable.
- Acceptance criteria. We agree before development: quality metrics on a holdout set, allowable false positive rate, response time, and behavior on rare events.
- Setup and training. We assemble the baseline loop, fine-tune on the client's data, version models and datasets — the result is reproducible and can be rolled back to the previous version.
- Load testing. Run on a peak profile: concurrent sessions, spikes on update days, long series. We see where the loop hits its limit and how it survives node failure.
- Handover of the loop to the team. We hand over configurations, integration scheme, retraining policies, access, and monitoring; the client's engineers run updates themselves with our support.
What the client receives: deliverables
The delivery is not only a working loop but also a set of artifacts that your team can use to maintain and evolve the system without us.
Everything is committed to your repository and handed over with an owner for each node, rather than remaining in individual engineers' messages.
- Integration schemes — how the game loop exchanges data with the AI layer: connection points, event streams, behavior when one side is unavailable, load limits.
- Environment configurations — agreed parameter sets for development, test, and production loops; one artifact deploys without manual edits.
- Access policies and roles — role matrix for employees and services, boundaries of service accounts, key rotation rules.
- Loop documentation — architecture diagram, component responsibility areas, list of external dependencies, known limitations log.
- Maintenance policies — on-call and escalation, what counts as an incident, procedure for rolling back changes and updating models without downtime.
- Metrics and alerts — which quality and latency indicators are collected and at what values an alert triggers.
- Operations guide — restart, scaling, data replenishment, troubleshooting typical failures.
Acceptance follows this same list: we check not a demo scenario but the reproducibility of the loop by your team. Each artifact is handed over with an owner and an entry point, so the question "who fixes this in six months" does not remain open.
Case study: before and after metrics after implementation in a game service
Service RealmForge Studios: chat with hundreds of thousands of messages per day, voice rooms, and reports about player behavior. Moderation was manual — a shift of six people, a complaint review took 4–6 hours, and nighttime toxicity spikes were only closed by morning.
A mistake on a disputed case was costly on both sides: a toxic player remained in the game, and a false ban sent a paying user into an appeal for several days.
What we changed
We assembled a loop from three parts: a text and voice toxicity classifier, auto-labeling of incoming complaints, and a priority queue for humans.
Confidence thresholds were separated by channel: in open chat the reaction is stricter, in voice rooms softer, and ambiguous cases go to manual review with the full dialogue context.
Alongside, we set up a regression set from historical complaints — any model update is run against it before release.
Before and after metrics
Complaint review dropped from 4–6 hours to 8 minutes, 62% of requests are closed without human involvement, and the share of false bans fell from 19% to 6%. Nighttime toxicity spikes no longer wait for the morning shift: people handle disputed cases, not the stream.
What transfers to other projects
The priority queue, labeling scheme, and agent feedback that the model uses to fine-tune on disputed cases every two weeks. Transferring to a new service means replacing dictionaries, labeling, and thresholds, from two to four weeks without rebuilding the core.
How an in-house loop differs from ready-made industry services
The choice between a ready-made service and an in-house loop is a choice between launch speed and control.
A ready-made service provides access to someone else's model via API: agent behavior is configured within given parameters, but training data and the release schedule remain with the vendor.
A typical scenario is closed faster this way, but if you deviate from it, you wait for someone else's update.
An in-house loop means models, datasets, and training scripts in your ownership. You set the fine-tuning frequency, weight versioning, and rollback to the previous version when agent response quality drops.
The price for this is operational load: monitoring responses, handling player complaints, regular fine-tuning on fresh logs, and validating changes on a test group.
Dependency on the contractor is removed at the handover stage.
Fix who owns the weights and datasets, whether model assembly is reproducible from the repository, and whether the labeling scheme and quality metrics are described: without this, changing the vendor means rewriting the system.
| Option | When it fits | Limitations |
|---|---|---|
| Ready-made industry service | Typical tasks, no in-house model team | Model and data are with the vendor, change flexibility is limited by their cycle |
| In-house loop | Game agent behavior and content generation are part of the game core | Quality monitoring and ongoing fine-tuning are required |
| Hybrid: core in-house, periphery on a service | Chat moderation and auxiliary assets separate from game logic | Two integration loops and a unified metrics scheme |
In practice, the hybrid more often wins: control over models and data is retained where it affects the game experience, while routine tasks are given to the service.
Doubts before starting: data, load, integrations
Objections before starting are almost always technical, and this is the right conversation: player data is a sensitive area, game load is spiky, and working services cannot be stopped. Below are answers to what is most often asked before a plan is approved.
How is player data protected?
We anonymize at the input: session identifiers and behavioral features go into the model, not payment or contact data. Access to exports is role-segmented and logged, retention period is set by policy — when consent is withdrawn, the record is removed from the training set.
What happens on peak evenings and at season start?
The inference layer scales with actual load, and the request queue is bounded. Under overload, the system returns an answer based on simple rules instead of a prediction, rather than failing: we agree on the degradation threshold in advance according to the game events calendar.
How will the system integrate into current services?
Through exchange interfaces and an event stream — the server side, payments, and anti-cheat do not need to be rewritten.
We make the first launch shadow: the model computes on real traffic but does not affect the player until discrepancies with the current logic become predictable.
What about model retraining?
The training loop runs on a schedule and on a quality-drop signal. The new version passes the same checks, is enabled gradually on a share of traffic, and the previous one remains available for instant rollback.
Will we be able to maintain the system with our team?
Yes, that is the goal of the work. We hand over code, configurations, schemes, and training policies, and go through each node with your engineers — rules change and the model retrains without us.
Let's discuss the task and come back with an estimate and work plan
To discuss the task in concrete terms, and then come back with an estimate and work plan, a minimum set of inputs is needed.
Below is what is needed to start: this is usually enough to outline the architecture, the order of launching work, and agree on the scope of the first iteration. Much of what is listed you already have — it is enough to show where it is and who grants access to it.
- Goal in player behavior — what exactly the system changes: newbie retention, enemy difficulty, matchmaking, level generation.
- Data — game session exports, match logs, labeling if available; access to analytics and the test loop.
- Constraints — platforms and target devices, response requirements, personal data processing rules, content generation prohibitions.
- Responsible people — who makes decisions on game mechanics, who on infrastructure, who approves scope and acceptance.
- Current loop — in-house models and services, data collection, labeling tools; what you plan to develop on your own.
- Acceptance criteria — by which metrics we consider the solution working: share of successful sessions, suggestion accuracy, share of scenarios without manual intervention.
- Boundaries of the first phase — which mode or feature is taken into work, and what is deliberately deferred.
Describe the task in free form: what the game is, which area hurts the most, what result is needed. We will come back with an estimate and work plan — stages and points where your decisions are required.
We agree on scope by stages, so you can start with one feature without rebuilding the entire project.
