Personalized VR training with an AI tutor: feedback and assessment

Why a VR AI mentor does not retain the skill

A typical picture: the virtual simulator has been assembled, the scenario is completed to the end, and at the finish there is a completion percentage and a green checkmark.

A few days later, at the real site, the same employee acts as if the session never happened: the order of operations gets confused, and there is no reaction to a deviation from the normal course.

The reason is that feedback in training is disconnected from action: it comes after the fact and evaluates the result, not the process.

It is not visible at which step the sequence diverged, where the gaze went to the wrong node, where a decision was made out of habit instead of following the regulation.

Assessment of the learner's actions degenerates into "completed / not completed," and the person does not understand what exactly to correct in the next attempt.

The second failure is the absence of adaptive learning. The scenario is the same for everyone, the difficulty does not change along the way, and a weak spot is not replayed under new conditions.

But skill transfer depends precisely on variability: the same task in a different setting, with a different order of inputs, with equipment failure that must be noticed and handled. Only then does the assessment show not knowledge of the scenario, but the stability of the skill.

An AI VR mentor addresses this at the level of an individual action, not a session: it compares posture, gaze direction, order of operations, and reaction time with a reference model of the skill.

We examine how such feedback is embedded in a virtual simulator and why skill transfer depends on it more than on graphics quality.

What changes after implementing an adaptive mentor

After implementing an adaptive mentor, the effect is measured by three values: the speed of reaching independence — how many attempts are needed to complete the scenario without hints, the share of critical errors per session, and the instructor time spent on one group.

The tutor raises the level only after stable completion of the current one: the threshold and growth step are set in the scenario description, so adaptive difficulty follows the learner's actual actions, not the course schedule.

Automatic assessment is assembled from events within the scenario: reaction time, order and completeness of steps, number of requests for a hint, returns to the previous stage.

From this data, a competency profile is built, and the next exercise is selected in the zone where the learner makes mistakes rarely but still with effort. The profile is recalculated after each session, so the training plan does not have to be rebuilt manually.

The instructor's workload changes qualitatively: instead of accompanying every participant, they work at the points where the assessment showed a risk of critical error.

Time per group is reduced, and there are more simultaneous learners; the debrief is based on the session log, which shows at which step and why a deviation occurred.

We record the cumulative effect by the levels of the Kirkpatrick model — from reaction to the lesson to transfer of the skill to a work task.

Formats of the AI VR mentor: training ground, classroom, autonomous mode

The mentor mode is chosen not by preference, but by where the lesson takes place and what the communication channel is like on site.

In a classroom, simultaneous control of the group and a unified picture of progress are needed; at an off-site location, work without an external network; and in independent practice, the learner is left alone with a headset, and the mentor must guide them without an instructor.

The difference between the modes is where the tutor model is located and how quickly a response comes.

Cloud training provides a fresh model and aggregate analytics across cohorts, but is sensitive to the channel; on-site deployment removes this dependency, while autonomous mode sacrifices part of the debrief for predictability.

In practice, the modes are combined: the lesson runs on the deployed site, and lesson results are pulled into the common loop when connectivity appears.

Implementation stages: from scenario audit to handover

Implementing an AI mentor in VR is not installing a ready-made product, but a sequence in which each stage relies on the result of the previous one.

If the scenario audit or calibration is skipped, the mentor assesses formally: it reacts to movement but does not distinguish a correct decision from a random one.

1. Scenario audit — we review the training modules and mark the points where mentor reaction is needed: tool selection, order of operations, typical errors. The output is a decision map on which the hint logic is built.
2. Assessment scheme — we formalize the criteria: what is considered a correct action, which deviations are acceptable, where assessment depends on context. We fix them before development so that metrics are not adjusted to a convenient result.
3. Mentor logic setup — we assemble reaction types: hint, clarifying question, pause, error debrief. Each type is assigned a trigger threshold.
4. Feedback calibration — we run reference completions and compare the mentor's reaction with the methodologist's assessment. We correct discrepancies in the rules, not in the labeling; otherwise the behavior will not reproduce on the next group.
5. Pilot group — a limited set of learners completes the module in full: we look at the share of completed tasks, the number of hint requests, and the spread of scores between learners.
6. Solution handover — we hand over the scenario configuration, assessment scheme, calibration logs, and instructions for methodologists: how to add new modules and revise criteria without us.

What remains with the customer: schemes, roles, regulations

The solution is handed over not as an assembled demonstration scene, but as a set of artifacts by which your team can change the training content without rebuilding the entire application.

Below is what is recorded in the documentation and remains with the customer based on the results of the work. Each item covers its own class of tasks: content, assessment, access, and support.

• Scenario map — a list of training situations, their order of completion, branching points, and places where the mentor enters the dialogue. Using it, a methodologist can add a new module without looking into the code.
• Assessment rules — criteria, scales, and weights for each type of response, pass thresholds, and a list of disputable cases with debrief. They allow changing the strictness of assessment without editing the application logic.
• Access rights matrix — the roles of methodologist, instructor, administrator, and observer, as well as what each of them sees and edits. The observer does not change scenarios; the instructor sees only their group's results.
• Solution documentation — component composition, deployment diagram, environment requirements, startup procedure, and recovery after failure.
• Support regulations — how scenarios and assessment models are updated, who approves changes, and within what timeframe failure reports are handled.
• Version log and rollback procedure — which revisions of scenarios and assessment rules were released, who changed them and when, and how to return to the previous one.
• Quality control — a set of metrics: share of completed scenarios, assessment accuracy on a labeled sample, and the number of mentor requests it could not handle.

All of this is maintained in a single repository with change history and described in Russian, without being tied to a specific development team.

Changing an assessment rule or access rights becomes an edit to a document, not a rebuild of the product, and acceptance proceeds according to a list where each item has an owner and a verification method.

Case study: how the share of errors in certification was reduced

The customer is a distributed service department, about 240 engineers across sites. Certification was conducted twice a year: a practical scenario according to the regulation and an instructor assessment based on formalized criteria.

The share of critical errors remained at 23%, and the debrief of attempts lagged behind the certification itself by two to three weeks.

How it was organized before implementation

Debriefing one attempt took the instructor 25–40 minutes: reviewing the recording, checking against the regulation, and writing a conclusion.

There was no automatic recording of actions, so disputable episodes were discussed from memory rather than by the actual sequence of operations.

Repeat attempts followed one after another, without targeted work on the error: 41% of engineers repeated the same point of the regulation on the second attempt, and 54% passed on the first attempt.

What was changed and what the result was

The scenarios were moved to VR on standalone Meta Quest headsets: the mentor records the trajectory of the hands and the tool, compares it with the reference from the regulation, and within a few minutes provides a personal debrief — which step was violated and what risk this poses according to the instruction. The instructor checks only disputable discrepancies.

The share of critical errors in certification fell from 23% to 7%, debriefing one attempt decreased from 25–40 minutes to 3–5, first-attempt passing grew from 54% to 86%, and repetition of the same error fell from 41% to 9%.

How a scenario-based mentor differs from a generative one

The difference is where the AI mentor in VR takes its next utterance from. A scenario-based mentor follows a pre-labeled graph: nodes are checkpoints of the exercise, transitions are correct and typical erroneous actions. A generative dialogue does not store branches and assembles the response from context.

This is where the predictability of assessment diverges: in the scenario-based model, the score is calculated by formal rules — whether the node was reached, in what time, how many attempts. The same path yields the same result, which is critical for grading.

A generative mentor assesses by a rubric, and without calibration on reference runs, the spread between evaluators is noticeable.

Scenario maintenance is the second divide. The graph is expanded manually for new equipment and regulations, but the behavior is explainable and does not change after a model update.

The generative approach is easier to scale, but a set of validation dialogues is needed: changing the model changes the strictness of feedback, and regressions are caught before release to the group.

Freedom of action is what generative dialogue is chosen for: it answers the unplanned "why this way and not otherwise." More often, a hybrid is assembled: the graph holds the corridor of actions and the assessment, while error debrief is given to the generative part with a topic restriction. Scenes are assembled in Unity; one environment serves both modes.

Technical concerns: integrations, load, data

The engineering questions at the selection stage are usually the same: will the mentor integrate into an already operating fleet of simulators, what happens during a group lesson, and where the recordings physically reside. Below are brief answers on each, including the details that most often surface already at launch.

How does this work in a group lesson?

The mentor guides each person at their own pace, while the instructor receives an aggregate picture: who is stuck on a step, who repeats the same error. The number of simultaneous streams is planned in advance, so a peak group does not affect hint latency.

Where is data processed and how objective is the assessment?

Processing takes place in your or a dedicated loop under your control — this is selected at the design stage. Recordings are anonymized before analysis, access is role-based, and export goes to your reporting.

Assessment criteria are fixed before launch and tied to observable features, so disputable cases are reviewed manually, and the report is reproducible.

Let's discuss the task: what to prepare for the first meeting

So that the assessment relies on your process rather than averaged assumptions, inputs are enough for the first meeting — not a finished technical specification.

The more specifically you describe the target skills and the current simulator fleet, the more accurately we will name the scope of work, stages, and acceptance points. Below is what makes sense to collect in advance.

• Target skills: a list of operations that an employee must perform after the course, and how this level is checked today — mentor checklist, exam, error debrief from a recording.
• Current simulator fleet: what equipment is already in operation, how many simultaneous participants need to be supported, and whether there is a requirement to work in a shared scene.
• Training scenarios: regulations and typical errors that the AI mentor must recognize and on which it provides feedback.
• Data for assessment: what is permissible to record during an attempt — sequence of actions, reaction time, movement, speech — and at what depth.
• Integration points: learning management system or HR loop, result export format, and who on your side reads the reports.
• Loop constraints: where the solution is deployed, personal data requirements, and who administers access and updates.
• Customer team: methodologist, subject-matter instructor, IT specialist — the speed of approving assessment scenarios depends on their availability.
• Pilot group: how many learners and attempts we allocate so that measurements can determine that the mentor works correctly.

Describe the task in free form — we will come back with an estimate and a work plan that separately sets out the stages, delivery scope, and criteria by which you accept the result.