GPU cluster setup: where model training breaks
A self-built setup calculates everything correctly on paper, but starts falling apart on the first long training run. GPU cluster setup is not about buying accelerators and drivers, but about scheduling modes, cooling, and fault tolerance. These determine how many epochs per day you actually get.
Accelerator oversubscription is the first source of losses. When the scheduler places several tasks on one card, they compete for memory and clock cycles, the training step grows, and with dense packing overheating and throttling are added. Formally all accelerators are busy, but in practice throughput drops severalfold.
The second failure is idle training queue. Tasks wait while a node holds a process that is hung or waiting for data, and the distribution across cards is set statically. Accelerators sit idle, and researchers launch experiments manually, bypassing the queue.
The third is loss of checkpoints when a node goes down. If checkpoints are written only to local disk and not frequently enough, the failure of one machine nullifies a multi-hour run.
On your own capacity this is the most expensive category of failures: it hits not the hardware, but the team's time.
All three problems are solved at the cluster architecture level, not by replacing accelerators. We break down how this is addressed.
What a dedicated cluster for model training provides
When training runs on scattered machines, the experiment cycle stretches out: some accelerators sit idle waiting for data, some are occupied by someone else's task, and the result is reproduced only on a specific node.
A dedicated training cluster removes these swings — resources are distributed among tasks explicitly, rather than based on which hardware happens to become free.
The speed of the experiment cycle is determined not by the peak performance of a single node, but by the time from launch to metric.
We separate data preparation and training across different groups of nodes so that data reading does not compete with computation, and the queue is configured so that short hypothesis checks do not wait behind long tasks. A cycle that used to take a day now fits into several hours.
Training fault tolerance is a consequence of environment control, not luck.
We continuously monitor temperature, power, and inter-node communication, so deviations are visible before a task fails midway through a run; checkpoints are saved automatically, and restart proceeds from the last one, not from scratch.
The class of failures caused by overheating and power drops goes away together with cooling headroom and correct power limits.
Accelerator utilization becomes a measurable quantity: you can see how much time a node is occupied with useful work, who holds a resource idle, and where tasks hit the data preparation bottleneck.
This leads to predictable planning — expansion is calculated based on actual node utilization, not by eye, and it does not turn into idle purchased hardware.
Options for organizing compute: own setup, rental, cloud
The compute mode is chosen not by convenience, but by workload profile: how long training lasts, how bursty the schedule is, whether there are requirements for the data perimeter.
The same project often starts on rental capacity and moves to its own setup — it is critical that the transition does not require rewriting launch code and rebuilding environments.
| Option | When it fits | Limitations |
|---|---|---|
| Own compute setup | Around-the-clock utilization, data does not leave the perimeter, predictable schedule needed | Deployment and cooling take weeks, accelerators sit idle between peaks |
| Rental of compute capacity | An isolated setup is needed for the project duration or for a one-off training peak | Access according to the site's schedule, limited choice of configurations, dependence on others' timelines |
| Cloud with pay-as-you-go | Bursty schedule: experiments, retraining, rare spikes | With continuous utilization it is more expensive than an own setup, sensitive data requires a separate setup |
| Hybrid compute mode | Heavy training on the site side, model inference close to data | A unified launch management layer and a common artifact registry are needed |
An own cluster is not needed until the workload reaches a stable plateau or until data can be moved out. Then rental or cloud is cheaper not only at the start, but also in maintenance: you do not need to keep a shift, monitor cooling, and stock hardware for peak.
Hybrid compute mode covers the case when training goes outside while the service must respond close to the data source. We separate these setups using a single launch template, so moving a stage from one environment to another remains a parameter change, not a separate project.
We fix the mode choice based on measured utilization over two to three weeks: where accelerators sit idle, where the data transfer bottleneck is, which stage can be moved out without risk to the perimeter.
Implementation procedure: from workload audit to handover into operation
A cluster is not built "for ML in general" — it is built for a specific workload profile. That is why we start with measurements: without numbers on utilization and queue wait time, the scheme remains a guess, and rework after commissioning costs more than design.
-
Training workload audit. We capture the task profile: the share of accelerator idle time waiting for data, gradient exchange time between nodes, where dataset reading bottlenecks. The result is a profile model with the peak number of concurrent launches.
-
Training infrastructure design. Based on the profile, we determine the number of nodes and accelerator density, communication topology, storage volume for datasets and checkpoints. Here we also fix how tasks are distributed across nodes.
-
Scheduler and storage setup. A queue with project priorities and resource quotas so that one team does not consume the entire cluster. We prepare storage for parallel reading: hot dataset cache, a separate volume for checkpoints, and their rotation policy.
-
Fault tolerance testing. We shut down a node, a network link, and a storage segment one by one during training. We see whether the task survives a segment restart without losing progress and how quickly it recovers.
-
Measurement on a reference task. We compare actual training speed with the calculated one and look for mismatch: a slow node, incorrect batch processing parameters, overheating.
-
Cluster handover into operation. We provide configurations as code, monitoring dashboards, response regulations, and instructions for the customer's engineers.
Handover artifacts: configurations, diagrams, regulations
Cluster handover is not an oral story "how everything is set up here," but a package of artifacts that lives in the customer's repository and is versioned together with changes.
If knowledge remains with one engineer, any driver update or node failure turns into downtime that no one knows how to resolve.
- Infrastructure configurations — declarative description of nodes, networks, storage, and task queue. Deployment from scratch and recovery after failure are reproduced without manual edits and "reconfiguration from memory."
- Compute setup diagrams — node topology, data path from storage to accelerators, failure points and bottlenecks. A new engineer sees where the throughput limit is before changing anything.
- Resource access policies — roles, quotas, team isolation, priority rules. One group of tasks does not displace another, and disputed cases are resolved according to a written rule, not in chat.
- ML infrastructure documentation — image composition, driver and library versions, connection procedure, dataset and intermediate training state catalogs. Launching a new model does not turn into archaeology.
- Maintenance regulations — planned work windows, driver update procedure, node decommissioning and replacement, storage integrity checks. Downtime becomes predictable instead of emergency.
- Incident instructions — scenarios "node dropped out of the queue," "task hung," "out of space" with step-by-step actions and escalation points.
The package is updated with every change to the setup: artifacts that diverge from reality stop being a support and begin to mislead.
Case study: how distributed training stopped sitting idle
A cluster of several dozen nodes with NVIDIA accelerators served a stream of distributed training tasks — from short experiments to multi-day runs across all nodes at once.
Formally, fault tolerance existed: jobs were issued through the scheduler, data resided on a shared file system, and each task had its own set of checkpoints. In practice, the failure of one node stopped the entire group, and the epoch progress was lost entirely.
What the metrics showed before
The share of useful accelerator utilization stayed around 40%: the rest of the time the hardware waited for a restart, node release, or manual incident triage.
Recovery after a failure took hours — the engineer brought the environment up again, moved the task, and started training from the last saved checkpoint, which by then lagged by several epochs. Some tasks did not resume at all: checkpoints were on the local disk of the failed machine.
What was changed
We separated storage and compute: checkpoints are written to object storage at a specified interval, so progress survives the loss of a node entirely.
The scheduler was configured for automatic restart of the group from the last checkpoint and for liveness checks — a hung process is removed and moved to a healthy node, without waiting for the entire group to fall apart.
Separately, we leveled data access so that a recovered node does not take the entire bandwidth from the others.
After the changes, the share of useful utilization grew from 40% to 87%, recovery time after a failure decreased from hours to a few minutes, and the share of tasks lost without the possibility of continuation fell to zero.
Per-run accounting also appeared: you can see how much time went to computation, how much to queue waiting, and how much to restarts — degradation is noticed before a researcher sees it.
How does a single node differ from a distributed training launch?
A single node is several accelerators with a shared fast bus and memory, with communication inside it going directly.
As long as model weights, optimizer states, and intermediate activations fit in that memory, a distributed launch more often loses: each step begins and ends with synchronization.
The boundary is not by the number of accelerators, but by what limits the training step. If the bottleneck is computation, and memory does not overflow, a second node will not speed up the work but add waiting on transfers.
A distributed launch is needed when the model or data batch no longer fits, or when the amount of computation required is unachievable on a single node.
The parallelization method is selected according to the cause of the shortage. By data, the workload is divided between nodes and gradients are synchronized — it scales most easily, but requires a full copy of the model on each node.
By model, weights and layers are split between accelerators: this is how models that do not fit in one node's memory are trained, at the cost of noticeably more communication.
Data exchange between nodes is the main source of losses when scaling. Collective gradient synchronization becomes more expensive with the number of participants, and its share in the step depends on network topology and layer placement across nodes. Eight nodes give a speedup not of eight times, but of five to six — this is a normal result.
Therefore, the decision is made based on measurements, not the number of cards: we capture the step profile on one node, look at the share of time for exchange and waiting, and calculate what gain the second stage will provide.
If the gain does not cover the added complexity of fault tolerance and checkpoints — we keep one node and squeeze everything out of it.
Technical concerns: fault tolerance, access, support
Before discussing economics, the customer checks cluster behavior in real operation: training on multiple nodes, planned updates, handover of the setup to their team.
These scenarios are included in the project from the start — refining them on a live setup is more expensive and riskier.
The scheduler sees the state of each node, and training proceeds with checkpoints: a dropped node does not nullify the run. Access is granted by roles, updates go in waves across node groups, so production tasks do not stop entirely. Below are answers to the concerns most often raised before the start.
What happens if one node fails?
The task saves checkpoints at a specified interval, the scheduler excludes the node from the pool and continues training from the last point — minutes are lost instead of a day.
For model inference we keep reserve capacity: failure of one node does not bring down the service, traffic is redistributed. Degradation alerts arrive before a second node fails.
How is access divided between teams?
We build permissions on a least-privilege basis: a researcher works in their namespaces and queues but does not change configuration; the on-duty engineer manages restarts and limits; administrative access — with you or with us according to the regulation. Secrets are stored separately from code, and actions are written to an audit log.
How is the setup updated without downtime?
We update in waves: first the control node, then groups of workers with task checks after each wave. Drivers and libraries are fixed in images, so rollback is a return to the previous image, not a manual reinstall. A separate test bench remains for validating new versions.
What is included in support after handover?
We hand over configuration as code, placement diagrams, and regulations: pool scaling, adding users, actions during degradation. After that — support with an agreed response time, updates for current library branches, and scheduled health checks together with your team.
Let's discuss the task — we'll come back with an estimate and work plan
To make the estimate substantive rather than "by eye," a few inputs are enough: current hardware and network configuration, workload profile — which models are trained and with how many accelerators, target launch time, and fault tolerance requirements.
Whatever is missing — we will clarify on a call: we will go through the task, walk through the topology, and come back with a plan.
- Describe the task: what you train or serve, what data volume, how many teams will share the cluster.
- Send the configuration of nodes, network, and storage — or say what has not yet been purchased and is still being selected.
- Specify the workload profile: model sizes, number of parallel experiments, environment isolation requirements.
- Note the constraints: access, regulations, downtime windows, by what deadline the working setup is needed.
- Formulate the success criterion: task launch time, accelerator utilization, number of simultaneous trainings.
- We will agree on the fault tolerance level: what we reserve, what we allow to lose, how we recover.
The review takes several business days. We come back not with a presentation, but with a work plan: stages, composition of configurations and policies, cluster diagram, and implementation timeline — from installation and setup to acceptance under load. We agree on the stages together so that the timelines are realistic, not optimistic.
Describe the task in a convenient way — we will come back with an estimate and work plan.
