NCCL: why training on multiple accelerators does not scale
When a model no longer fits into the memory of a single accelerator, it is split across several, expecting multiple-fold speedup. In practice, the second card provides an increase noticeably less than double, and on eight, a run may be longer than on one. The reason is that time is spent not on computation but on communication between accelerators.
Communication consists of collective operations — broadcasting weights, gathering tensor parts, averaging gradients on each iteration.
The NVIDIA NCCL library decides over which channels and in what order to arrange these transfers so that no accelerator waits for the others longer than necessary.
The communication bottleneck reveals itself indirectly: the share of accelerator idle time grows with cluster size, and the speedup curve reaches a plateau. If transfers go through shared memory and the CPU, the latency on each iteration accumulates into hours over a long run.
Tuning collective operations — buffer size, number of concurrent transfers, channel priority, process-to-device binding — changes the picture more than replacing hardware.
We explain how this is addressed: from measuring the communication profile to a configuration in which nonlinear speedup becomes linear again.
What tuned inter-accelerator communication provides
By default, NVIDIA NCCL selects the communication topology itself — ring or tree, number of channels, inter-node transport.
On a heterogeneous configuration, this automatic selection makes mistakes: traffic goes through a slow bus instead of direct links between accelerators, and utilization drops by almost half.
After explicitly setting the scheme (algorithm, direct access level, required network interface, process-to-link binding), communication throughput increases by 1.5-2x, and the synchronization step stops being the most expensive part of the run.
The effect is especially visible when scaling the cluster. If on eight accelerators synchronization consumed part of the step, on several dozen it consumes the entire time budget, and adding nodes provides no speedup.
Tuned communication restores linearity: run time decreases in proportion to the number of accelerators, and each accelerator's utilization stays in the upper part of the range instead of idling while waiting for the others.
Separately, the class of failures "hang on synchronization" goes away. One slow link or a lost packet makes all participants wait, and the job hangs until manual intervention.
Communication timeouts, a diagnostic log, and link health monitoring turn this into a predictable failure with a clear cause: the process stops itself, and the faulty path is removed from the communication group.
We keep control via four metrics: communication throughput per step, run time, accelerator utilization, and number of restarts. They show exactly where performance is lost — in computation, in the network, or in waiting.
NCCL: accelerator communication schemes, options, and limitations
The choice of communication scheme in NCCL is determined by two things: the physical topology of the node and the workload profile — message size and how often accelerators synchronize.
The library selects the mode itself, but on non-standard topology auto-detection makes mistakes, and training starts waiting for communication instead of computing.
Schemes and their applicability limits
| Option | When it fits | Limitations |
|---|---|---|
| Ring scheme | Homogeneous topology, large messages, many accelerators in one group | Sensitive to a slow link: one lagging participant sets the pace for everyone |
| Hierarchical communication | Accelerators are divided into groups within a node and between nodes | Requires explicit group description; on small tasks, overhead is higher than the gain |
| Direct accelerator connection | Peak tasks where communication occurs between neighboring accelerators | Works only where the topology physically supports it and does not extend beyond the node |
| Communication through node memory | Heterogeneous topology, debugging, small transfer volume | Node memory bandwidth is lower than direct links — communication becomes the bottleneck |
The schemes are not mutually exclusive: for small messages one mode is more reasonable, for large ones another, and threshold-based switching provides a noticeable gain on mixed workloads.
We capture the communication profile on a real task, compare it with the topology map, and fix the mode in the launch configuration — this way behavior is reproducible from run to run and does not depend on what the library guessed this time.
Tuning stages: from node audit to handover of the runbook
NCCL tuning is not a matter of trying environment variables one by one, but a sequence where each step relies on a measured workload profile.
Without a topology audit, editing collective operations turns into guesswork: the bottleneck may be in the bus between accelerators, in the network transport, or in the training code.
- Cluster audit. We capture the connectivity map between accelerators and nodes: where communication goes directly, where through an intermediate link, and whether the channel is the same on all nodes. The output is a list of nodes that behave differently from the rest.
- Workload profile. We run real training while measuring step time and the share taken by inter-device communication — it becomes clear what is slowing things down: the network, gradient synchronization, or data preparation.
- Communication scheme. For the profile, we select collective operation algorithms, buffer size, and the order of communication between groups: a large buffer provides a steady stream, a small one provides lower latency on short steps.
- Tuning. We make changes one at a time and repeat the measurement after each — this way the contribution of a change is visible, rather than the combined effect of a dozen variables. We keep the configuration in a repository.
- Load testing. Several jobs at once, node disconnection, reconnection: the spread of step time must remain within acceptable limits.
- Acceptance tests. We compare step time and accelerator utilization before and after, run failure scenarios, and record reference values that the team uses to check the cluster after changes.
- Handover. We deliver the runbook: how to change parameters, where to look when speed drops, and which measurements to take before release.
Configurations, topology schemes, and launch runbooks
After the cluster is put into operation, the customer's team is left not with a "set of scripts from the contractor", but with a set of artifacts by which the launch can be repeated without us.
Below is what we hand over and check during acceptance together with your engineers: each item has an owner and a storage location in your repository.
- Training configuration files: NCCL channel topology parameters, communication buffer sizes, timeouts, and retry thresholds — with a comment explaining why that particular value was chosen.
- Topology scheme: accelerator connectivity map, domain distribution, and routes used during gradient exchange between nodes.
- Launch runbook: warm-up order, sequence of link checks, and criteria under which a launch is stopped rather than forced through.
- Access policies: roles on nodes, restrictions on configuration edits, and the procedure for granting rights to test and production environments.
- Diagnostic profiles and alert thresholds: what is considered communication degradation and at what point the on-call engineer must intervene.
- Operations documentation: instructions for degradation, node replacement, and rollback to the previous launch profile.
- Change log: what edits were made to configurations, for what purpose, and what measurable effect they produced.
- Acceptance protocol: what was checked, by whom it was confirmed, and which items remain under maintenance.
Case study: how we eliminated accelerator idle time on long runs
What was broken
A cluster of several dozen accelerators, long training runs. Jobs failed at hour 5–8: collective gradient exchange hung, ranks waited for each other, and the job was killed by a watchdog timeout.
A restart took 40–60 minutes — cache warm-up, data reloading, state alignment; less than half of the useful time per day remained.
The analysis identified three causes. Part of the inter-node traffic went over a slow path: link topology was not taken into account when choosing the route. Timeouts were factory defaults and did not reflect the real duration of operations.
Checkpoint writing ran in parallel with communication and consumed bandwidth, and configurations on nodes diverged — the same code behaved differently.
What we changed
We brought NVIDIA NCCL to a single profile across the cluster: fixed the order of path selection between nodes, separated gradient exchange and checkpoint writing across different channels, raised timeouts to real values, and enabled a detailed log — it shows which rank fell behind and at which operation.
An 11-hour run completes without a single synchronization failure, and the success rate of launches is 9 out of 10 versus 1 out of 3. Recovery after a failure takes about 6 minutes: state is restored from the last checkpoint.
Epoch time decreased by about one fifth, and reproducibility stays within noise — the same data and seed produce matching metrics.
NCCL: what to do if the cluster is already deployed but communication is slow
Short answer: if training is already running and the cluster is working, there is no need to rebuild the environment. NCCL handles collective operations such as all-reduce, and most losses are eliminated by selecting the transport, number of channels, and buffering parameters.
Communication diagnostics begins with comparing actual step time with the calculated transfer volume: a discrepancy by a factor means that accelerators are idling while waiting rather than computing.
Next, we separate inter-server communication from intra-server communication. If the step degrades only when scaling to several nodes, the cause is in the network interface and route — a poorly chosen adapter diverts traffic around the fast path.
Within a node, the bottleneck is usually in the interconnect topology between accelerators and in how processes are distributed across compute lanes.
Step-by-step optimization runs on a short run in parallel with production training: first we record the baseline step time, then change one parameter at a time — transport, number of channels, buffer size, the threshold at which transfer grouping is enabled. We compare each change on the same dataset to separate the tuning effect from random variation.
Environment reconfiguration is performed without stopping training: edits to variables and configuration are first tested in a backup run and only then transferred to the main one.
In practice, the share of step time spent on communication drops by a factor, and the cluster stops hitting a ceiling when adding new accelerators.
Common technical concerns before starting work
Before starting, the same concerns come up: will tuning work on already assembled hardware, will training have to be rewritten, and what will remain with the team after handover.
Cluster nodes are heterogeneous — won't communication degrade?
We do not assume symmetry but measure it: we capture the profile of collective operations between all pairs of nodes and specify an explicit topology for NCCL instead of automatic selection. Slow channels remain, but they stop slowing down the others — communication levels out predictably.
We have several different training models — is one configuration used for all?
We select the profile for the type of workload: parallel training of a large model, fine-tuning, and inference differ in buffer sizes and the scheme for distributing work across accelerators. We keep separate profiles so that editing one mode does not affect the others.
On long runs, communication slows down — how can this be caught?
Degradation accumulates unnoticed: latencies grow, and the share of useful accelerator time falls. We set up observability from the first days — communication metrics, timeout logs, and comparison of throughput at the beginning and end of the run.
What will remain with us after handover, and will it break neighboring jobs?
We hand over configurations, validation scripts, and the topology scheme with explanations — tuning can be changed without us. We isolate environments: pinned library versions, containers, resource limits, so neighboring jobs on the same nodes are not affected.
Let's discuss the task: what to send for an estimate and plan
To come back with an estimate and work plan rather than general wording, we need inputs on the cluster and the workload. Some data is captured directly from the running system — we will advise which metrics and commands are needed for that. Describe the task in free form, even if some items are still unknown.
- Topology: how many nodes, how many accelerators on each, and how the nodes are connected to each other.
- Inter-accelerator communication: whether NCCL is enabled, which environment variables are set, and which transport is used.
- Training profile: number of model parameters, batch size, target time per step, required run duration.
- Current metrics: how long a step takes now and where speed is lost — in computation, communication, or data reading.
- Environment: current-branch drivers and libraries, job management system, containers, scheduler.
- Constraints: allowed downtime window, requirements for rollback to the previous configuration, access policies for nodes.
- What hurts: performance drop when scaling, hangs at startup, errors in communication logs.
- Timelines and acceptance: when the launch is needed and who on your side accepts the result.
Based on these inputs, we state the implementation timeline and team composition — how many engineers and for what period. Then we will discuss the task in detail and come back with an estimate and plan: stages, checkpoints, and rollback options.
