DeepSpeed setup: why training hits a memory wall
Training large models hits a wall not because of accelerator speed, but because of memory capacity. Weights, gradients, and optimizer states reside on the device at the same time, and the optimizer takes several times more space than the model itself.
Memory shortage does not show up at the start but in the middle of a run — when hours have already been spent on data preparation and warm-up.
The cost of such a failure is higher than it seems. Hardware sits idle between restarts, progress rolls back to the last checkpoint, and some experiments have to be cut down: reduce batch size, shorten context length, use a weaker model than needed.
A training process crash slows down not just one iteration but the entire direction — from research to shipping the model to production.
Configuring DeepSpeed is about controlling exactly where states live.
ZeRO stages determine what is sharded across accelerators, offloading moves part of the state to host memory, and the configuration profile ties the chosen scheme to specific hardware and communication topology.
Below, we break down how this is addressed in practice — from stage selection to a stable run without crashes or manual restarts.
What changes after proper training configuration
Without distributed training configuration, there is almost always one bottleneck: optimizer states and activations do not fit into accelerator memory, so the engineer cuts the batch size and loses utilization.
Devices sit idle waiting for synchronization, and the run stretches out. Proper configuration distributes states across devices and removes this bottleneck — the same model trains several times faster.
DeepSpeed provides a set of modes for this: sharding optimizer states and gradients across devices, offloading weights and activations to host memory, and gradient accumulation in small batches.
The point is not to enable everything at once, but to select a combination for the specific model and node configuration: excessive sharding adds inter-device communication and eats up the gain, while insufficient sharding brings back the memory shortage.
After selecting the modes, hardware utilization levels out at 85–95% instead of dropping during synchronization, and training speed stops depending on sequence length.
Stability is ensured by checkpoints: we save optimizer and scheduler state, not just weights, and write them asynchronously so that writing does not slow down computation.
Recovery time after a node failure drops from hours to minutes — the process resumes from the last checkpoint rather than from scratch.
In total, this yields a high share of runs that reach the target metric without manual intervention, and predictable timelines: the training plan is built on measured step speed, not on an optimistic estimate.
DeepSpeed setup: ZeRO modes and memory offloading
The choice of mode is determined by which training states remain on each accelerator. Optimizer states take several times more space than the parameters themselves, followed by gradients, and only then weights.
ZeRO modes differ in which of these groups is sharded across devices, and memory offloading adds a second level — to CPU memory or local storage.
| Option | When it fits | Limitations |
|---|---|---|
| ZeRO-1: optimizer states are sharded | The model already fits in accelerator memory, but you want to increase batch size | Parameters and gradients are duplicated on every device — moderate memory savings |
| ZeRO-2: plus gradients | Training on multiple accelerators when gradient communication has already become a bottleneck | Inter-device communication volume grows, sensitivity to interconnect topology |
| ZeRO-3: plus parameters | The model as a whole does not fit in accelerator memory; long context and large batch are needed | Parameters are fetched at every step — higher latency and requirements for the communication channel |
| Offloading states to CPU memory | Accelerator memory is scarce, and the step is sensitive to latency | Limited by bus throughput between CPU and accelerators |
| Offloading states to disk | The model does not fit even with full parameter sharding | Noticeable drop in step speed, fast local storage is needed, wear increases |
How we choose in practice
First, we fix the accelerator memory budget and the required batch size, then measure consumption on one training step. If the weights fit, we stop at the first or second mode; if not, we move to full sharding and only then to offloading.
Changing the mode is a single configuration parameter, so rolling back to the previous option takes minutes rather than rewriting the training code.
Implementation sequence: from workload audit to handover
Before changing anything, we need to understand exactly where training hits a wall: memory capacity or data exchange between accelerators.
That is why the work starts with measuring the current mode, not with random configuration edits — otherwise there is nothing to compare the effect of changes against.
Seven stages
- Training audit and workload profile. We profile one step: how much memory weights, optimizer states, and activations take, and what share of time accelerators sit idle waiting for communication. The output is a bottleneck named by a number, not a guess.
- Placement scheme. We select the degree of partitioning for model states (ZeRO) and the offloading mode: what remains on the accelerator, what goes to CPU memory, and what is recomputed on the fly. We fix the scheme before making edits.
- Training implementation. We change parameters one at a time: micro-batch size, gradient accumulation step, overlapping communication with computation, and checkpoint frequency. After each change — measurement.
- Series of control runs. Short runs on the real dataset with comparison of the convergence curve against the baseline. The goal is to make sure the speed gain is not bought with quality degradation.
- Stabilizing long runs. We enable checkpoint recovery, limit checkpoint growth, and check behavior when one process fails — training continues instead of crashing entirely.
- Documentation. Placement scheme, parameter values, and rationale for each change. Without this, the configuration becomes a "black box" again after the first edit.
- Handover to operations. We run training together with your team, go over typical failures and edit points, and leave a procedure for launch and parameter changes.
What the team receives at the end
We hand over not only a working run, but everything needed for the team to reproduce and maintain it without us. The training configuration is documented as a set of profiles for the specific hardware composition, not as a lucky parameter selection held in one engineer's head.
- Working launch profiles: number of processes, data batch size per device, gradient accumulation step, model state partitioning stage, and the mode for moving them to CPU memory or disk.
- Model and state placement scheme: what lives on accelerators, what is moved beyond them, how much each state takes, and at what model size the scheme stops fitting within resources.
- Launch procedure: the sequence of checks before starting — availability of all devices, free memory capacity, integrity of the last saved checkpoint, and whether the profile matches the actual hardware.
- Restart procedure: from which checkpoint training resumes, how the saved state is verified, and how quickly the process returns to work after a failure.
- Instructions for the on-call shift: which log entries to read, which signs of degradation count as emergencies, what is fixed on site, and when the responsible person is called.
- Configuration change log: every edit with the reason and before/after measurement — step time, memory consumption, and quality dynamics on the validation set.
- Alerting scheme: a set of metrics and thresholds by which the shift learns about a problem before training stops for a day.
The folder with profiles, schemes, and procedures sits next to the code and is versioned with it. We conduct acceptance together with your team: the on-call engineer independently runs launch and recovery on the test environment, and we answer questions on each item.
Case study: how we removed the memory limitation during training
A team came to us that did not have enough memory for the next version of their model.
Training did not crash on the first step but after several hours: measurements showed that optimizer states and gradients took more memory than the weights themselves, and some accelerators sat idle waiting for communication.
What we measured
We profiled step by step: peak memory consumption, utilization of each accelerator, share of time spent on transfers between cards, and the number of runs that survived to completion.
This provided a baseline for comparison — without it, any changes to the placement mode look like faith rather than an engineering decision.
What we changed and what we achieved
We moved training to DeepSpeed: enabled sharding of optimizer states, gradients, and parameters across cards (the ZeRO family), moved part of the states to CPU, and replaced increasing the micro-batch with gradient accumulation.
We recompute activations during the backward pass instead of keeping them in memory entirely — this trade of computation for memory is more beneficial here.
The model size grew almost threefold on the same number of accelerators. The share of runs reaching completion rose from about a third to all of them, out-of-memory failures disappeared as a class, and step time increased by 10–15% — a conscious price for predictability.
Checkpoints are saved and restored, and rolling back to the previous placement configuration takes minutes.
How is ZeRO different from offloading states to disk?
These are different mechanisms, not alternatives. ZeRO divides model states — optimizer, gradients, parameters — across accelerators, and each device holds only its share. Offloading does not divide anything: it moves states to CPU memory or disk, freeing video memory.
ZeRO stages build on the effect: the first divides optimizer state, the second adds gradients, and the third adds the parameters themselves.
The price is inter-accelerator communication at every step: the higher the stage, the more traffic and the more training depends on interconnect throughput.
| Option | What it removes from the accelerator | Cost | When it fits |
|---|---|---|---|
| ZeRO-1 | optimizer state | inter-accelerator communication | the model fits, but the optimizer is heavy |
| ZeRO-2 | optimizer and gradients | traffic grows noticeably | headroom is needed for large batches |
| ZeRO-3 | plus the parameters themselves | maximum traffic | otherwise the model does not fit on a group |
| Offloading to CPU | optimizer, gradients, buffers | bus throughput | memory is low, step time can tolerate it |
| Offloading to disk | states and parameters | slowest step | the model greatly exceeds total memory |
Offloading places the same tensors in CPU memory or on disk and fetches them by computation time. Peak video memory consumption drops, but the step hits a wall at bus transfer speed — the memory gain is paid for with step time.
In practice, the mechanisms are combined: in DeepSpeed they are enabled independently, and the combination of ZeRO-3 with parameter offloading trains a model that otherwise would not fit on a group of accelerators.
We calculate the memory budget for state and activations, then select a mode and verify it against real step measurements.
Common technical concerns before starting
Before starting, engineering teams check three things: how the setup will fit the existing training pipeline, what will happen to the collected checkpoints, and how to roll back if the result is not satisfactory. Below are answers to what we most often discuss on the first calls.
Is this compatible with our training pipeline?
We build the configuration around your run: the trainer, dataset format, and checkpoint scheme remain the same. What changes is the launch profile — ZeRO depth, offloading optimizer states to CPU and disk, gradient accumulation — not the model code. Before scaling, we compare one training step against the current run.
Will training time increase?
Offloading trades memory for communication speed, so we select the profile based on measurements: how many steps per second each option gives and what batch size fits.
Usually the freed memory allows increasing batch size and the number of devices to cover the overhead, but without measurements this is not a statement.
How do we roll back changes if the result is not satisfactory?
The changes are isolated: the configuration and launch wrappers are stored separately, and the previous run remains operational.
Rolling back means switching to the old profile and continuing from the last checkpoint; we do not change the checkpoint format, so intermediate results are readable by both the old and the new run.
How do we make sure quality has not dropped?
We compare on a short run: the same data and order, baseline run and new profile, loss curves and metrics on the holdout set.
If the discrepancy is within noise — we continue; if there is a systematic shift — we look for the cause in gradient accumulation precision and optimizer settings.
Let's discuss the task and come back with an estimate and plan
The plan and estimate do not appear "by eye" but from input data.
To calculate the stages of model training optimization, we need to see the current launch configuration, workload profile on several accelerators, and step time requirements — without this, the estimate will remain a range, not a plan.
To start, the following is enough:
- access to the repository with current launch scripts and sharding and offloading configs — read permissions are enough;
- model description and size, number of accelerators, and type of connection between them;
- real workload profile: sequence lengths, fine-tuning share, throughput requirements;
- logs of failed runs or out-of-memory reports, if they were preserved;
- environment constraints: on-prem infrastructure or cloud, access and isolation requirements;
- contact of the engineer who accepts the result — to coordinate changes without intermediaries.
The estimate is returned as a document: list of stages, expected effect of each (accelerator utilization, volume of offloaded states, step time), risks, and rollback points.
The first stage starts not with configuration edits, but with a reproducible measurement: we record baseline throughput, take a memory profile, and only then change the sharding strategy and offloading order.
Describe the task in free form — we will come back with an estimate and work plan.
