FSDP PyTorch: why training a large model hits memory
Training a large model hits the limit not because of the number of parameters itself, but because of what each accelerator must hold. The full model state — weights, gradients, and optimizer state — is replicated on every device in distributed training.
The optimizer state is several times heavier than the weights themselves: several accumulated values are stored per parameter.
As soon as the total volume stops fitting into accelerator memory, training either crashes with an allocation error or forced economizing begins: the batch size is cut, gradient accumulation is enabled, layers are manually split.
Each such step is a trade-off: more free memory, but step time grows, and some devices idle while waiting.
FSDP PyTorch solves this by sharding model state: weights, gradients, and optimizer state are split across devices rather than stored entirely on each one.
Before computing a layer, its parameters are gathered on the required accelerators; afterwards they are released again, while optimizer state remains sharded permanently.
That is where the main part of memory is freed, and training large models no longer depends on whether the model fits entirely into one accelerator.
We break down how this is solved in practice: which sharding modes to choose, how to configure device groups, and where performance is most often lost.
What sharding of model and optimizer state provides
Sharding model and optimizer state distributes parameters, gradients, and moments across devices instead of replicating them on each one. Training usually hits the limit not because of the model itself but because of its optimizer: weights still fit, but states no longer do.
In PyTorch FSDP, this is enabled by the sharding mode, and after that the result depends on how communication and checkpoints are configured.
We check point by point what changes measurably.
- Batch size. Freed memory goes to activations, so global batch size grows without changing hardware. A larger batch gives a smoother gradient and makes it possible to raise the learning rate without losing convergence.
- Training step time. Gradient communication is overlapped with computation, so the step lengthens noticeably slower than the data volume grows. Total training time is reduced by a factor, not by percentages.
- Fault tolerance. Sharded state is assembled into a checkpoint without replication, so failure of an individual process does not mean loss of progress. A class of failures related to memory overflow on a single device goes away.
- Recovery time. Starting from the last checkpoint takes minutes: state comes up in the same partition, and training continues from the same step.
It is worth verifying this on your own workload profile — for different architectures, the bottleneck shifts between optimizer memory, activations, and the communication network.
We take measurements before and after, record batch size, step time, and recovery time, and only then consider the mode configured.
Comparison of sharding modes and their limitations
The choice of mode is determined by two values: how much memory is freed on each accelerator and what gradient and parameter communication costs. Full sharding cuts parameters, gradients, and optimizer states — the memory gain is maximal, but there are also the most transfers.
Parameter groups allow selectively excluding individual layers from sharding — embeddings, normalizations, output head — when sharding them produces more communication than savings. Below are three modes and the boundaries beyond which each loses.
| Mode | When it fits | What it cannot do |
|---|---|---|
| Full sharding | Model does not fit on one accelerator: parameters, gradients, and optimizer states are split | Communication at every step is maximal; with weak interconnect, accelerators idle while waiting |
| Hybrid sharding | Multiple nodes: within a node the channel is fast, between nodes it is slower | Requires dividing the world into groups that are multiples of the number of accelerators per node; with incorrect division, the gain is lost |
| Selective, gradients only | Model fits in memory, but headroom is needed for activations and fine-tuning | Parameters and optimizer states remain replicated — memory savings are noticeably lower |
The mode is locked in by measurements, not by description: they look at the share of time spent in communication and memory utilization at the working batch size.
In PyTorch, switching between modes is a change of wrapper configuration; the model itself does not need to be rewritten.
How implementation proceeds: from audit to handover to the team
Training a large model hits the memory of one accelerator, so model and optimizer state are distributed across several. During this transition, configuration errors surface: mismatched precision modes, extra transfers between devices, convergence divergence from the previous run. We conduct implementation in short steps, each confirmed by measurement.
-
Training configuration audit. We review the current recipe: batch size and its splitting, precision mode, layer composition, learning rate scheduler. We record the baseline — step time, memory consumption, convergence curve on a short run.
-
Sharding plan. We decide what to distribute across devices: parameters, gradients, optimizer state, and what to leave replicated. We determine which blocks to wrap entirely so as not to pay for extra communication.
-
FSDP configuration. We assemble configuration for the chosen plan: wrapping policy, parameter rebuild order, gradient accumulation. We verify that the run is reproducible on the same number of devices.
-
Convergence check on the real task. We compare with the baseline by steps, not only by the final metric: divergence at the beginning indicates precision or communication order. We adjust until the full run.
-
Checkpoints. We configure saving and loading of distributed state, recovery after node failure, and rotation: the run continues from the last point, not from scratch.
-
Handover to the team. We hand over configurations, the sharding scheme, launch and failure triage regulations, and conduct a joint run on your testbed. After that, engineers change the recipe themselves.
Configurations, schemes, and regulations the customer receives
Everything that remains after implementation lives in your repository and documentation, not in chat and not in an individual engineer's head. The person who joins the project after us deploys training from the same files and gets the same result. Below is what exactly is recorded and handed over.
- Launch configurations — training and model splitting parameters in version control: changes go through approval rather than being edited directly on the server, so any run is reproducible.
- Sharding schemes — how FSDP in PyTorch lays out parameters and optimizer states across devices and how much memory it reserves, plus calculations for your model and number of nodes.
- Access policies — who is allowed to launch training, change configurations, and offload weights; roles are separated, access to data and artifacts is least privilege.
- Recovery regulations — checkpoint frequency, retention period, and restart procedure from the last saved after node failure.
- Documentation — deployment from scratch, description of configurations and thresholds at which training degrades, troubleshooting of common errors.
- Observability — loss and throughput curves by step: it's immediately clear whether training is hitting data, network, or accelerator limits.
In sum, this is not a set of scripts but a reproducible process: it can be handed to another team or returned to a year later without archaeology.
Case: training a large model no longer hits memory
The task was typical for large models: training did not fit into accelerator memory at a meaningful batch size. The team made compromises — reduced the batch, cut context length, manually split layers — but every other run still crashed with an out-of-memory error. Engineering hours went not into model quality but into restarts and configuration tuning.
What it was before
Baseline measurement on the real task: a training step took about 3.8 seconds, 6 out of 10 runs successfully reached the checkpoint, the rest failed due to memory.
Over a workweek, 12–14 such failures accumulated, some accelerators idled, and recovery after failure required manual state assembly. Increasing the batch size, which would improve convergence, was impossible in principle.
What was changed
Training was moved to FSDP in PyTorch — parameters, gradients, and optimizer states are sharded across devices, so a full copy of the model no longer has to live in one accelerator's memory.
Additionally, activation recomputation on the forward pass was enabled, batch sizes were aligned between nodes, and checkpoint saving was converted to a sharded form — state is assembled only on recovery.
What became
The training step dropped to 2.1 seconds, the success rate rose to 10 out of 10, and there have been no memory crashes for a month. The batch size was doubled — convergence became smoother, and rollback to the previous checkpoint takes minutes instead of manual reassembly. We measure such metrics on every project: they show whether distribution tuning paid off.
How is FSDP PyTorch different from conventional distributed training?
Classic data parallelism keeps a full copy of the model on each accelerator: processes take different parts of the batch, compute gradients, and synchronize them before the optimizer step.
As long as weights, optimizer states, and activations fit into memory of one device, the scheme scales almost linearly. As soon as the model grows, replicas stop fitting entirely — and adding devices no longer helps.
Model parallelism solves the problem differently: layers or blocks are split between devices, and each holds only its part of the weights. The price — manual partitioning, load balancing, and constant activation transfers between stages.
This approach scales model size but transfers poorly between architectures and requires separate debugging for each.
FSDP PyTorch occupies an intermediate position. Parameters, gradients, and optimizer states are sharded across all processes in the group: before computation a layer is gathered entirely, after — released.
On each accelerator only a fraction of model state lives permanently, not a full copy, while the training loop remains familiar — modules are wrapped, then work proceeds as with a normal model.
Hence the criterion for when to apply: small and medium model — data parallelism, fewer communications and simpler debugging; model does not fit into memory of one accelerator even with gradient accumulation — state sharding.
For very deep networks, a combination makes sense: FSDP within a node, data parallelism between nodes. The difference between approaches comes down to one question — are we scaling replica throughput, state volume, or both at once.
Technical concerns before launching sharded training
Engineering-level doubts are usually the same: will FSDP work with the existing model code, will inter-node links handle gradient communication, what to do with accumulated checkpoints, and will quality drop with mixed precision. Let's go through them in order.
FSDP in PyTorch does not require rewriting the architecture: it is connected by a wrapping policy over modules, while the optimizer and training loop remain in place.
Separately, we check hardware requirements: the scaling limit is more often inter-node communication, not memory per device.
Will FSDP be compatible with our model?
Layers are wrapped by policy, no changes to model code are required. Before launch, we run short training on two nodes and compare the loss curve with a single-node run — divergence within noise means the transfer is correct.
What to do with accumulated checkpoints?
Old files remain valid: we parse them by layers and assemble into a sharded format with a separate script; reverse conversion enables rollback to a non-sharded run. Verification — comparing model output before and after conversion on one step.
Will inter-node links suffice and what about mixed precision?
We measure the share of time spent on inter-node communication: if it grows, we increase the shard size and reduce the number of synchronizations.
Mixed precision is enabled at the wrapper level; the risk in recomputing normalization is removed by type casting at block boundaries and a short run.
Let's discuss the task: what to clarify before starting work
The more specific the task inputs, the fewer assumptions in the estimate. We do not need production access at this stage — answers to the questions below are enough to understand the scope of work and risks.
- Current training loop: what it runs on now, what bottlenecks are already visible — insufficient memory on device, accelerator idle time, long pauses on state saving.
- Model size and memory composition: how many parameters, which layers are the heaviest, what exactly does not fit — weights, gradients, optimizer states, or intermediate activations.
- Hardware configuration: number of accelerators, how they are connected, communication channel throughput — this determines the choice of sharding scheme and whether it pays off.
- Placement scheme: what we distribute across devices, which blocks we keep entirely on each, whether rebuilding is needed when the number of nodes changes.
- Checkpoint handling: in what form state is saved, whether training resume from the same point after failure is required, and how fast it should happen.
- Failure scenario: what we consider acceptable — loss of a step, restart from the last checkpoint, recovery on fewer devices.
- Acceptance criteria: by which metrics we judge the result — step time, speedup as the number of devices grows, convergence, and final model quality.
Then we return with an estimate and plan: scope of work, timeline, team composition, and control points.
For FSDP in PyTorch, as a separate item we propose a pilot run on your model and a real data segment — it shows actual speedup and removes compatibility questions before the main work begins.
Describe the task as it is — a draft, without preparation. We will ask clarifying questions ourselves, and we will fix the work timeline and responsibility boundaries before the start.
