FSDP

What it does

FSDP (Fully Sharded Data Parallel) is PyTorch's built-in method for training models too large for one GPU's memory. It shards parameters, gradients and optimizer states across data-parallel workers and gathers each layer's weights only when needed. The newer FSDP2 API, fully_shard, shards each parameter as a DTensor, and PyTorch recommends migrating to it from the original wrapper class.

Compare with

• vsMegatron-LM
• vsNanotron
• vsDeepSpeed

People who use it also use

Not enough marks yet to tell. When people who use FSDP mark the rest of their stack, it shows up here.