🚀 Feature

Motivation

This feature request is motivated by #17356 to support sparse tensors in DistributedDataParallel. But it might have a broader impact as reducing sparse tensors is not a rare use case.

Pitch

Enhance the existing reduce API in both C++ and Python to support sparse tensors for sum and product reduce ops (reduce ops are restricted by available methods in sparse tensors). Other distributed communication collectives will follow, but will have their own tracking issues.