@csarofeen 's simple DistributedDataParallel (DDP) version (with sync buffers ON, for fair comparison since we always sync buffer in our DDP) can hit 0.159 sec per iteration on 8 GPU training using 8 DDP processes for Resnet50 32 mini-batch size / GPU, on single node with 8 P100s. And our current DDP can hit 0.164 sec per iteration as the current state with 8 processes single node training.

This PR improves the performance from 0.164 sec/iteration to 0.162 sec/iteration on single-node dist training on 8 P100s