Fix warning if backend registers timer (#91702) (#95363)

jeejakp12 · web-flow · commit 4bd5c1e4f401 · 2023-02-23T18:57:09.000-05:00
currently logger timer is registered default for cpu/cuda. for other backends, it may or may not registers this timer. It reports warning for other backends and return which is not expected. The above may fail, if the backends has have registered this timer. For example, HPU(habana) backend registers this timer. so, in this case it reports a warning and return which is incorrect. Other case is where lazy backend timer is never registered. so, this returns a warning, and this is the reason the check was added, but it fails for other cases. Add a generic check if the timer is registered, then don’t report warning. Signed-off-by: Jeeja <jeejakp@habana.ai> Fixes #ISSUE_NUMBER Pull Request resolved: #91702 Approved by: https://github.com/kit1980
diff --git a/torch/csrc/distributed/c10d/logger.cpp b/torch/csrc/distributed/c10d/logger.cpp
@@ -320,7 +320,9 @@ void Logger::set_runtime_stats_and_log() {
         "Cuda time stats are not collected for multi-device modules.");
     return;
   }
-  if (!reducer_->params_[0].is_cuda() && !reducer_->params_[0].is_cpu()) {
+
+  if (!reducer_->timer_ &&
+      (!reducer_->params_[0].is_cuda() && !reducer_->params_[0].is_cpu())) {
     TORCH_WARN_ONCE(
         "Time stats are currently only collected for CPU and CUDA devices. "
         "Please refer to CpuTimer or CudaTimer for how to register timer "
