The following line of code :
if not hvd_utils.is_using_hvd() or hvd.local_rank() == 0:
in ./runtime/runner.py should be replaced by :
if not hvd_utils.is_using_hvd() or hvd.rank() == 0:
Indeed if you run the script ./scripts/UNet_FP32_8GPU.sh on multiple nodes(eg 2 nodes with 4P100 GPUs each) and you let only the local_rank() == 0 it will crash between the end of the training and the beginning of the evaluation part. And you get this error :
Traceback (most recent call last): File "/DeepLearningExamples/TensorFlow/Segmentation/UNet_Industrial/scripts/../main.py", line 147, in <module> is_benchmark=RUNNING_CONFIG.exec_mode == 'training_benchmark' File "/DeepLearningExamples/TensorFlow/Segmentation/UNet_Industrial/runtime/runner.py", line 507, in train hooks=training_hooks, File "/powerai-1.6/lib/python3.6/site packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/ibm/powerai-1.6/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/ibm/powerai-1.6/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default saving_listeners) File "/ibm/powerai-1.6/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1495, in _train_with_estimator_spec any_step_done = True File "/ibm/powerai-1.6/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 861, in __exit__ self._close_internal(exception_type) File "/ibm/powerai-1.6/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 894, in _close_internal h.end(self._coordinated_creator.tf_sess) File "/DeepLearningExamples/TensorFlow/Segmentation/UNet_Industrial/utils/hooks/profiler_hook.py", line 214, in end with open(os.path.join(self._sample_dir, "..", perf_filename), 'w') as f: File "/ibm/powerai-1.6/lib/python3.6/posixpath.py", line 80, in join a = os.fspath(a) TypeError: expected str, bytes or os.PathLike object, not NoneType
and
Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[5380,1],4] Exit code: 1
For you hvd.local_rank()=0 worked because you ran the script ./scripts/UNet_FP32_8GPU.sh on one node DGX-1 with 8x V100 16G GPUs.
The following line of code :
if not hvd_utils.is_using_hvd() or hvd.local_rank() == 0:in ./runtime/runner.py should be replaced by :
if not hvd_utils.is_using_hvd() or hvd.rank() == 0:Indeed if you run the script ./scripts/UNet_FP32_8GPU.sh on multiple nodes(eg 2 nodes with 4P100 GPUs each) and you let only the local_rank() == 0 it will crash between the end of the training and the beginning of the evaluation part. And you get this error :
Traceback (most recent call last): File "/DeepLearningExamples/TensorFlow/Segmentation/UNet_Industrial/scripts/../main.py", line 147, in <module> is_benchmark=RUNNING_CONFIG.exec_mode == 'training_benchmark' File "/DeepLearningExamples/TensorFlow/Segmentation/UNet_Industrial/runtime/runner.py", line 507, in train hooks=training_hooks, File "/powerai-1.6/lib/python3.6/site packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/ibm/powerai-1.6/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/ibm/powerai-1.6/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default saving_listeners) File "/ibm/powerai-1.6/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1495, in _train_with_estimator_spec any_step_done = True File "/ibm/powerai-1.6/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 861, in __exit__ self._close_internal(exception_type) File "/ibm/powerai-1.6/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 894, in _close_internal h.end(self._coordinated_creator.tf_sess) File "/DeepLearningExamples/TensorFlow/Segmentation/UNet_Industrial/utils/hooks/profiler_hook.py", line 214, in end with open(os.path.join(self._sample_dir, "..", perf_filename), 'w') as f: File "/ibm/powerai-1.6/lib/python3.6/posixpath.py", line 80, in join a = os.fspath(a) TypeError: expected str, bytes or os.PathLike object, not NoneTypeand
Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[5380,1],4] Exit code: 1For you hvd.local_rank()=0 worked because you ran the script ./scripts/UNet_FP32_8GPU.sh on one node DGX-1 with 8x V100 16G GPUs.