Skip to content

Fix [PyT/DLRM] bug of model.py - #517

Merged
tgrel merged 2 commits into
NVIDIA:masterfrom
t-kusanagi:dlrm-device-bug
May 29, 2020
Merged

Fix [PyT/DLRM] bug of model.py#517
tgrel merged 2 commits into
NVIDIA:masterfrom
t-kusanagi:dlrm-device-bug

Conversation

@t-kusanagi

Copy link
Copy Markdown
Contributor

I executed following command.

python3 -m dlrm.scripts.main --mode train --max_steps 500 --benchmark_warmup_steps 250 --synthetic_dataset --print_freq 1

and got this error.

(...snip)

Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/DeepLearningExamples/PyTorch/Recommendation/DLRM/dlrm/scripts/main.py", line 509, in <module>
    app.run(main)
  File "/usr/local/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "/root/DeepLearningExamples/PyTorch/Recommendation/DLRM/dlrm/scripts/main.py", line 293, in main
    train(model, loss_fn, optimizer, data_loader_train, data_loader_test, scaled_lr)
  File "/root/DeepLearningExamples/PyTorch/Recommendation/DLRM/dlrm/scripts/main.py", line 355, in train
    output = model(numerical_features, categorical_features).squeeze().float()
  File "/usr/local/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/DeepLearningExamples/PyTorch/Recommendation/DLRM/dlrm/model.py", line 220, in forward
    interaction_output = self._interaction(bottom_mlp_output, embedding_outputs, batch_size)
  File "/root/DeepLearningExamples/PyTorch/Recommendation/DLRM/dlrm/model.py", line 129, in _interaction
    (bottom_mlp_output, interaction_flat, interaction_padding), dim=1)
RuntimeError: Expected object of backend CUDA but got backend CPU for sequence element 2 in sequence argument at position #1 'tensors'

torch.cat requires inputs that are on the same device (GPU). I fixed it.

@t-kusanagi
t-kusanagi marked this pull request as ready for review May 19, 2020 00:40

@tgrel tgrel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will break the code on CPU. My preferred method of fixing this problem would be to pass device=base_device argument here:

https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/Recommendation/DLRM/dlrm/model.py#L98

@t-kusanagi Would you like to make submit this change yourself as a PR? If not I will fix it in the next release (most likely in a couple of weeks)

@t-kusanagi

Copy link
Copy Markdown
Contributor Author

Thank you for your reply.

I fixed it as you said and made sure it worked fine in my environment.

Pass device=base_device argument to self._interaction_padding.

@tgrel tgrel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me :)

@tgrel
tgrel merged commit 748b0d4 into NVIDIA:master May 29, 2020
@t-kusanagi
t-kusanagi deleted the dlrm-device-bug branch May 29, 2020 14:17
PeganovAnton pushed a commit to PeganovAnton/DeepLearningExamples that referenced this pull request Sep 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants