Whenever I attempt to run the code in multi-gpu setup I immediately get this error:

RuntimeError: element 0 of variables does not require grad and does not have a grad_fn

The error happens when I initiate loss.backward() and it's only specific to multi-gpu setup, as the code works fine on either GPU alone. Here's a snippet for context:

inputs = Variable(inputs).cuda()
model(inputs, labels)
loss = model.loss(inputs, labels)
loss.backward() # <--- error here
optimizer.step()
optimizer.zero_grad()

Environment details:

• OS: Ubuntu 17.10
• PyTorch version: 0.4.0a0+15cc24a
• How you installed PyTorch: source
• Python version: 3.6.4
• CUDA/cuDNN version: 9.0 / 7.0.5
• GPU models and configuration: 2x 1080 Ti (same model)
• GCC version: 4.8.5