Issue description

This is probably halfway between feature request and bug report.

When doing a forward pass on a GPU LSTM layer with a GPU input tensor but a CPU hidden state tensor, the argument type check doesn't trip, a CUDNN_STATUS_EXECUTION_FAILED is raised, and the CUDA stack is left in a corrupted state (.to('cuda') raises error 77).

While running these operations is obviously stupid, doing stupid stuff shouldn't leave the CUDA environment in a broken mess.

Code example

import torch
from torch import nn

layer = nn.LSTM(10, 100, batch_first=True)
layer.to('cuda')
i = torch.randn(1, 30, 10).to('cuda')
h = [torch.zeros(1, 1, 100).to('cuda'), torch.zeros(1, 1, 100).to('cuda')]
layer(i, h) # works fine
layer(i, [x.to('cpu') for x in h]) # raises cudnn error
i # accessing i raises cuda runtime error

System Info

Scientific Linux 6.9, pytorch 0.4.0, manually compiled against cuda 8 with cudnn 7.1 because of an ancient glibc in the distribution.