🐛 Describe the bug

I'm sorry this is going to be very generic, but I've been occasionally testing InvokeAI, a Stable Diffusion front end with the pytorch nightlies to see if the speed lost since 13.0 has returned.
Today is the first time I've tested in a while and it was broken, generating colour noise not an image,
so I've been going back through the nightlies to see where the breakage is.

torch-1.14.0.dev20221102 works,
torch-1.14.0.dev20221103 works if #87974 is reverted
torch-1.14.0.dev20221104 is broken if #87974 is reverted
torch-1.14.0.dev20221105 is broken

That in theory narrows it down to #82809 and #88319 if its a MPS only issue

Versions

PyTorch version: 1.14.0.dev20221103
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.0.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.202)
CMake version: version 3.25.0
Libc version: N/A

Python version: 3.10.8 (main, Nov 9 2022, 16:53:27) [Clang 12.0.5 (clang-1205.0.22.9)] (64-bit runtime)
Python platform: macOS-13.0.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.4
[pip3] pytorch-lightning==1.7.7
[pip3] torch==1.14.0.dev20221103
[pip3] torch-fidelity==0.3.0
[pip3] torchdiffeq==0.2.3
[pip3] torchmetrics==0.10.0
[pip3] torchsde==0.2.5
[pip3] torchvision==0.15.0.dev20221128
[conda] Could not collect

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev