Why did this have to change?

This was a bug exposed by the new checks in masked_fill. It first broadcasts self.logits with value, then writes to the broadcasted (i.e. zero-strided) array in a way that overwrites entire rows instead of only where value == 0.

Why did this have to change?

The next line writes into a broadcasted array:
expected[expected == float('inf')] = 0.

This seems BC-breaking. Even if expected is a broadcasted array, expected[expected == float('inf')] = 0. produces a correct result because it is setting the right elements to 0

Note that this is a masked_fill_ which is exactly the operation in the original issue. To preserve the correct uses, like

expected[expected == float('inf')] = 0.

then we could inspect the mask along the broadcasted dimensions and only allow the fill if the mask is constant along broadcasted dimensions. Maybe deprecating that behavior as well.

I did an audit of TensorIterator::unary_op, TensorIterator::binary_op, and TensorIterator::nullary_op cases in the code that use the default. The default before this PR was check_mem_overlap=False, but after this PR, those are True. I found a few that we didn't test, so it would be nice to add tests for them (to check that we've added the checks for both CPU and CUDA:

For: TensorIterator::unary_op

• elu: test in-place variant:

  pytorch/aten/src/ATen/native/Activation.cpp

  Lines 111 to 117 in 2b7108a

  	Tensor & elu_(
  	Tensor & self,
  	Scalar alpha,
  	Scalar scale,
  	Scalar input_scale) {
  	return at::elu_out(self, self, alpha, scale, input_scale);
  	}

• hardswish: test in-place variant:

  pytorch/aten/src/ATen/native/Activation.cpp

  Lines 150 to 154 in 2b7108a

  	Tensor& hardswish_out(Tensor& result, const Tensor& self) {
  	auto iter = TensorIterator::unary_op(result, self);
  	hardswish_stub(iter.device_type(), iter);
  	return result;
  	}

• leaky_relu: test in-place variant

  pytorch/aten/src/ATen/native/Activation.cpp

  Lines 713 to 735 in 2b7108a

  	Tensor& leaky_relu_out(
  	Tensor& result,
  	const Tensor& self,
  	Scalar negval) {
  	auto iter = TensorIterator::unary_op(result, self);
  	leaky_relu_stub(iter.device_type(), iter, negval);
  	return result;
  	}
  	Tensor leaky_relu(
  	const Tensor& self,
  	Scalar negval) {
  	Tensor result;
  	auto iter = TensorIterator::unary_op(result, self);
  	leaky_relu_stub(iter.device_type(), iter, negval);
  	return iter.output();
  	}
  	Tensor & leaky_relu_(
  	Tensor & self,
  	Scalar neg_val) {
  	return at::leaky_relu_out(self, self, neg_val);
  	}

For TensorIterator::binary_op:

• index_add_:

  pytorch/aten/src/ATen/native/TensorAdvancedIndexing.cpp

  Line 375 in 2b7108a

  auto iter = TensorIterator::binary_op(selfSlice, selfSlice, sourceSlice);

  has two paths.
• atan2: test out variant and in-place variant:

  pytorch/aten/src/ATen/native/BinaryOps.cpp

  Lines 337 to 350 in 2b7108a

  	Tensor& atan2_out(Tensor& result, const Tensor& self, const Tensor& other) {
  	auto iter = TensorIterator::binary_op(result, self, other);
  	atan2_stub(iter.device_type(), iter);
  	return result;
  	}
  	Tensor atan2(const Tensor& self, const Tensor& other) {
  	Tensor result = at::empty({0}, self.options());
  	return native::atan2_out(result, self, other);
  	}
  	Tensor& atan2_(Tensor& self, const Tensor& other) {
  	return native::atan2_out(self, self, other);
  	}

• ilshift:

  pytorch/aten/src/ATen/native/BinaryOps.cpp

  Lines 596 to 607 in 2b7108a

  	Tensor& __ilshift__(Tensor& self, const Tensor& other) {
  	auto iter = TensorIterator::binary_op(self, self, other);
  	lshift_stub(iter.device_type(), iter);
  	return self;
  	}
  	Tensor& __ilshift__(Tensor& self, Scalar other) {
  	auto wrapper = wrapped_scalar_tensor(other).toType(self.scalar_type());
  	auto iter = TensorIterator::binary_op(self, self, wrapper);
  	lshift_stub(iter.device_type(), iter);
  	return self;
  	}

• irshift:

  pytorch/aten/src/ATen/native/BinaryOps.cpp

  Lines 624 to 635 in 2b7108a

  	Tensor& __irshift__(Tensor& self, const Tensor& other) {
  	auto iter = TensorIterator::binary_op(self, self, other);
  	rshift_stub(iter.device_type(), iter);
  	return self;
  	}
  	Tensor& __irshift__(Tensor& self, Scalar other) {
  	auto wrapper = wrapped_scalar_tensor(other).toType(self.scalar_type());
  	auto iter = TensorIterator::binary_op(self, self, wrapper);
  	rshift_stub(iter.device_type(), iter);
  	return self;
  	}

• hypot: test out variant and in-place variant:

  pytorch/aten/src/ATen/native/BinaryOps.cpp

  Lines 888 to 903 in 2b7108a

  	Tensor& hypot_out(Tensor& result, const Tensor& self, const Tensor& other) {
  	auto iter = TensorIterator::binary_op(result, self, other);
  	hypot_stub(iter.device_type(), iter);
  	return result;
  	}
  	Tensor hypot(const Tensor& self, const Tensor& other) {
  	Tensor result;
  	auto iter = TensorIterator::binary_op(result, self, other);
  	hypot_stub(iter.device_type(), iter);
  	return iter.output();
  	}
  	Tensor& hypot_(Tensor& self, const Tensor& other) {
  	return at::hypot_out(self, self, other);
  	}

• nextafter: test out variant and in-place variant:

  pytorch/aten/src/ATen/native/BinaryOps.cpp

  Lines 905 to 920 in 2b7108a

  	Tensor& nextafter_out(Tensor& result, const Tensor& self, const Tensor& other) {
  	auto iter = TensorIterator::binary_op(result, self, other);
  	nextafter_stub(iter.device_type(), iter);
  	return result;
  	}
  	Tensor nextafter(const Tensor& self, const Tensor& other) {
  	Tensor result;
  	auto iter = TensorIterator::binary_op(result, self, other);
  	nextafter_stub(iter.device_type(), iter);
  	return iter.output();
  	}
  	Tensor& nextafter_(Tensor& self, const Tensor& other) {
  	return at::nextafter_out(self, self, other);
  	}

• threshold: test in-place variant:

  pytorch/aten/src/ATen/native/Activation.cpp

  Lines 380 to 392 in 2b7108a

  	Tensor threshold(const Tensor& self, Scalar threshold, Scalar value) {
  	return threshold_out(nullopt, self, threshold, value, self);
  	}
  	Tensor& threshold_(Tensor& self, Scalar threshold, Scalar value) {
  	threshold_out(make_optional(self), self, threshold, value, self);
  	return self;
  	}
  	Tensor& threshold_out(Tensor& result, const Tensor& self, Scalar threshold, Scalar value) {
  	threshold_out(make_optional(result), self, threshold, value, self);
  	return result;
  	}

In retrospect, because there are so many ops, I think this PR would be easier to review if we split it up into, e.g.:

• a PR to remove TensorIterator::unary_op's check_mem_overlap (and instead default to True)
• a PR to remove TensorIterator::binary_op's check_mem_overlap (and instead default to True)
• ...
• a PR that hits the switch for the default of check_mem_overlap in TensorIterator to True.

However, I'm fine with reviewing all the changes in one single PR (this PR) because the ultimate line count will probably still be manageable.

I will try to split this up into a clean stack of PRs. The longer each individual change takes to land, the more ops will be ported to TensorIterator with the old default and will need changing. So anything to make review go smoother should be worth it IMO.

We should add some prescriptive documentation to