clamp now has subgradient 1 at min and max #7049
Conversation
ssnl
requested review from
apaszke,
colesbury,
ezyang,
gchanan,
soumith and
zdevito
as code owners
tools/autograd/derivatives.yaml
Outdated
| self: grad * ((self >= min) * (self <= max)).type_as(grad) | ||
|
|
||
| - name: clamp_min(Tensor self, Scalar min) | ||
| self: grad * (self > min).type_as(grad) |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
|
Yes, this subtle formula tweak definitely merits a comment. A link to the issue is a good start, and maybe something like "Gradient is not defined at the boundaries, but empirically it's helpful to be able to get gradient on min and max." EDIT: And perhaps it points its way to a more general principle: if you need to return the gradient at a nondifferentiable point, and you have |
apaszke
left a comment
apaszke
left a comment
There was a problem hiding this comment.
I agree a comment would be nice. @ezyang I don't think we should commit to that principle. We could use it to guide some choices, but I don't want to have ops that will have to do an extra pass to check for boundary conditions and fill the gradients if it's expensive.
* subgradient 1 at min and max for clamp * clamp max and clamp min too * add comment
* subgradient 1 at min and max for clamp * clamp max and clamp min too * add comment
4 participants
addresses #7002