Will there be a torch.nn.Swish module as well? (hopefully with inplace = True support)

I think that if you want to have the full nn experience, we should add it to nn.functional and a Module that uses the functional version yes.
But if you want to leave it as a follow up up to keep this one small as it is a first task, it is fine as well!

Please modify this pull request to rename the swish to the SiLU.

The swish was originally coined the "SiLU" in https://arxiv.org/pdf/1606.08415.pdf and https://arxiv.org/abs/1702.03118 long before the swish paper. Renaming other peoples' exact same ideas is unacceptable, as we all know.
A recent discussion and a recent jax and tensorflow issue show that we should call this the SiLU. In light of recent efforts to make the ML community more equitable and fair, this is a no-brainer.
PyTorch hasn't added the swish precisely because its renaming was unfair (just search previous pull requests). Changing the swish to the SiLU, as Jax just did and tensorflow will do, will put this problem to rest.

Here's a possible docstring:

    r"""silu(input) -> Tensor
    Applies element-wise the function
    :math:`\text{SiLU}(x) = x * \sigma(x)`
    where :math:`\sigma(x)` is the logistic sigmoid.
    See `Gaussian Error Linear Units (GELUs) <https://arxiv.org/abs/1606.08415>`_ where the SiLU (Sigmoid Linear Unit) was originally coined,
    and see `Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning <https://arxiv.org/abs/1702.03118>`_ and `Swish: a Self-Gated Activation Function <https://arxiv.org/abs/1710.05941v1>`_ where the SiLU was experimented with later.
    """

💊 CI failures summary and remediations

As of commit 20ea162 (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

That sounds like a good point @TFUsers

@heitorschueroff could you make the following changes then:

• Change all the namings in the current PR from swish to silu.
• Add in torch/functional.py a new function like:

def swish(inp):
  """
  Some doc explaining it is called silu here and in torch.nn
  """
  return torch.silu(inp)

This is just to make sure that people looking for swish in our documentation page hit something that points them to what they're looking for.

Also the doc is missing from your PR:

• Add entries for silu and silu_ in torch/_tensor_docs.py similar to the sigmoid ones.
• Add entries for silu in torch/_torch_docs.py similar to the sigmoid one.
• Make sure to add entries in docs/source/torch.rst next to sigmoid for both swish and silu.
• Make sure to add entries in docs/source/tensor.rst for silu and silu_.

CC @mruberry: we have an alias mechanism, right? That we can use to alias swish to silu.

CC @mruberry: we have an alias mechanism, right? That we can use to alias swish to silu.

Not yet. File a follow-up issue and assign it to me and I'll add it when we finalize aliasing in a few weeks.

this is related to pytorch_module: nn, but I don't think we need to make this a method; relu isn't a method, for example.

@gchanan
relu has torch.relu, check here

Should we remove python_module:nn and just provide torch.silu?

Originally torch.gelu (the activation function used in BERT) was provided, but then it became F.gelu.

To be consistent, let's do nn.functional.silu (F.silu)

@heitorschueroff merged this pull request in 75a4862.

Did this not make it into 1.6 in the end?

The original branch cut happened on the 23rd of July IIRC. And then only important bugfix are cherry picked into the release.
So I don't think so :/