[ready] Layer Normalization #4922

ssnl · 2018-01-30T00:13:24Z

Commits:

Renames ATen/Check.h to ATen/TensorUtils.h with an additional method at::maybe_data_ptr added. Make UndefinedTensor's data_ptr() return nullptr #4851
THNN BN code so that running_mean and running_var can be optional when training=True. Feature request: instance norm without the computation of running statistics #4509
ATen changes and cuDNN changes. Still Feature request: instance norm without the computation of running statistics #4509
Python nn.* changes, including changing InstanceNorm*d's use_running_stats from Fix setting using running stats in InstanceNorm*d #4444 to the new option track_running_stats on BN. Improves IN and BN docs.
Adds test for the new option for IN and BN. Improves other IN tests.
Adds Layer Normalization [Feature Request] Layer Normalization #1959 .
Fixes LRN doc.
Functional interface for IN and LN.
Tests for LN.
Fix BN double backward returning undefined tensor when it shouldn't.
Fix Jit tests that use wrong dim inputs for BN
Add/Improve BN, IN and LN GPU tests with half.
Update IN BN docs to be consistent with conv notation; Fix onnx failures.

pytorchbot · 2018-01-30T00:13:25Z

@ssnl, thanks for your PR! We identified @zdevito to be a potential reviewer.

torch/nn/modules/layernorm.py

Kaixhin · 2018-01-30T03:51:28Z

Sorry but could you also fix the LRN docs with this PR? Just need to remove the extra colon here.

ssnl · 2018-01-31T20:37:37Z

@pytorchbot retest this please

ssnl · 2018-01-31T20:57:33Z

@pytorchbot retest this please

ssnl · 2018-02-01T00:43:50Z

@pytorchbot retest this please

test/test_nn.py

        self.assertEqual(grad1, grad2)

+        # track_running_stats=False
+        module = nn.BatchNorm1d(3, track_running_stats=False).type(test_type)


torch/nn/functional.py

-    """Applies instance normalization over an input. The implementation is
-    based on batch_norm, in which we do reshape, batchnorm, and reshape again.
+def batch_norm(input, running_mean, running_var, weight=None, bias=None,
+               training=False, momentum=0.1, eps=1e-5):


torch/nn/modules/batchnorm.py

    Args:
        num_features: num_features from an expected input of size
-            `[batch_size x num_features (x width)]`
+            :math:`(N, C, L)` or :math:`(N, L)`


torch/nn/modules/instancenorm.py

    Args:
        num_features: num_features from an expected input of size
-            `[batch_size x num_features x width]`
+            :math:`(N, C, L)` or :math:`(N, L)`


zou3519

LGTM!

ruotianluo · 2018-02-22T20:53:37Z

The docs of Layernorm and LRN have these:
are these expected?

http://pytorch.org/docs/master/nn.html?highlight=layernorm#torch.nn.LayerNorm.forward
http://pytorch.org/docs/master/nn.html?highlight=layernorm#torch.nn.LocalResponseNorm.forward

ssnl · 2018-02-22T20:57:27Z

@ruotianluo definitely not. This phenomenon is not unique to these two classes through. I'll figure out what's wrong.

meder411 · 2018-02-23T02:00:09Z

Is there a particular reason that the normalized_shape input must be a tensor? For fully connected layers or other 1D inputs, it seems that the constructor ought to be able to accept an integer. As of now, I am using

nn.LayerNorm(torch.LongTensor([256])

but why not

nn.LayerNorm(256)

as one would with BatchNorm. From the paper, there doesn't seem to be any reason you wouldn't use Layer Normalization in this way. It's a minor change, but it may be a more seemless interface.

ssnl · 2018-02-23T06:15:25Z

@meder411 It doesn't need to be a tensor. It can be a list, tuple, torch.Size, etc., basically anything that can be accepted by torch.Size's constructor. In fact, the doc never suggested passing in a tensor, so I'm not sure where you get that idea.

The idea is to be able to normalize over multiple dimensions. Hence the argument is called normalized_shape. The doc explains it pretty clearly I think.

Kaixhin · 2018-02-23T14:33:18Z

@meder411 While the docs do look correct to me for describing normalized_shape, having it accept an integer for the most common use-case by far does seem worthwhile. If you're willing to submit a PR for this that'd be great, otherwise raise an issue (please tag me within) and someone will address it.

torch/nn/modules/normalization.py

+            self.running_mean.zero_()
+            self.running_var.fill_(1)
+        if self.elementwise_affine:
+            self.weight.data.uniform_()


* at::maybe_data_ptr and Check.h => TensorUtils.h * THNN support for optional BN running_* * ATen support for optional BN running_* * Python nn.* support for optional BN running_*; Improve IN and BN doc * Add tests for IN and BN new option * Layer Norm * Fix LRN doc * functional interface for LN and IN * Layer norm tests * fix BN double backward returning undefined tensors * fix jit test using wrong dim inputs for BN * add/improve BN, IN and LN GPU tests with half type * Udpate docs to be consistent with Conv notation Fix onnx Clarified onnx symbokic wrapper * fix typo * Address comments

JustinLin610 · 2018-03-30T09:56:45Z

How can I use it with nn.LSTM or nn.GRU? it will be much more helpful if LN can be used with the two classes

ssnl · 2018-03-30T15:40:14Z

@JustinLin610 Using the *Cell classes is the only way.

calclavia · 2018-05-10T17:19:02Z

@ssnl I don't think there's a way to modify the cell? The only way would be to copy and paste the code for the nn.GRUCell and then add layernorm manually. Would be nice if there's an option to easily enable it...

jinserk · 2018-09-05T03:36:10Z

I'd like to apply LayerNorm to multi-layer LSTM but have no idea how to use LSTMCell classes. @ssnl, could you let me know any simple example for this? LayerNorm is quite new to PyTorch so it is difficult to find a good example for this..

MBAnslow · 2019-04-09T10:28:52Z

@ssnl I agree with jinserk. I've looked far and wide for a feature full example of LSTMs and GRUs that support layer normalisation including bidirectionality and multiple layers and I haven't really found anything. There are some incomplete LSTM versions around but they don't seem to be feature complete or optimised very well.

zou3519 · 2019-04-09T17:12:01Z

@MBAnslow, @jinserk see https://github.com/pytorch/benchmark/blob/master/rnns/fastrnns/custom_lstms.py for some examples

onnxbot-worker-1 mentioned this pull request Jan 30, 2018

[auto] pytorch-pr-4922 onnxbot/onnx-fb-universe#469

Closed

jekbradbury reviewed Jan 30, 2018

View reviewed changes

This was referenced Jan 30, 2018

Add LayerNorm (1D only) #2019

Closed

LayerNorm #2112

Closed

ssnl force-pushed the layer_norm branch 2 times, most recently from f989db0 to c68b7e5 Compare January 30, 2018 17:47

ssnl mentioned this pull request Jan 30, 2018

Fix BatchNorm2d test using inputs of wrong number of dimensions onnxbot/onnx-fb-universe#475

Merged

ssnl force-pushed the layer_norm branch 3 times, most recently from 142707f to 1f3df3d Compare January 30, 2018 21:23

ssnl changed the title ~~[WIP] Layer Normalization~~ [ready] Layer Normalization Jan 30, 2018

ssnl changed the title ~~[ready] Layer Normalization~~ [WIP] Layer Normalization Jan 30, 2018

ssnl force-pushed the layer_norm branch 12 times, most recently from 45b8220 to 9beb186 Compare January 31, 2018 20:15

ssnl force-pushed the layer_norm branch from 9beb186 to e282e75 Compare January 31, 2018 22:38

fix typo

155faeb

zou3519 reviewed Feb 20, 2018

View reviewed changes

test/test_nn.py

self.assertEqual(grad1, grad2)

# track_running_stats=False

module = nn.BatchNorm1d(3, track_running_stats=False).type(test_type)

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

zou3519 reviewed Feb 20, 2018

View reviewed changes

torch/nn/modules/batchnorm.py Outdated

Args:

num_features: num_features from an expected input of size

`[batch_size x num_features (x width)]`

:math:`(N, C, L)` or :math:`(N, L)`

This comment was marked as off-topic.

Sign in to view

zou3519 reviewed Feb 20, 2018

View reviewed changes

torch/nn/modules/instancenorm.py Outdated

Args:

num_features: num_features from an expected input of size

`[batch_size x num_features x width]`

:math:`(N, C, L)` or :math:`(N, L)`

This comment was marked as off-topic.

Sign in to view

ssnl force-pushed the layer_norm branch from 72e64ad to c9e81eb Compare February 20, 2018 19:27

Address comments

b1359c8

ssnl force-pushed the layer_norm branch from c9e81eb to b1359c8 Compare February 20, 2018 19:40

zou3519 approved these changes Feb 21, 2018

View reviewed changes

soumith merged commit 1848cad into pytorch:master Feb 22, 2018

ssnl deleted the layer_norm branch February 22, 2018 16:57

Kaixhin mentioned this pull request Feb 22, 2018

[Feature Request] Layer Normalization #1959

Closed

avati reviewed Feb 23, 2018

View reviewed changes

ssnl mentioned this pull request Mar 7, 2018

[bug?] Problem with load_state_dict after installing latest pytorch by source #5602

Closed

ssnl mentioned this pull request Apr 28, 2018

Make UndefinedTensor's data_ptr() return nullptr #4851

Closed

ezyang added the open source label Jun 24, 2019

cdeln mentioned this pull request Jun 11, 2024

Feature request: instance norm without the computation of running statistics #4509

Closed

[ready] Layer Normalization #4922

[ready] Layer Normalization #4922

Uh oh!

Conversation

ssnl commented Jan 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorchbot commented Jan 30, 2018

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

Kaixhin commented Jan 30, 2018

Uh oh!

ssnl commented Jan 31, 2018

Uh oh!

ssnl commented Jan 31, 2018

Uh oh!

ssnl commented Feb 1, 2018

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

zou3519 left a comment

Choose a reason for hiding this comment

Uh oh!

ruotianluo commented Feb 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ssnl commented Feb 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

meder411 commented Feb 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ssnl commented Feb 23, 2018

Uh oh!

Kaixhin commented Feb 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

JustinLin610 commented Mar 30, 2018

Uh oh!

ssnl commented Mar 30, 2018

Uh oh!

calclavia commented May 10, 2018

Uh oh!

jinserk commented Sep 5, 2018

Uh oh!

ssnl commented Jan 30, 2018 •

edited

Loading

ruotianluo commented Feb 22, 2018 •

edited

Loading

ssnl commented Feb 22, 2018 •

edited

Loading

meder411 commented Feb 23, 2018 •

edited

Loading

Kaixhin commented Feb 23, 2018 •

edited

Loading