add spectral normalization [pytorch] #6929

crcrpar · 2018-04-25T03:19:57Z

related to #5027.
This PR aims at implementing Spectral Normalization in a way similar to torch.nn.utils.weight_norm.

soumith · 2018-04-25T18:06:55Z

@pytorchbot retest this please

torch/nn/utils/spectral_norm.py

+    return v / denom
+
+
+class SpectralNorm(object):


ezyang · 2018-04-25T19:21:47Z

@soumith Do we want this one?

soumith · 2018-04-25T19:23:29Z

We want spectral norm, yea. It's being used quite a lot these days.

torch/nn/utils/spectral_norm.py

+         \mathbf{W} &= \dfrac{\mathbf{W}}{\sigma(\mathbf{W})} \\
+         \sigma(\mathbf{W}) &= \max_{\mathbf{h}: \mathbf{h} \ne 0} \dfrac{\|\mathbf{W} \mathbf{h}\|_2}{\|\mathbf{h}\|_2}
+
+    Spectral normalization stabilize the training of discriminators(critics)


torch/nn/utils/spectral_norm.py

+    Spectral normalization stabilize the training of discriminators(critics)
+    in GANs by rescaling the weight tensor by spectral norm "sigma" of the
+    weight matrix calculated by power iteration method. If the dimension of the
+    weight tensor is greater than 2, reahaped to 2D in power iteration method


torch/nn/utils/spectral_norm.py

+    def compute_weight(self, module):
+        weight = module._parameters[self.name + '_org']
+        u = module._buffers[self.name + '_u']
+        height, _cuda = weight.size(0), weight.is_cuda


torch/nn/utils/spectral_norm.py

+from torch.nn.parameter import Parameter
+
+
+def l2normalize(v, eps=1e-12):


torch/nn/utils/spectral_norm.py

        for _ in range(self.n_power_iterations):
-            v = l2normalize(torch.matmul(weight_mat.t(), u), self.eps)
-            u = l2normalize(torch.matmul(weight_mat, v), self.eps)
+            v = normalize(torch.matmul(weight_mat.t(), u), dim=0, eps=self.eps)


torch/nn/utils/spectral_norm.py

+
+    Spectral normalization stabilizes the training of discriminators (critics)
+    in GANs by rescaling the weight tensor by spectral norm "sigma" of the
+    weight matrix calculated by power iteration method. If the dimension of the


ezyang · 2018-04-30T03:09:34Z

@fmassa @ssnl is this ok to merge now?

ssnl

Code looks solid. There are still some minor nits around doc and comments. I think this is good to go once those are fixed.

torch/nn/utils/spectral_norm.py

+    Spectral normalization stabilizes the training of discriminators (critics)
+    in GANs by rescaling the weight tensor with spectral norm :math:`\sigma` of
+    the weight matrix calculated using power iteration method. If the dimension
+    of the weight tensor is greater than 2, reshaped to 2D in power iteration method


torch/nn/utils/spectral_norm.py

+    spectral norm and rescales weight before every :meth:`~Module.forward`
+    call.
+
+    See https://arxiv.org/abs/1802.05957


torch/nn/utils/spectral_norm.py

+    r"""Removes the spectral normalization reparameterization from a module.
+
+    Args:
+        module (nn.Modue): containing module


torch/nn/utils/spectral_norm.py

+        weight = module._parameters[name]
+        height = weight.size(0)
+
+        u = normalize(weight.data.new(height).normal_(0, 1), dim=0, eps=fn.eps)


torch/nn/utils/spectral_norm.py

+         \sigma(\mathbf{W}) &= \max_{\mathbf{h}: \mathbf{h} \ne 0} \dfrac{\|\mathbf{W} \mathbf{h}\|_2}{\|\mathbf{h}\|_2}
+
+    Spectral normalization stabilizes the training of discriminators (critics)
+    in GANs by rescaling the weight tensor with spectral norm :math:`\sigma` of


torch/nn/utils/spectral_norm.py

+        u = module._buffers[self.name + '_u']
+        height = weight.size(0)
+        weight_mat = weight.view(height, -1)
+        for _ in range(self.n_power_iterations):


torch/nn/utils/spectral_norm.py

+        name (str, optional): name of weight parameter
+        n_power_iterations (int, optional): number of power iterations to
+            calculate spectal norm
+        eps (float, optional): epsilon for numerical stability


fix typos add comments related to power iteration and epsilon update link to the paper make some comments specific

ssnl

LGTM except the typo

torch/nn/utils/spectral_norm.py

        weight_mat = weight.view(height, -1)
        for _ in range(self.n_power_iterations):
+            # Spectral norm of weight equals to `u^T W v`, where `u` and `v`
+            # are the first left and right singular vecors.


ssnl

LGTM. @fmassa , do you want to take an extra look?

ssnl · 2018-05-01T09:00:39Z

Thank you! @crcrpar

crcrpar · 2018-05-01T09:07:19Z

Thank you for your very kind reviews and suggestions!

* initial commit for spectral norm * fix comment * edit rst * fix doc * remove redundant empty line * fix nit mistakes in doc * replace l2normalize with F.normalize * fix chained `by` * fix docs fix typos add comments related to power iteration and epsilon update link to the paper make some comments specific * fix typo

torch/nn/utils/spectral_norm.py

+            u = normalize(torch.matmul(weight_mat, v), dim=0, eps=self.eps)
+
+        sigma = torch.dot(u, torch.matmul(weight_mat, v))
+        weight.data /= sigma


crcrpar added 2 commits April 25, 2018 12:10

initial commit for spectral norm

e5f29f3

fix comment

2fc2749

crcrpar requested review from apaszke, colesbury, ezyang, gchanan, soumith and zdevito as code owners April 25, 2018 03:19

crcrpar changed the title ~~add spectral normalization~~ add spectral normalization [pytorch] Apr 25, 2018

edit rst

f97e236

ezyang reviewed Apr 25, 2018

View reviewed changes

torch/nn/utils/spectral_norm.py

return v / denom

class SpectralNorm(object):

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

crcrpar added 2 commits April 26, 2018 10:06

fix doc

4213f5c

remove redundant empty line

41f901a

ssnl reviewed Apr 26, 2018

View reviewed changes

fix nit mistakes in doc

5620873

fmassa reviewed Apr 27, 2018

View reviewed changes

torch/nn/utils/spectral_norm.py Outdated

from torch.nn.parameter import Parameter

def l2normalize(v, eps=1e-12):

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

replace l2normalize with F.normalize

f83f867

fmassa reviewed Apr 27, 2018

View reviewed changes

ssnl reviewed Apr 27, 2018

View reviewed changes

fix chained by

3ea9271

ssnl reviewed Apr 30, 2018

View reviewed changes

fix docs

76f8956

fix typos add comments related to power iteration and epsilon update link to the paper make some comments specific

ssnl reviewed Apr 30, 2018

View reviewed changes

fix typo

540b4b5

ssnl approved these changes Apr 30, 2018

View reviewed changes

ezyang approved these changes Apr 30, 2018

View reviewed changes

ssnl merged commit ba04633 into pytorch:master May 1, 2018

ssnl mentioned this pull request May 1, 2018

[feature request]: Spectral Normalization #5027

Closed

ferrine reviewed Jul 27, 2018

View reviewed changes

torch/nn/utils/spectral_norm.py

u = normalize(torch.matmul(weight_mat, v), dim=0, eps=self.eps)

sigma = torch.dot(u, torch.matmul(weight_mat, v))

weight.data /= sigma

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

crcrpar deleted the spectral-norm branch April 1, 2019 23:57

ezyang added the open source label Jun 24, 2019

		from torch.nn.parameter import Parameter


		def l2normalize(v, eps=1e-12):

add spectral normalization [pytorch] #6929

add spectral normalization [pytorch] #6929

Uh oh!

Conversation

crcrpar commented Apr 25, 2018

Uh oh!

soumith commented Apr 25, 2018

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

ezyang commented Apr 25, 2018

Uh oh!

soumith commented Apr 25, 2018

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

ezyang commented Apr 30, 2018

Uh oh!

ssnl left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

ssnl left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!