Implement torch.pinverse : Pseudo-inverse #9052

vishwakftw · 2018-06-29T22:22:58Z

Used SVD to compute.
Tests in test_autograd, test_cuda and test_torch
Doc strings in _torch_docs.py and _tensor_docs.py

1. Used SVD to compute. 2. Tests in test_cuda and test_torch 3. Doc strings in _torch_docs.py and _tensor_docs.py Closes pytorch#6187

…erse

aten/src/ATen/native/LinearAlgebra.cpp

 }

+Tensor pinverse(const Tensor& self) {
+  if (!at::isFloatingType(self.type().scalarType()) ||


aten/src/ATen/native/LinearAlgebra.cpp

+  }
+  Tensor U, S, V;
+  std::tie(U, S, V) = self.svd();
+  Tensor S_pseudoinv = at::where(S != 0.0, S.reciprocal(), at::zeros({}, self.type()));


1. Use AT_CHECK 2. Use at::zeros({}, self.options);

vishwakftw · 2018-06-30T14:50:52Z

Not sure why the CUDA tests are failing. @ssnl

aten/src/ATen/native/LinearAlgebra.cpp

+           "of floating types");
+  Tensor U, S, V;
+  std::tie(U, S, V) = self.svd();
+  Tensor S_pseudoinv = at::where(S != 0.0, S.reciprocal(), at::zeros({}, self.options()));


vishwakftw · 2018-06-30T18:38:23Z

A few comments: @ssnl

Naming - pinverse instead of pinv: PyTorch has inverse for matrix inverse, hence the convention.
Tests in test_autograd.py: I had a thought about it initially, but pseudoinverses are discontinuous. Derivatives exist for a constant rank only. Ref: https://mathoverflow.net/questions/25778/analytical-formula-for-numerical-derivative-of-the-matrix-pseudo-inverse

ssnl · 2018-06-30T18:51:12Z

Very valid reasons. Thanks for your explanation!

Our current general goal is to align with numpy interface. Unfortunately we didn't have this guideline before. I believe that we probably want to alias inverse with inv in future. @soumith , should we just name this as pinv? Or have both pinv and pinverse as its aliases?
Thanks for the reference. I agree. However, the current implementation is still backprop-able, so users may attempt to backprop. How about this: let's test the following function:
```
assumption: 
  1. m <= n
  2. two fixed orthonormal matrices U and V
input: x of shape [m]
output: S = torch.cat([x, torch.zeros(n - m)], 0).diag(); return U @ S @ V
```
If we test it on sufficiently large positive input x, it should guarantee constant rank m.
Also, given its instability, I suggest we add a .. note:: in its doc (just as we did for svd). I think the note probably should also mention double backward instability from svd (similar to the note in det).

Let me know what you think!

ssnl · 2018-06-30T19:01:42Z

I think, svd has similar issue with U and V matrices when the input is not full rank, but we tested it in the "good" case anyways. :D

vishwakftw · 2018-06-30T19:54:52Z

@ssnl Sounds good to me. I have added those tests in test_autograd.

However, the build is still failing. Could it be a precision problem with GPUs?

ssnl · 2018-06-30T19:56:13Z

It could be. Let's wait until CI finishes.

test/test_autograd.py

    s = torch.arange(1., l + 1).mul_(1.0 / (l + 1))
    return u.mm(torch.diag(s)).mm(v.t())

+def random_matrix_large_singular_value(m, n):


ssnl · 2018-06-30T20:38:52Z

how big is the difference in CUDA failure?

vishwakftw · 2018-06-30T21:42:58Z

About 5.5 - 6 .

Summary: Fixes #9079 There is room for speed-up for both functions (see #9083), but let's get this in to unblock #9052 . Closes #9082 Reviewed By: ezyang Differential Revision: D8711687 Pulled By: SsnL fbshipit-source-id: f043a9bf55cb6aec5126c3331d35761f7aa3f8e3

vishwakftw · 2018-07-02T12:27:46Z

Mac OS build failure seems unrelated.

vishwakftw · 2018-07-02T17:12:14Z

@pytorchbot retest this please

test/test_autograd.py

+            return M.pinverse()
+
+        gradcheck(func, [torch.rand(m) + 1])
+        gradcheck(func, [torch.rand(m) + 10])


facebook-github-bot

@ssnl has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@ssnl has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

ssnl · 2018-07-02T21:22:10Z

@pytorchbot retest this please

ssnl · 2018-07-02T21:55:46Z

@pytorchbot retest this please

vishwakftw · 2018-07-03T17:28:30Z

@pytorchbot retest this please

test/test_autograd.py

        run_test((10,), 0)

+    def test_pinverse(self):
+        m, n = 5, 10


facebook-github-bot

@ssnl has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@ssnl is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@ssnl has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

ssnl · 2018-07-03T22:39:56Z

@vishwakftw The PR is great! Thanks for doing this :).

There are just some internal problems that I'm trying to fix before I can merge this.

Summary: 1. Used SVD to compute. 2. Tests in test_autograd, test_cuda and test_torch 3. Doc strings in _torch_docs.py and _tensor_docs.py Closes #6187 Closes pytorch/pytorch#9052 Reviewed By: soumith Differential Revision: D8714628 Pulled By: SsnL fbshipit-source-id: 7e006c9d138b9f49e703bd0ffdabe6253be78dd9

Summary: Fixes pytorch#9079 There is room for speed-up for both functions (see pytorch#9083), but let's get this in to unblock pytorch#9052 . Closes pytorch#9082 Reviewed By: ezyang Differential Revision: D8711687 Pulled By: SsnL fbshipit-source-id: f043a9bf55cb6aec5126c3331d35761f7aa3f8e3

Summary: 1. Used SVD to compute. 2. Tests in test_autograd, test_cuda and test_torch 3. Doc strings in _torch_docs.py and _tensor_docs.py Closes pytorch#6187 Closes pytorch#9052 Reviewed By: soumith Differential Revision: D8714628 Pulled By: SsnL fbshipit-source-id: 7e006c9d138b9f49e703bd0ffdabe6253be78dd9

mscipio · 2018-12-18T10:15:59Z

It could be useful to adapt torch.pinverse to work with batches of tensors like done for torch.inverse in v1.0.0

vishwakftw · 2018-12-18T10:19:32Z

Hi @mscipio, batching for pinverse is blocked due to unavailability of batched SVD, which is used internally to compute the pinverse.

Implement torch.pinv : Pseudo-inverse

60eed6a

1. Used SVD to compute. 2. Tests in test_cuda and test_torch 3. Doc strings in _torch_docs.py and _tensor_docs.py Closes pytorch#6187

vishwakftw requested review from apaszke, colesbury, ezyang, gchanan, soumith, ssnl, zdevito and zou3519 as code owners June 29, 2018 22:22

vishwakftw added 4 commits June 29, 2018 18:52

Fix nit in tests

d985492

Fix one more nit in test

d4b601a

Fix tests finally, sorry

332b0be

Add actual Moore-Penrose conditions in the tests, rename pinv to pinv…

33c75a0

…erse

vishwakftw changed the title ~~Implement torch.pinv : Pseudo-inverse~~ Implement torch.pinverse : Pseudo-inverse Jun 30, 2018

ssnl reviewed Jun 30, 2018

View reviewed changes

vishwakftw added 2 commits June 30, 2018 08:31

Address comments

bb047cc

1. Use AT_CHECK 2. Use at::zeros({}, self.options);

Invert bool expressions

1bb5614

vishwakftw commented Jun 30, 2018

View reviewed changes

Use NumPy-based cutoff for removing smaller singular values

494c2a1

Add tests in test_autograd

ac0e33c

ssnl reviewed Jun 30, 2018

View reviewed changes

Modify test as per suggestion, add note in the docs

6117801

ssnl mentioned this pull request Jul 1, 2018

Fix MAGMA svd and eig #9082

Closed

Merge branch 'master' into pinverse

ef914c9

ssnl reviewed Jul 2, 2018

View reviewed changes

test/test_autograd.py

return M.pinverse()

gradcheck(func, [torch.rand(m) + 1])

gradcheck(func, [torch.rand(m) + 10])

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

facebook-github-bot reviewed Jul 2, 2018

View reviewed changes

add gradgradcheck

16720f3

facebook-github-bot reviewed Jul 2, 2018

View reviewed changes

ssnl reviewed Jul 3, 2018

View reviewed changes

test/test_autograd.py

run_test((10,), 0)

def test_pinverse(self):

m, n = 5, 10

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

Update test_autograd.py

6bac866

facebook-github-bot reviewed Jul 3, 2018

View reviewed changes

facebook-github-bot closed this in 14cbd9a Jul 5, 2018

vishwakftw deleted the pinverse branch July 5, 2018 16:16

ezyang added open source labels Jun 24, 2019

Implement torch.pinverse : Pseudo-inverse #9052

Implement torch.pinverse : Pseudo-inverse #9052

Uh oh!

Conversation

vishwakftw commented Jun 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

vishwakftw commented Jun 30, 2018

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

vishwakftw commented Jun 30, 2018

Uh oh!

ssnl commented Jun 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ssnl commented Jun 30, 2018

Uh oh!

vishwakftw commented Jun 30, 2018

Uh oh!

ssnl commented Jun 30, 2018

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

ssnl commented Jun 30, 2018

Uh oh!

vishwakftw commented Jun 30, 2018

Uh oh!

vishwakftw commented Jul 2, 2018

Uh oh!

vishwakftw commented Jul 2, 2018

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

ssnl commented Jul 2, 2018

Uh oh!

ssnl commented Jul 2, 2018

Uh oh!

vishwakftw commented Jul 3, 2018

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

ssnl commented Jul 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mscipio commented Dec 18, 2018

Uh oh!

vishwakftw commented Jun 29, 2018 •

edited

Loading

ssnl commented Jun 30, 2018 •

edited

Loading

ssnl commented Jul 3, 2018 •

edited

Loading