Doc note update for complex autograd #45270

anjali411 · 2020-09-24T13:50:38Z

Stack from ghstack:

Doc note update for complex autograd #45270 Doc note update for complex autograd

Differential Revision: D24203257

[ghstack-poisoned]

anjali411 · 2020-09-24T14:29:18Z

cc. @boeddeker

albanD

Looks mostly good for me.
Just small comments.

docs/source/notes/autograd.rst

[ghstack-poisoned]

dr-ci · 2020-09-24T15:06:03Z

💊 CI failures summary and remediations

As of commit 851ca95 (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 21 times.

[ghstack-poisoned]

ghstack-source-id: b4d871c Pull Request resolved: #45270

[ghstack-poisoned]

docs/source/notes/autograd.rst

zou3519 · 2020-09-24T16:52:49Z

docs/source/notes/autograd.rst

        def F(z):
            x, y = real(z), imag(z)
-            return u(x, y) + v(x, y) * 1j
+            s = u(x, y) + v(x, y) * 1j
+            return s

-where :math:`1j` is a unit imaginary number.
+where :math:`1j` is a unit imaginary number and :math:`u` and :math:`v` are :math:`ℝ^{2} → ℝ` functions.

-We define the :math:`JVP` for function :math:`F` at :math:`(x, y)` applied to a tangent
-vector :math:`c+dj \in C` as:
+We define the Vector-Jacobian Product `VJP` of :math:`F` at :math:`x + yj` for grad output vector :math:`grad\_out` as:


As a complex numbers n00b, I think this documentation is a great step to helping developers understand how to write (or update) gradient formulas to support complex numbers.

I think something else that would be helpful is if there was an Appendix somewhere that stepped through the derivation of some operator (like torch.mul, or torch.dot). When I started out understanding how complex number gradients worked, I tried to derive some formulas (e.g., for torch.mul) and compare them against the existing gradient formulas.

Perhaps that would be better for developer documentation though -- I'm not sure if the purpose of notes/autograd is for users to look at and understand what's going on, or for developers to read.

@zou3519 thanks for the feedback! I added the screenshots in the description as per your suggestion. I agree that it would be useful to step through the derivation for one of the operators. However, I am not sure if this doc note is the right place to do that. We can possibly include a gradient derivation for one of the operators here: https://pytorch.org/docs/stable/complex_numbers.html#autograd along with an example of how the gradients can be used in an optimization problem.

[ghstack-poisoned]

ghstack-source-id: f623cc0 Pull Request resolved: #45270

codecov · 2020-09-24T20:40:30Z

Codecov Report

Merging #45270 into gh/anjali411/58/base will increase coverage by 0.00%.
The diff coverage is n/a.

@@                  Coverage Diff                  @@
##           gh/anjali411/58/base   #45270   +/-   ##
=====================================================
  Coverage                 68.25%   68.25%           
=====================================================
  Files                       410      410           
  Lines                     53232    53232           
=====================================================
+ Hits                      36335    36336    +1     
+ Misses                    16897    16896    -1

Impacted Files	Coverage Δ
torch/testing/_internal/expecttest.py	`78.57% <0.00%> (+1.02%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 83d2c9a...851ca95. Read the comment docs.

boeddeker · 2020-09-24T21:30:25Z

docs/source/notes/autograd.rst

-mode automatic differentiation. Notice that d and :math:`1j` are negated in the formula above. Please look at
-the `JAX docs <https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html#Complex-numbers-and-differentiation>`_
-to get explanation for the negative signs in the formula.
+The above formula can also be verified to be correct for cross-domain functions functions.


Typo: One functions too much in functions functions.

boeddeker · 2020-09-24T21:45:05Z

docs/source/notes/autograd.rst

            x, y = real(z), imag(z)
-            return u(x, y) + v(x, y) * 1j
+            s = u(x, y) + v(x, y) * 1j
+            return s


Just an idea. You start here directly with the assumption, that the chain rule is known.
Maybe it would be better for a beginner, when you start with the core assumption:
e.g.: Interpreted the complex number as two numbers and calculate the derivative:

grad_x = dG / dx where G \in \mathcal{R}

grad_y = dG / dy where G \in \mathcal{R}

Since complex numbers can contain 2 real numbers, we can define:

grad = grad_x + 1j * grad_y

or

grad = grad_x - 1j * grad_y

And this is known as Wirtinger Calculus.

(Note: Dropping the 1/2 in eq. (3.2) of https://arxiv.org/pdf/1701.00392.pdf is usually what NNs do. At the time of writing the paper we weren't careful, because this equation is never implemented)

boeddeker · 2020-09-24T21:53:11Z

docs/source/notes/autograd.rst

+            return s

-The :math:`JVP` and :math:`VJP` for a :math:`f1: ℝ^2 → ℂ` are defined as:
+at :math:`x + yj` can be simplified to:


This is always difficult to describe. But you cannot evaluate it at x + yj, because z is real valued.
Or should this operation be C -> R -> C -> R i.e. G(F(real_to_complex(real(z))))) (Not sure, if this matches your notation. Probably is G in your case the complete function) ?

boeddeker · 2020-10-02T08:20:50Z

For the example of the complex derivation, it would be nice to extend the example with PyTorch code.
Depending on your intention, either some code to verfy your example or a custom function with a manual defined gradient.

<img width="1672" alt="Screen Shot 2020-09-24 at 1 09 05 PM" src="https://user-images.githubusercontent.com/20081078/94177334-5f2b9600-fe67-11ea-97cf-b2949153a117.png"> <img width="1673" alt="Screen Shot 2020-09-24 at 1 09 16 PM" src="https://user-images.githubusercontent.com/20081078/94177346-62bf1d00-fe67-11ea-94cc-d429b55a73b1.png"> [ghstack-poisoned]

anjali411 · 2020-10-07T17:54:43Z

For the example of the complex derivation, it would be nice to extend the example with PyTorch code.
Depending on your intention, either some code to verify your example or a custom function with a manual defined gradient.

@boeddeker - @ezyang and I rewrote the doc note and incorporated more background knowledge for complex derivatives in the updated version. you can check out how the html rendering in the PR description.

<img width="1679" alt="Screen Shot 2020-10-07 at 1 45 59 PM" src="https://user-images.githubusercontent.com/20081078/95368324-fa7b2d00-08a3-11eb-9066-2e659a4085a2.png"> <img width="1673" alt="Screen Shot 2020-10-07 at 1 46 10 PM" src="https://user-images.githubusercontent.com/20081078/95368332-fbac5a00-08a3-11eb-9be5-77ce6deb8967.png"> <img width="1667" alt="Screen Shot 2020-10-07 at 1 46 30 PM" src="https://user-images.githubusercontent.com/20081078/95368337-fe0eb400-08a3-11eb-80a2-5ad23feeeb83.png"> <img width="1679" alt="Screen Shot 2020-10-07 at 1 46 48 PM" src="https://user-images.githubusercontent.com/20081078/95368345-00710e00-08a4-11eb-96d9-e2d544554a4b.png"> <img width="1680" alt="Screen Shot 2020-10-07 at 1 47 03 PM" src="https://user-images.githubusercontent.com/20081078/95368350-023ad180-08a4-11eb-89b3-f079480741f4.png"> <img width="1680" alt="Screen Shot 2020-10-07 at 1 47 12 PM" src="https://user-images.githubusercontent.com/20081078/95368364-0535c200-08a4-11eb-82fc-9435a046e4ca.png"> [ghstack-poisoned]

ghstack-source-id: 8247aa4 Pull Request resolved: #45270

<img width="1679" alt="Screen Shot 2020-10-07 at 1 45 59 PM" src="https://user-images.githubusercontent.com/20081078/95368324-fa7b2d00-08a3-11eb-9066-2e659a4085a2.png"> <img width="1673" alt="Screen Shot 2020-10-07 at 1 46 10 PM" src="https://user-images.githubusercontent.com/20081078/95368332-fbac5a00-08a3-11eb-9be5-77ce6deb8967.png"> <img width="1667" alt="Screen Shot 2020-10-07 at 1 46 30 PM" src="https://user-images.githubusercontent.com/20081078/95368337-fe0eb400-08a3-11eb-80a2-5ad23feeeb83.png"> <img width="1679" alt="Screen Shot 2020-10-07 at 1 46 48 PM" src="https://user-images.githubusercontent.com/20081078/95368345-00710e00-08a4-11eb-96d9-e2d544554a4b.png"> <img width="1680" alt="Screen Shot 2020-10-07 at 1 47 03 PM" src="https://user-images.githubusercontent.com/20081078/95368350-023ad180-08a4-11eb-89b3-f079480741f4.png"> <img width="1680" alt="Screen Shot 2020-10-07 at 1 47 12 PM" src="https://user-images.githubusercontent.com/20081078/95368364-0535c200-08a4-11eb-82fc-9435a046e4ca.png"> [ghstack-poisoned]

ghstack-source-id: cf6af37 Pull Request resolved: #45270

ezyang · 2020-10-07T23:14:52Z

docs/source/notes/autograd.rst

-convention for autograd for Complex Numbers.
+- PyTorch Autograd returns gradients that are directly usable for optimization and gradient descent (no conjugation needed),
+  so existing optimizers work out of the box. We don’t make any attempt to support holomorphic functions specially;
+  in particular, derivatives are only well defined for real-valued loss functions.


I realize this is my own damn fault because I wrote this text in the first place, but taken literally this text is wrong (I'm not sure what I even meant when I wrote this)

hmmm I rephrased it. does it look any better?

<img width="1679" alt="Screen Shot 2020-10-07 at 1 45 59 PM" src="https://user-images.githubusercontent.com/20081078/95368324-fa7b2d00-08a3-11eb-9066-2e659a4085a2.png"> <img width="1673" alt="Screen Shot 2020-10-07 at 1 46 10 PM" src="https://user-images.githubusercontent.com/20081078/95368332-fbac5a00-08a3-11eb-9be5-77ce6deb8967.png"> <img width="1667" alt="Screen Shot 2020-10-07 at 1 46 30 PM" src="https://user-images.githubusercontent.com/20081078/95368337-fe0eb400-08a3-11eb-80a2-5ad23feeeb83.png"> <img width="1679" alt="Screen Shot 2020-10-07 at 1 46 48 PM" src="https://user-images.githubusercontent.com/20081078/95368345-00710e00-08a4-11eb-96d9-e2d544554a4b.png"> <img width="1680" alt="Screen Shot 2020-10-07 at 1 47 03 PM" src="https://user-images.githubusercontent.com/20081078/95368350-023ad180-08a4-11eb-89b3-f079480741f4.png"> <img width="1680" alt="Screen Shot 2020-10-07 at 1 47 12 PM" src="https://user-images.githubusercontent.com/20081078/95368364-0535c200-08a4-11eb-82fc-9435a046e4ca.png"> [ghstack-poisoned]

ghstack-source-id: 15a3e1b Pull Request resolved: #45270

<img width="1679" alt="Screen Shot 2020-10-07 at 1 45 59 PM" src="https://user-images.githubusercontent.com/20081078/95368324-fa7b2d00-08a3-11eb-9066-2e659a4085a2.png"> <img width="1673" alt="Screen Shot 2020-10-07 at 1 46 10 PM" src="https://user-images.githubusercontent.com/20081078/95368332-fbac5a00-08a3-11eb-9be5-77ce6deb8967.png"> <img width="1667" alt="Screen Shot 2020-10-07 at 1 46 30 PM" src="https://user-images.githubusercontent.com/20081078/95368337-fe0eb400-08a3-11eb-80a2-5ad23feeeb83.png"> <img width="1679" alt="Screen Shot 2020-10-07 at 1 46 48 PM" src="https://user-images.githubusercontent.com/20081078/95368345-00710e00-08a4-11eb-96d9-e2d544554a4b.png"> <img width="1680" alt="Screen Shot 2020-10-07 at 1 47 03 PM" src="https://user-images.githubusercontent.com/20081078/95368350-023ad180-08a4-11eb-89b3-f079480741f4.png"> <img width="1680" alt="Screen Shot 2020-10-07 at 1 47 12 PM" src="https://user-images.githubusercontent.com/20081078/95368364-0535c200-08a4-11eb-82fc-9435a046e4ca.png"> [ghstack-poisoned]

ghstack-source-id: 262088d Pull Request resolved: #45270

<img width="1679" alt="Screen Shot 2020-10-07 at 1 45 59 PM" src="https://user-images.githubusercontent.com/20081078/95368324-fa7b2d00-08a3-11eb-9066-2e659a4085a2.png"> <img width="1673" alt="Screen Shot 2020-10-07 at 1 46 10 PM" src="https://user-images.githubusercontent.com/20081078/95368332-fbac5a00-08a3-11eb-9be5-77ce6deb8967.png"> <img width="1667" alt="Screen Shot 2020-10-07 at 1 46 30 PM" src="https://user-images.githubusercontent.com/20081078/95368337-fe0eb400-08a3-11eb-80a2-5ad23feeeb83.png"> <img width="1679" alt="Screen Shot 2020-10-07 at 1 46 48 PM" src="https://user-images.githubusercontent.com/20081078/95368345-00710e00-08a4-11eb-96d9-e2d544554a4b.png"> <img width="1680" alt="Screen Shot 2020-10-07 at 1 47 03 PM" src="https://user-images.githubusercontent.com/20081078/95368350-023ad180-08a4-11eb-89b3-f079480741f4.png"> <img width="1680" alt="Screen Shot 2020-10-07 at 1 47 12 PM" src="https://user-images.githubusercontent.com/20081078/95368364-0535c200-08a4-11eb-82fc-9435a046e4ca.png"> [ghstack-poisoned]

ghstack-source-id: a3a705f Pull Request resolved: #45270

facebook-github-bot · 2020-10-08T22:19:11Z

@anjali411 merged this pull request in 8925661.

Summary: Pull Request resolved: pytorch#45270 <img width="1679" alt="Screen Shot 2020-10-07 at 1 45 59 PM" src="https://user-images.githubusercontent.com/20081078/95368324-fa7b2d00-08a3-11eb-9066-2e659a4085a2.png"> <img width="1673" alt="Screen Shot 2020-10-07 at 1 46 10 PM" src="https://user-images.githubusercontent.com/20081078/95368332-fbac5a00-08a3-11eb-9be5-77ce6deb8967.png"> <img width="1667" alt="Screen Shot 2020-10-07 at 1 46 30 PM" src="https://user-images.githubusercontent.com/20081078/95368337-fe0eb400-08a3-11eb-80a2-5ad23feeeb83.png"> <img width="1679" alt="Screen Shot 2020-10-07 at 1 46 48 PM" src="https://user-images.githubusercontent.com/20081078/95368345-00710e00-08a4-11eb-96d9-e2d544554a4b.png"> <img width="1680" alt="Screen Shot 2020-10-07 at 1 47 03 PM" src="https://user-images.githubusercontent.com/20081078/95368350-023ad180-08a4-11eb-89b3-f079480741f4.png"> <img width="1680" alt="Screen Shot 2020-10-07 at 1 47 12 PM" src="https://user-images.githubusercontent.com/20081078/95368364-0535c200-08a4-11eb-82fc-9435a046e4ca.png"> Test Plan: Imported from OSS Reviewed By: navahgar Differential Revision: D24203257 Pulled By: anjali411 fbshipit-source-id: cd637dade5fb40cecf5d9f4bd03d508d36e26fcd

Summary: Pull Request resolved: #45270 <img width="1679" alt="Screen Shot 2020-10-07 at 1 45 59 PM" src="https://user-images.githubusercontent.com/20081078/95368324-fa7b2d00-08a3-11eb-9066-2e659a4085a2.png"> <img width="1673" alt="Screen Shot 2020-10-07 at 1 46 10 PM" src="https://user-images.githubusercontent.com/20081078/95368332-fbac5a00-08a3-11eb-9be5-77ce6deb8967.png"> <img width="1667" alt="Screen Shot 2020-10-07 at 1 46 30 PM" src="https://user-images.githubusercontent.com/20081078/95368337-fe0eb400-08a3-11eb-80a2-5ad23feeeb83.png"> <img width="1679" alt="Screen Shot 2020-10-07 at 1 46 48 PM" src="https://user-images.githubusercontent.com/20081078/95368345-00710e00-08a4-11eb-96d9-e2d544554a4b.png"> <img width="1680" alt="Screen Shot 2020-10-07 at 1 47 03 PM" src="https://user-images.githubusercontent.com/20081078/95368350-023ad180-08a4-11eb-89b3-f079480741f4.png"> <img width="1680" alt="Screen Shot 2020-10-07 at 1 47 12 PM" src="https://user-images.githubusercontent.com/20081078/95368364-0535c200-08a4-11eb-82fc-9435a046e4ca.png"> Test Plan: Imported from OSS Reviewed By: navahgar Differential Revision: D24203257 Pulled By: anjali411 fbshipit-source-id: cd637dade5fb40cecf5d9f4bd03d508d36e26fcd Co-authored-by: anjali411 <chourdiaanjali123@gmail.com>

Doc note update for complex numbers

dcc2a46

[ghstack-poisoned]

anjali411 changed the title ~~Doc note update for complex numbers~~ Doc note update for complex autograd Sep 24, 2020

anjali411 requested review from albanD and ezyang September 24, 2020 13:51

Update on "Doc note update for complex autograd"

e33dc36

[ghstack-poisoned]

albanD reviewed Sep 24, 2020

View reviewed changes

Update on "Doc note update for complex autograd"

1d0c8b3

[ghstack-poisoned]

Update on "Doc note update for complex autograd"

287f1df

[ghstack-poisoned]

Update on "Doc note update for complex autograd"

0289ace

[ghstack-poisoned]

anjali411 added a commit that referenced this pull request Sep 24, 2020

Doc note update for complex numbers

54a6f90

ghstack-source-id: b4d871c Pull Request resolved: #45270

Update on "Doc note update for complex autograd"

5e04a7f

[ghstack-poisoned]

Update on "Doc note update for complex autograd"

a5cb378

[ghstack-poisoned]

anjali411 added this to the 1.7.0 milestone Sep 24, 2020

anjali411 requested a review from zou3519 September 24, 2020 16:04

zou3519 reviewed Sep 24, 2020

View reviewed changes

Update on "Doc note update for complex autograd"

14ac806

[ghstack-poisoned]

anjali411 added a commit that referenced this pull request Sep 24, 2020

Doc note update for complex numbers

98be197

ghstack-source-id: f623cc0 Pull Request resolved: #45270

anjali411 mentioned this pull request Sep 24, 2020

Updated cholesky_backward for complex inputs #45267

Closed

boeddeker reviewed Sep 24, 2020

View reviewed changes

anjali411 added a commit that referenced this pull request Oct 7, 2020

Doc note update for complex numbers

8897fe1

ghstack-source-id: 8247aa4 Pull Request resolved: #45270

anjali411 requested review from albanD and ezyang and removed request for albanD and ezyang October 7, 2020 17:57

anjali411 added a commit that referenced this pull request Oct 7, 2020

Doc note update for complex numbers

fdda292

ghstack-source-id: cf6af37 Pull Request resolved: #45270

ezyang reviewed Oct 7, 2020

View reviewed changes

anjali411 added a commit that referenced this pull request Oct 8, 2020

Doc note update for complex numbers

f5d817a

ghstack-source-id: 15a3e1b Pull Request resolved: #45270

ezyang added a commit that referenced this pull request Oct 8, 2020

Doc note update for complex numbers

56dca8a

ghstack-source-id: 262088d Pull Request resolved: #45270

anjali411 added a commit that referenced this pull request Oct 8, 2020

Doc note update for complex numbers

bb9a5f4

ghstack-source-id: a3a705f Pull Request resolved: #45270

ezyang approved these changes Oct 8, 2020

View reviewed changes

This was referenced Oct 8, 2020

Doc note update for complex numbers #46058

Closed

[v1.7.0] Release Tracker #45592

Closed

facebook-github-bot closed this in 8925661 Oct 8, 2020

facebook-github-bot added the Merged label Oct 8, 2020

malfet mentioned this pull request Oct 9, 2020

Doc note update for complex autograd (CP-45270) #46072

Merged

facebook-github-bot deleted the gh/anjali411/58/head branch October 12, 2020 14:17

Doc note update for complex autograd #45270

Doc note update for complex autograd #45270

Uh oh!

Conversation

anjali411 commented Sep 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anjali411 commented Sep 24, 2020

Uh oh!

albanD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dr-ci bot commented Sep 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zou3519 Sep 24, 2020

Choose a reason for hiding this comment

Uh oh!

anjali411 Sep 24, 2020

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Sep 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

boeddeker Sep 24, 2020

Choose a reason for hiding this comment

Uh oh!

boeddeker Sep 24, 2020

Choose a reason for hiding this comment

Uh oh!

boeddeker Sep 24, 2020

Choose a reason for hiding this comment

Uh oh!

boeddeker commented Oct 2, 2020

Uh oh!

anjali411 commented Oct 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ezyang Oct 7, 2020

Choose a reason for hiding this comment

Uh oh!

anjali411 Oct 8, 2020

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Oct 8, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

anjali411 commented Sep 24, 2020 •

edited

Loading

dr-ci bot commented Sep 24, 2020 •

edited

Loading

codecov bot commented Sep 24, 2020 •

edited

Loading

anjali411 commented Oct 7, 2020 •

edited

Loading