Improve repr for torch.iinfo & torch.finfo #40488

Kiyosora · 2020-06-24T03:30:14Z

fix [feature request] Useful repr for torch.finfo/iinfo #39991
Include directly min/max/eps/tiny values in repr of torch.iinfo & torch.finfo for inspection
Use torch.float16 / torch.int16 instead of uncorrespond names Half / Short
The improved repr is shown just like:

>>> torch.iinfo(torch.int8)
iinfo(type=torch.int8, max=127, min=-128)
>>> torch.iinfo(torch.int16)
iinfo(type=torch.int16, max=32767, min=-32768)
>>> torch.iinfo(torch.int32)
iinfo(type=torch.int32, max=2.14748e+09, min=-2.14748e+09)
>>> torch.iinfo(torch.int64)
iinfo(type=torch.int64, max=9.22337e+18, min=-9.22337e+18)
>>> torch.finfo(torch.float16)
finfo(type=torch.float16, eps=0.000976563, max=65504, min=-65504, tiny=6.10352e-05)
>>> torch.finfo(torch.float32)
finfo(type=torch.float32, eps=1.19209e-07, max=3.40282e+38, min=-3.40282e+38, tiny=1.17549e-38)
>>> torch.finfo(torch.float64)
finfo(type=torch.float64, eps=2.22045e-16, max=1.79769e+308, min=-1.79769e+308, tiny=2.22507e-308)

dr-ci · 2020-06-24T05:09:11Z

💊 CI failures summary and remediations

As of commit 42356a9 (more details on the Dr. CI page):

1/2 failures possibly* introduced in this PR
- 1/1 non-CircleCI failure(s)
1/2 broken upstream at merge base 6ef9459 since Jul 07

🚧 1 ongoing upstream failure:

These were probably caused by upstream breakages that are not fixed yet:

pytorch_linux_backward_compatibility_check_test since Jul 07
- 🔁 rerun

Extra GitHub checks: 1 failed

Failed: GitHub Actions - clang-tidy

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 71 times.

gchanan · 2020-06-25T20:14:42Z

Can we make this closer to NumPy?

E.g. for iinfo:

In [15]: np.__version__
Out[15]: '1.18.1'

In [16]: np.iinfo(np.int8)
Out[16]: iinfo(min=-128, max=127, dtype=int8)

Similarly for finfo:

np.finfo(np.float32)
Out[17]: finfo(resolution=1e-06, min=-3.4028235e+38, max=3.4028235e+38, dtype=float32)

We don't have resolution; either adding it or just skipping it for now should be fine.

mruberry · 2020-06-26T05:17:13Z

Hey @Kiyosora, thank you for this PR! Following on what @gchanan said, we should try to be like NumPy here. Can you print each dtype using your PR and in NumPy? I think torch.bool, torch.uint8, torch.int8, torch.int16, torch.int32, torch.int64, torch.float16, torch.bfloat16, torch.float32, torch.float64, torch.complex64, torch.complex128. NumPy doesn't have np.bfloat16, so we won't compare torch.bfloat16 to it but that's OK.

The complex types are especially interesting, since NumPy will print their corresponding float types when they're given to finfo. @anjali411, what do you think about mimicking that behavior?

anjali411 · 2020-06-29T14:00:53Z

Hey @Kiyosora, thank you for this PR! Following on what @gchanan said, we should try to be like NumPy here. Can you print each dtype using your PR and in NumPy? I think torch.bool, torch.uint8, torch.int8, torch.int16, torch.int32, torch.int64, torch.float16, torch.bfloat16, torch.float32, torch.float64, torch.complex64, torch.complex128. NumPy doesn't have np.bfloat16, so we won't compare torch.bfloat16 to it but that's OK.

The complex types are especially interesting, since NumPy will print their corresponding float types when they're given to finfo. @anjali411, what do you think about mimicking that behavior?

sounds good!

Kiyosora · 2020-06-30T06:43:25Z

Hey @mruberry @gchanan @anjali411 , Thanks for your kindly advice! I have improved my code to make our iinfo and finfo much closer to the NumPy. Including tweak the output format, adds the resolution to torch.finfo, and made it available to receive torch.complex64 & torch.complex128 types input. Here is the comparison of iinfo/finfo between NumPy and the one fixed in my PR, please have a check wheather does it reach the goal.

>>> import torch
>>> import numpy as np

>>> np.iinfo(np.uint8)
iinfo(min=0, max=255, dtype=uint8)
>>> torch.iinfo(torch.uint8)
iinfo(min=0, max=255, dtype=uint8)

>>> np.iinfo(np.int8)
iinfo(min=-128, max=127, dtype=int8)
>>> torch.iinfo(torch.int8)
iinfo(min=-128, max=127, dtype=int8)

>>> np.iinfo(np.int16)
iinfo(min=-32768, max=32767, dtype=int16)
>>> torch.iinfo(torch.int16)
iinfo(min=-32768, max=32767, dtype=int16)

>>> np.iinfo(np.int32)
iinfo(min=-2147483648, max=2147483647, dtype=int32)
>>> torch.iinfo(torch.int32)
iinfo(min=-2.14748e+09, max=2.14748e+09, dtype=int32)

>>> np.iinfo(np.int64)
iinfo(min=-9223372036854775808, max=9223372036854775807, dtype=int64)
>>> torch.iinfo(torch.int64)
iinfo(min=-9.22337e+18, max=9.22337e+18, dtype=int64)

>>> np.finfo(np.float16)
finfo(resolution=0.001, min=-6.55040e+04, max=6.55040e+04, dtype=float16)
>>> torch.finfo(torch.float16)
finfo(resolution=0.001, min=-65504, max=65504, eps=0.000976563, tiny=6.10352e-05, dtype=float16)

>>> np.finfo(np.float32)
finfo(resolution=1e-06, min=-3.4028235e+38, max=3.4028235e+38, dtype=float32)
>>> torch.finfo(torch.float32)
finfo(resolution=1e-06, min=-3.40282e+38, max=3.40282e+38, eps=1.19209e-07, tiny=1.17549e-38, dtype=float32)

>>> np.finfo(np.float64)
finfo(resolution=1e-15, min=-1.7976931348623157e+308, max=1.7976931348623157e+308, dtype=float64)
>>> torch.finfo(torch.float64)
finfo(resolution=1e-15, min=-1.79769e+308, max=1.79769e+308, eps=2.22045e-16, tiny=2.22507e-308, dtype=float64)

>>> np.finfo(np.complex64)
finfo(resolution=1e-06, min=-3.4028235e+38, max=3.4028235e+38, dtype=float32)
>>> torch.finfo(torch.complex64)
finfo(resolution=1e-06, min=-3.40282e+38, max=3.40282e+38, eps=1.19209e-07, tiny=1.17549e-38, dtype=float32)

>>> np.finfo(np.complex128)
finfo(resolution=1e-15, min=-1.7976931348623157e+308, max=1.7976931348623157e+308, dtype=float64)
>>> torch.finfo(torch.complex128)
finfo(resolution=1e-15, min=-1.79769e+308, max=1.79769e+308, eps=2.22045e-16, tiny=2.22507e-308, dtype=float64)

>>> np.finfo(np.bfloat16)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'numpy' has no attribute 'bfloat16'
>>> torch.finfo(torch.bfloat16)
finfo(dtype=bfloat16)

>>> np.iinfo(np.bool)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\anaconda\envs\pytorch\lib\site-packages\numpy\core\getlimits.py", line 506, in __init__
    raise ValueError("Invalid integer data type %r." % (self.kind,))
ValueError: Invalid integer data type 'b'.
>>> np.finfo(np.bool)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\anaconda\envs\pytorch\lib\site-packages\numpy\core\getlimits.py", line 381, in __new__
    raise ValueError("data type %r not inexact" % (dtype))
ValueError: data type <class 'numpy.bool_'> not inexact

>>> torch.iinfo(torch.bool)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: torch.bool is not supported by torch.iinfo
>>> torch.finfo(torch.bool)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: torch.finfo() requires a floating point input type. Use torch.iinfo to handle 'torch.finfo'

Kiyosora · 2020-06-30T10:04:57Z

torch/csrc/TypeInfo.cpp

By the way, std::numeric_limits seem does not be well adapted to BFloat16 type. Calling it directly in the current version would cause a Python crash and the error was just like:

>>> torch.finfo(torch.bfloat16).max terminate called after throwing an instance of 'c10::Error' what(): "max" not implemented for 'BFloat16

So I wrote this logic. Make it only output the dtype info, but not max/min/tiny or others when the input is a BFloat16 type. Hope I haven't done anything redundant.

It's true you can't use std::numeric_limits, but can you use the values from

pytorch/c10/util/BFloat16-inl.h

Line 234 in 1a0b95e

static constexpr c10::BFloat16 min() {

?

Thanks for reminding me, I have made some improvements and the methods of finfo can successfully adapt to BFloat16 type now!

>>> torch.finfo(torch.bfloat16).max 3.3895313892515355e+38 >>> torch.finfo(torch.bfloat16).min -3.3895313892515355e+38 >>> torch.finfo(torch.bfloat16).bits 16 >>> torch.finfo(torch.bfloat16).dtype 'bfloat16' >>> torch.finfo(torch.bfloat16).eps 0.0078125 >>> torch.finfo(torch.bfloat16).tiny 1.1754943508222875e-38 >>> torch.finfo(torch.bfloat16).resolution 0.01 >>> torch.finfo(torch.bfloat16) finfo(resolution=0.01, min=-3.38953e+38, max=3.38953e+38, eps=0.0078125, tiny=1.17549e-38, dtype=bfloat16) >>>

Kiyosora · 2020-07-07T09:51:47Z

Hi @mruberry , Sorry to take up your time. Since this PR has been around for a while, would you please make a review on it? Any suggestion would be helpful. 😃

mruberry · 2020-07-07T18:10:30Z

Hi @mruberry , Sorry to take up your time. Since this PR has been around for a while, would you please make a review on it? Any suggestion would be helpful. 😃

Please don't apologize. It's my fault for not seeing the notification. Thank you for reminding me. I'll take a look now.

mruberry · 2020-07-07T18:17:21Z

test/test_type_info.py

Should torch.bfloat16, torch.complex64, and torch.complex128 be added to this list?

Also torch.bool?

mruberry · 2020-07-07T18:17:47Z

test/test_type_info.py

Should torch.bool be added to this list, too?

test/test_type_info.py

mruberry

This looks really good. Thank you for the updates, @Kiyosora! Sorry I didn't realize this was ready for review again. I'll watch it more carefully.

I made a few small comments about testing and adding in the values for bfloat16. Once that's done I think this is good to go!

Kiyosora · 2020-07-08T11:55:23Z

Hi @mruberry , Thanks for your suggestions, I have completed the changes. As a reference, I re-compared the iinfo/finfo methods in my PR and Numpy, and here are the results:

>>> import torch
>>> import numpy as np

>>> np.iinfo(np.uint8)
iinfo(min=0, max=255, dtype=uint8)
>>> torch.iinfo(torch.uint8)
iinfo(min=0, max=255, dtype=uint8)

>>> np.iinfo(np.int8)
iinfo(min=-128, max=127, dtype=int8)
>>> torch.iinfo(torch.int8)
iinfo(min=-128, max=127, dtype=int8)

>>> np.iinfo(np.int16)
iinfo(min=-32768, max=32767, dtype=int16)
>>> torch.iinfo(torch.int16)
iinfo(min=-32768, max=32767, dtype=int16)

>>> np.iinfo(np.int32)
iinfo(min=-2147483648, max=2147483647, dtype=int32)
>>> torch.iinfo(torch.int32)
iinfo(min=-2.14748e+09, max=2.14748e+09, dtype=int32)

>>> np.iinfo(np.int64)
iinfo(min=-9223372036854775808, max=9223372036854775807, dtype=int64)
>>> torch.iinfo(torch.int64)
iinfo(min=-9.22337e+18, max=9.22337e+18, dtype=int64)

>>> np.finfo(np.float16)
finfo(resolution=0.001, min=-6.55040e+04, max=6.55040e+04, dtype=float16)
>>> torch.finfo(torch.float16)
finfo(resolution=0.001, min=-65504, max=65504, eps=0.000976563, tiny=6.10352e-05, dtype=float16)

>>> np.finfo(np.float32)
finfo(resolution=1e-06, min=-3.4028235e+38, max=3.4028235e+38, dtype=float32)
>>> torch.finfo(torch.float32)
finfo(resolution=1e-06, min=-3.40282e+38, max=3.40282e+38, eps=1.19209e-07, tiny=1.17549e-38, dtype=float32)

>>> np.finfo(np.float64)
finfo(resolution=1e-15, min=-1.7976931348623157e+308, max=1.7976931348623157e+308, dtype=float64)
>>> torch.finfo(torch.float64)
finfo(resolution=1e-15, min=-1.79769e+308, max=1.79769e+308, eps=2.22045e-16, tiny=2.22507e-308, dtype=float64)

>>> np.finfo(np.complex64)
finfo(resolution=1e-06, min=-3.4028235e+38, max=3.4028235e+38, dtype=float32)
>>> torch.finfo(torch.complex64)
finfo(resolution=1e-06, min=-3.40282e+38, max=3.40282e+38, eps=1.19209e-07, tiny=1.17549e-38, dtype=float32)

>>> np.finfo(np.complex128)
finfo(resolution=1e-15, min=-1.7976931348623157e+308, max=1.7976931348623157e+308, dtype=float64)
>>> torch.finfo(torch.complex128)
finfo(resolution=1e-15, min=-1.79769e+308, max=1.79769e+308, eps=2.22045e-16, tiny=2.22507e-308, dtype=float64)

>>> np.finfo(np.bfloat16)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'numpy' has no attribute 'bfloat16'
>>> torch.finfo(torch.bfloat16)
finfo(resolution=0.01, min=-3.38953e+38, max=3.38953e+38, eps=0.0078125, tiny=1.17549e-38, dtype=bfloat16)

>>> np.iinfo(np.bool)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\anaconda\envs\pytorch\lib\site-packages\numpy\core\getlimits.py", line 506, in __init__
    raise ValueError("Invalid integer data type %r." % (self.kind,))
ValueError: Invalid integer data type 'b'.
>>> np.finfo(np.bool)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\anaconda\envs\pytorch\lib\site-packages\numpy\core\getlimits.py", line 381, in __new__
    raise ValueError("data type %r not inexact" % (dtype))
ValueError: data type <class 'numpy.bool_'> not inexact

>>> torch.iinfo(torch.bool)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: torch.bool is not supported by torch.iinfo
>>> torch.finfo(torch.bool)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: torch.finfo() requires a floating point input type. Use torch.iinfo to handle 'torch.finfo'

Please let me know if there's anything I can help, I'd be glad to assist. 😃

mruberry · 2020-07-08T22:50:35Z

Great! Here are the bfloat16 values in JAX for comparison (new review coming in a moment):

jax.dtypes.finfo(jax.dtypes.bfloat16).resolution
: 0.01
jax.dtypes.finfo(jax.dtypes.bfloat16).min
: -3.38953e+38
jax.dtypes.finfo(jax.dtypes.bfloat16).max
: 3.38953e+38
jax.dtypes.finfo(jax.dtypes.bfloat16).eps
: 0.0078125
jax.dtypes.finfo(jax.dtypes.bfloat16).tiny
: 1.17549e-38

And here's a copy of the values you're producing:

finfo(resolution=0.01, min=-3.38953e+38, max=3.38953e+38, eps=0.0078125, tiny=1.17549e-38, dtype=bfloat16)

They agree perfectly!

mruberry

Great work, @Kiyosora!

facebook-github-bot

@mruberry has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

@mruberry has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2020-07-11T00:14:08Z

@mruberry merged this pull request in 0651887.

mruberry · 2020-07-11T06:56:26Z

Thanks again for the great work, @Kiyosora! If you're interested in more open PyTorch issues let me know.

Kiyosora · 2020-07-13T01:23:52Z

Glad to be help @mruberry , please feel free to assign me another issue, I‘d like to have a try! 😃

mruberry · 2020-07-14T06:36:07Z

@Kiyosora awesome! This issue #38349 has a list of functions missing that we'd like to implement. If adding a new function to PyTorch sounds interesting, then you might want to pick one like nan_to_num. The PRs cited in that issue contain a lot of the pointers you'd need to implement it, and I'm also available if you have additional questions (you can just file a new issue like "implementing nan_to_num" to discuss them).

pytorchbot added the open source label Jun 24, 2020

Kiyosora changed the title ~~improve repr for torch.iinfo & torch.finfo~~ Improve repr for torch.iinfo & torch.finfo Jun 24, 2020

Kiyosora force-pushed the repr_improvement branch 2 times, most recently from 07a52ba to 2968455 Compare June 24, 2020 16:27

Kiyosora marked this pull request as ready for review June 24, 2020 23:09

gchanan requested a review from mruberry June 25, 2020 20:13

gchanan added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jun 25, 2020

Kiyosora force-pushed the repr_improvement branch from 6ea705a to cf6fbfd Compare June 29, 2020 12:24

Kiyosora force-pushed the repr_improvement branch 3 times, most recently from ae0f931 to c14d4fb Compare June 30, 2020 06:02

Kiyosora force-pushed the repr_improvement branch 2 times, most recently from 9479f1a to 3e67d4f Compare June 30, 2020 08:18

Kiyosora commented Jun 30, 2020

View reviewed changes

Kiyosora force-pushed the repr_improvement branch 3 times, most recently from e1d580a to 502b6a5 Compare June 30, 2020 15:00

mruberry reviewed Jul 7, 2020

View reviewed changes

test/test_type_info.py Outdated Show resolved Hide resolved

mruberry reviewed Jul 7, 2020

View reviewed changes

test/test_type_info.py Outdated Show resolved Hide resolved

mruberry requested changes Jul 7, 2020

View reviewed changes

improve repr for torch.iinfo & torch.finfo

00f1be4

xueht-fnst added 4 commits July 8, 2020 16:49

Fix repr tring to be like NumPy

58a4d99

fix for torch.complex type

6fcb581

fix bits for complex type

c13c8ca

fix set_default_dtype error in test

700dde6

Kiyosora force-pushed the repr_improvement branch from 7ec42bf to 0e21b5f Compare July 8, 2020 08:52

improve test case & adding values for bfloat16

42356a9

Kiyosora force-pushed the repr_improvement branch from 0e21b5f to 42356a9 Compare July 8, 2020 09:33

Kiyosora requested a review from mruberry July 8, 2020 11:42

mruberry approved these changes Jul 8, 2020

View reviewed changes

facebook-github-bot reviewed Jul 8, 2020

View reviewed changes

mruberry mentioned this pull request Jul 8, 2020

[WIP] #35954 torch.finfo support for complex #40047

Closed

facebook-github-bot reviewed Jul 10, 2020

View reviewed changes

facebook-github-bot closed this in 0651887 Jul 10, 2020

facebook-github-bot added the merged label Jul 11, 2020

Kiyosora deleted the repr_improvement branch July 13, 2020 01:24

mruberry mentioned this pull request Sep 1, 2020

torch.finfo doesn't work for complex #35954

Closed

mruberry added the Merged label Oct 28, 2020

Improve repr for torch.iinfo & torch.finfo #40488

Improve repr for torch.iinfo & torch.finfo #40488

Uh oh!

Conversation

Kiyosora commented Jun 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr-ci bot commented Jun 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

🚧 1 ongoing upstream failure:

Extra GitHub checks: 1 failed

Uh oh!

gchanan commented Jun 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mruberry commented Jun 26, 2020

Uh oh!

anjali411 commented Jun 29, 2020

Uh oh!

Kiyosora commented Jun 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kiyosora commented Jul 7, 2020

Uh oh!

mruberry commented Jul 7, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mruberry left a comment

Choose a reason for hiding this comment

Uh oh!

Kiyosora commented Jul 8, 2020

Uh oh!

mruberry commented Jul 8, 2020

Uh oh!

mruberry left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jul 11, 2020

Uh oh!

mruberry commented Jul 11, 2020

Uh oh!

Kiyosora commented Jul 13, 2020

Uh oh!

mruberry commented Jul 14, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Kiyosora commented Jun 24, 2020 •

edited

Loading

dr-ci bot commented Jun 24, 2020 •

edited

Loading

gchanan commented Jun 25, 2020 •

edited

Loading

Kiyosora commented Jun 30, 2020 •

edited

Loading