Skip to content

Configurable flushing denormal numbers on CPU #5294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
soumith merged 4 commits into from
Feb 20, 2018
Merged

Configurable flushing denormal numbers on CPU #5294

soumith merged 4 commits into from
Feb 20, 2018

Conversation

@tunz
Copy link
Contributor

@tunz tunz commented Feb 18, 2018 •
edited
Loading

Issue #4737

This PR provides a function to enable/disable flush denormal mode on x86 supporting SSE3. Intel is providing this two macros _MM_SET_FLUSH_ZERO_MODE and _MM_SET_DENORMALS_ZERO_MODE, so I used these.

it seems some old versions of gcc have an issue to prevent the use of intrinsics (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=57202), so always add -msse3 option.

I'm not sure if we have to enable this option by default or not, but I think this approach seems reasonable as a first step.

apaszke
apaszke reviewed Feb 18, 2018
View reviewed changes
Copy link
Contributor

@apaszke apaszke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Just curious why is that useful? Does it change the CPU performance a lot?

torch/_torch_docs.py Outdated
r"""
set_flush_denormal(on) -> bool
Returns ``True`` if it successfully configures flush denormal mode.

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

@tunz
Copy link
Contributor Author

tunz commented Feb 19, 2018

Yes. The benchmark of #4737 gets about 13x faster on my machine.

Without flushing denormal numbers:

Starting...
p=100, q=100, r=10
  iter   V_maxdiff       W_maxdiff             obj            time
--------------------------------------------------------------------------------
   100  1.9202e-03      8.9862e-03      6.2078e+02         0.00460
   200  6.6091e-04      3.3026e-03      6.1045e+02         0.00391
   300  6.3047e-04      3.7313e-03      6.0724e+02         0.00365
...
 17000  2.3246e-06      1.0252e-05      5.9653e+02         0.05804
 17100  2.2724e-06      1.0252e-05      5.9653e+02         0.05816
 17200  2.1532e-06      9.4175e-06      5.9653e+02         0.05767
--------------------------------------------------------------------------------
Completed. total time: 8.237918376922607

With flushing denormal numbers:

Starting...
p=100, q=100, r=10
  iter   V_maxdiff       W_maxdiff             obj            time
--------------------------------------------------------------------------------
   100  1.9202e-03      8.9862e-03      6.2078e+02         0.00458
   200  6.6091e-04      3.3026e-03      6.1045e+02         0.00393
   300  6.3047e-04      3.7313e-03      6.0724e+02         0.00367
...
 16200  4.4415e-06      1.1805e-05      5.9654e+02         0.00366
 16300  3.0734e-06      1.0103e-05      5.9654e+02         0.00365
 16400  2.2054e-06      9.6560e-06      5.9654e+02         0.00365
--------------------------------------------------------------------------------
Completed. total time: 0.5998060703277588

@apaszke
Copy link
Contributor

apaszke commented Feb 19, 2018

@pytorchbot test this please

@soumith soumith merged commit fae6c67 into pytorch:master Feb 20, 2018
@soumith
Copy link
Contributor

soumith commented Feb 20, 2018

thanks @tunz !

@soumith soumith mentioned this pull request Feb 22, 2018
Closed
@kose-y kose-y mentioned this pull request Apr 3, 2018
Closed
@sidazhang sidazhang mentioned this pull request Feb 13, 2019
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@apaszke apaszke apaszke approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

open source

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tunz @apaszke @soumith @ezyang