🔗 Helpful links

• 🧪  See artifacts and rendered test results at hud.pytorch.org/pr/62558
• 📄  Preview docs built from this PR
• 📄  Preview C++ docs built from this PR
• 🔧  Opt-in to CIFlow to control what jobs run on your PRs

💊 CI failures summary and remediations

As of commit 9f09c88 (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

Hi! Can you please clarify why we can't use https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/cpu/vec/vec256/vec256_bfloat16.h#L154 here?

@VitalyFedyunin Hi! This is because Vectorized<BFloat16> is casted to Vectorized<float> at the beginning, which will reduce the type conversion overhead of intermediate operations.

Overall looks good, I would like to see benchmark numbers for different input sizes.

Hi~
Single core performance is tested on Xeon(R) Platinum 8180 @2.5 Ghz.

input: 1x3x1x6: 
    Using Vecterized<BFloat16> directly:      forward: 0.185ms,  backward: 0.179ms  
    Casted to Vecterized<float> at beginning: forward: 0.183ms,  backward: 0.175ms 

input: 1x3x1x128: 
    Using Vecterized<BFloat16> directly:       forward: 0.198ms,  backward: 0.194ms   
    Casted to Vecterized<float> at beginning: forward: 0.201ms,  backward: 0.197ms    

input: 1x3x128x128: 
    Using Vecterized<BFloat16> directly:       forward: 2.118ms,  backward: 2.453ms   
    Casted to Vecterized<float> at beginning: forward: 2.004ms,  backward: 2.992ms    

Rounding error (the greatest difference compared with float) is tested on Intel(R) Core(TM) i7-10700K CPU.

input: 1x3x1x6: 
    Using Vecterized<BFloat16> directly:      forward: 0.0029454827308654785,  backward: 0.000244140625  
    Casted to Vecterized<float> at beginning: forward: 0.0009607672691345215,  backward: 0.00010850653052330017 

input: 1x3x1x128: 
    Using Vecterized<BFloat16> directly:      forward: 0.0005249381065368652,  backward: 1.52587890625e-05   
    Casted to Vecterized<float> at beginning: forward: 0.0005249381065368652,  backward: 7.62939453125e-06    

input: 1x3x128x128: 
    Using Vecterized<BFloat16> directly:      forward: 0.002125561237335205,  backward: 1.1920928955078125e-07 
    Casted to Vecterized<float> at beginning: forward: 0.002125561237335205,  backward: 5.960464477539063e-08  

In this case, there is no advantage to cast Vecterized<BFloat16> to Vecterized<float> at beginning, but it will reduce rounding errors.

Codecov Report

Merging #62558 (4a451c6) into master (feefc94) will increase coverage by 0.38%.
The diff coverage is n/a.

❗ Current head 4a451c6 differs from pull request most recent head c280eb9. Consider uploading reports for the commit c280eb9 to get more accurate results

@@            Coverage Diff             @@
##           master   #62558      +/-   ##
==========================================
+ Coverage   66.37%   66.76%   +0.38%     
==========================================
  Files         739      695      -44     
  Lines       94299    90736    -3563     
==========================================
- Hits        62595    60580    -2015     
+ Misses      31704    30156    -1548     

When the input becomes larger, the second method get better optimization.
Another single core performance test on Xeon(R) Platinum 8180 @2.5 Ghz (AVX2).

input: 1x3x256x256: 
    Using Vecterized<BFloat16> directly:      forward: 0.404 ms,  backward:  0.387 ms  
    Casted to Vecterized<float> at beginning: forward: 0.134 ms,  backward:  0.196 ms 

input: 4x3x256x1024: 
    Using Vecterized<BFloat16> directly:      forward: 5.886 ms,  backward:   5.913 ms  
    Casted to Vecterized<float> at beginning: forward: 1.746 ms,  backward:   2.800 ms 

input: 10x10x1024x1024: 
    Using Vecterized<BFloat16> directly:      forward: 209.155 ms,  backward:  220.582 ms  
    Casted to Vecterized<float> at beginning: forward: 77.022 ms,  backward:  118.064 ms 

Rebased @VitalyFedyunin

Hi @VitalyFedyunin, could you please review it ? Thank you.

@frank-wei has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@frank-wei has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Hey @CaoE.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.