Skip to content

Conversation

@jonbob
Copy link
Contributor

@jonbob jonbob commented Sep 24, 2019

This PR fixes a threading issue that was introduced in E3SM PR #3057 that brought in the initial implementation of 3D varying GM bolus. After that commit, some E3SM threading tests were observed to fail with different results.

Tested with:

  • SMS_P12x2.T62_oQU240.CMPASO-NYF.compy_intel - BFB results 10 times
  • SMS_P12x2.ne4_oQU240.A_WCYCL1850.compy_intel.allactive-mach_mods - BFB results 10 times

This last test is one that was failing on sandiatoss3.

Copy link
Contributor

@vanroekel vanroekel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved by visual inspection

@jonbob
Copy link
Contributor Author

jonbob commented Sep 26, 2019

Just a quick note -- I had intel inspector working and did some checking on memory as well. The tools points at invalid memory access in the gm code as well, particularly this loop:

      !$omp do schedule(runtime) private(k)
      do iEdge = 1, nEdges + 1
         do k = 1, nVertLevels
            gradDensityEdge(k, iEdge) = huge(0D0)
            gradDensityTopOfEdge(k, iEdge) = huge(0D0)
            dDensityDzTopOfEdge(k, iEdge) = huge(0D0)
            gradZMidEdge(k, iEdge) = huge(0D0)
            gradZMidTopOfEdge(k, iEdge) = huge(0D0)
            relativeSlopeTopOfEdge(k, iEdge) = 0.0_RKIND
            relativeSlopeTapering(k, iEdge) = 0.0_RKIND
            normalGMBolusVelocity(k, iEdge) = 0.0_RKIND
         end do
      end do
      !$omp end do

I believe this is because some of the arrays are dimensioned nEdges and the loop goes over nEdges+1. Also the "private(k)" is missing in the code -- I'll commit it to this PR.

@mark-petersen
Copy link
Contributor

@amametjanov or @philipwjones could you please look at the openMP pragmas and confirm that they are done correctly?

@philipwjones
Copy link
Contributor

@mark-petersen OpenMP pragmas look ok to me - I know @amametjanov has been removing the private clauses since there is a default private clause on the top-level parallel region, so they're redundant but do no harm.

@amametjanov
Copy link
Contributor

amametjanov commented Oct 3, 2019

Yes, this looks good. I think the last two commits, which make loop extents match array sizes, fix the real culprits. Formatting is so much better too!

@mark-petersen mark-petersen force-pushed the jonbob/ocn/fix-threading-issue branch from 70baf5d to e5075b9 Compare October 4, 2019 16:02
@mark-petersen mark-petersen changed the base branch from e3sm/develop to ocean/develop October 4, 2019 16:03
@mark-petersen
Copy link
Contributor

I moved the commits over to ocean/develop, otherwise we will bring extra things over with the merge.

mark-petersen added a commit that referenced this pull request Oct 5, 2019
This PR fixes a threading issue that was introduced in [E3SM PR#3057]
(E3SM-Project/E3SM#3057) that brought in
the initial implementation of 3D varying GM bolus. After that commit,
some E3SM threading tests were observed to fail with different results.

Tested with:
* SMS_P12x2.T62_oQU240.CMPASO-NYF.compy_intel - BFB results 10 times
* SMS_P12x2.ne4_oQU240.A_WCYCL1850.compy_intel.allactive-mach_mods - BFB
* results 10 times

This last test is one that was failing on sandiatoss3.
@mark-petersen mark-petersen merged commit e5075b9 into MPAS-Dev:ocean/develop Oct 5, 2019
mark-petersen added a commit to mark-petersen/MPAS-Model that referenced this pull request Mar 24, 2020
…_update_maint1.2

Fix threading issue in MPAS-O GM routine MPAS-Dev#376
This PR fixes a threading issue that was introduced in E3SM PR #3057
that brought in the initial implementation of 3D varying GM bolus. After
that commit, some E3SM threading tests were observed to fail with
different results.

Tested with:

SMS_P12x2.T62_oQU240.CMPASO-NYF.compy_intel - BFB results 10 times
SMS_P12x2.ne4_oQU240.A_WCYCL1850.compy_intel.allactive-mach_mods
- BFB results 10 times

This last test is one that was failing on sandiatoss3.
mark-petersen added a commit that referenced this pull request Mar 24, 2020
Update maint1.2: 3D varying GM

This update adds the following PRs from ocean/develop:
- Adds options for 3d varying GM bolus and 2d varying phase speed #288
- Add GM bolus eddy stats #339
- Fix threading issue in MPAS-O GM routine #376
- compute landIceMask using geometric_features #447
- bug fixes for the nonlocal source term in KPP #305
jonbob added a commit to E3SM-Project/E3SM that referenced this pull request Mar 27, 2020
#3513)

Update maint1.2: add 3D varying GM

This update brings in a new mpas-source submodule with the following PRs from
MPAS ocean/develop:
* Adds options for 3d varying GM bolus and 2d varying phase speed
  (MPAS-Dev/MPAS-Model#288)
* Add GM bolus eddy stats (MPAS-Dev/MPAS-Model#339)
* Fix threading issue in MPAS-O GM routine (MPAS-Dev/MPAS-Model#376)
* Compute landIceMask using geometric_features (MPAS-Dev/MPAS-Model#447)
* Bug fixes for the nonlocal source term in KPP (MPAS-Dev/MPAS-Model#305)
and from E3SM master:
* Change all ocean output files to single precision (E3SM #3360)

[non-BFB]
ashwathsv pushed a commit to ashwathsv/MPAS-Model that referenced this pull request Jul 21, 2020
…elop

This PR fixes a threading issue that was introduced in [E3SM PR#3057]
(E3SM-Project/E3SM#3057) that brought in
the initial implementation of 3D varying GM bolus. After that commit,
some E3SM threading tests were observed to fail with different results.

Tested with:
* SMS_P12x2.T62_oQU240.CMPASO-NYF.compy_intel - BFB results 10 times
* SMS_P12x2.ne4_oQU240.A_WCYCL1850.compy_intel.allactive-mach_mods - BFB
* results 10 times

This last test is one that was failing on sandiatoss3.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants