fix Rayleigh drag logic on implicit vertical mix #676

mark-petersen · 2020-08-29T17:37:10Z

Fix if statement for Rayleigh drag introduced in #485. When Rayleigh drag was on, it turned off implicit vertical mixing. We only use Rayleigh drag for spin-up, so this does not change normal forward model runs or E3SM simulations.

mark-petersen · 2020-08-29T17:41:28Z

src/core_ocean/shared/mpas_ocn_vmix.F

-            dt, kineticEnergyCell, &
-            vertViscTopOfEdge, layerThickness, layerThicknessEdge, normalVelocity, &
-            ssh, bottomDepth, err)
-        else if (config_Rayleigh_damping_depth_variable) then


The actual bug was that this line should have been:

- else if (config_Rayleigh_damping_depth_variable) then + else if (config_Rayleigh_friction.or. & + config_Rayleigh_bottom_friction.or. & + config_Rayleigh_damping_depth_variable) then

but the logic is funny (nested if), so I changed the branching to be more direct.

mark-petersen · 2020-08-29T17:43:21Z

@vanroekel and @xylar, with this fix the EC60to30 prep_spinup1 works as it did before. Without this fix, implicit vertical mixing is off when Rayleigh damping is turned on, causing it to crash in the second day.

vanroekel

approved by visual inspection and test run through pre_spinup1 with this PR.

vanroekel · 2020-09-03T18:03:01Z

@mark-petersen further testing of this in debug with gcc gives an error in the first timestep. Stack trace is below

#1  0x8ec46d in __ocn_thick_ale_MOD_ocn_ale_thickness
	at /usr/projects/climate/lvanroekel/meshprs/src/core_ocean/shared/mpas_ocn_thick_ale.F:162
#2  0x91834f in __ocn_diagnostics_MOD_ocn_vert_transport_velocity_top
	at /usr/projects/climate/lvanroekel/meshprs/src/core_ocean/shared/mpas_ocn_diagnostics.F:1060
#3  0xa5c300 in __ocn_time_integration_split_MOD_ocn_time_integrator_split
	at /usr/projects/climate/lvanroekel/meshprs/src/core_ocean/mode_forward/mpas_ocn_time_integration_split.F:1519
#4  0xa4b88e in __ocn_time_integration_MOD_ocn_timestep
	at /usr/projects/climate/lvanroekel/meshprs/src/core_ocean/mode_forward/mpas_ocn_time_integration.F:111
#5  0x8dac07 in __ocn_forward_mode_MOD_ocn_forward_mode_run
	at /usr/projects/climate/lvanroekel/meshprs/src/core_ocean/mode_forward/mpas_ocn_forward_mode.F:588
#6  0x9fdcb6 in __ocn_core_MOD_ocn_core_run
	at /usr/projects/climate/lvanroekel/meshprs/src/core_ocean/driver/mpas_ocn_core.F:107
#7  0x4094a9 in __mpas_subdriver_MOD_mpas_run
	at /usr/projects/climate/lvanroekel/meshprs/src/driver/mpas_subdriver.F:347
#8  0x4082b3 in mpas
	at /usr/projects/climate/lvanroekel/meshprs/src/driver/mpas.F:16
#9  0x40830f in main
	at /usr/projects/climate/lvanroekel/meshprs/src/driver/mpas.F:10

mark-petersen · 2020-09-03T22:07:13Z

@vanroekel thanks for testing. What test case was that, and which part? EC60to30 prep_spinup1?

mark-petersen · 2020-09-03T22:56:29Z

@vanroekel I was not able to reproduce your error, either with gnu debug or intel debug, with EC60to30 prep_spinup1 or with QU240 run with Rayleigh drag turned on. Can you send me your run directory with that error?

vanroekel · 2020-09-04T04:17:13Z

@mark-petersen Here is my directory

/lustre/scratch4/turquoise/lvanroekel/EC6030v2_1900/ocean/global_ocean/EC60to30/spin_up/prep_spin_up1

I wonder if this is something in my environment or bad node. I've also been getting random OOM errors on grizzly in this test case as well.

vanroekel · 2020-09-04T04:17:55Z

here are my modules loaded

  1) gcc/6.4.0
  2) openmpi/2.1.2
  3) cmake/3.12.1
  4) mkl/2019.0.4
  5) openmpi/2.1.2-bheb4xe/gcc/6.4.0/netcdf/4.4.1.1-zei2j6r
  6) openmpi/2.1.2-bheb4xe/gcc/6.4.0/netcdf-fortran/4.4.4-v6vwmxs
  7) openmpi/2.1.2-bheb4xe/gcc/6.4.0/parallel-netcdf/1.8.0-2qwcdbn
  8) openmpi/2.1.2-bheb4xe/gcc/6.4.0/pio/1.10.0-ljj73au
  9) openmpi/2.1.2-bheb4xe/gcc/6.4.0/boost/1.66.0-u6kxo2f
 10) openmpi/2.1.2-bheb4xe/gcc/6.4.0/hdf5/1.10.1-rpkszo5
 11) openmpi/2.1.2-bheb4xe/gcc/6.4.0/parmetis/4.0.3-p3d7xsi
 12) gcc/6.4.0/metis/5.1.0-il7wbho

xylar · 2020-09-04T11:47:21Z

I'm running with this and #668 merged into ocean/develop on Grizzly and it's working well for me. I'm running with:

$ module list

Currently Loaded Modules:
  1) git/2.21.0         4) intel-mpi/2019.4       7) netcdf-h5parallel/4.7.3
  2) friendly-testing   5) hdf5-parallel/1.8.16   8) mkl/2019.0.4
  3) intel/19.0.4       6) pnetcdf/1.11.2         9) scorpio/pio2/1.10.1

@vanroekel, I think you would benefit with updating to SCORPIO/PIO2

Here is the full script I use to get myself set up on Grizzly:

#!/usr/bin/env bash

echo "Loading modules for grizzly with intel and scorpio"
module purge
module use /usr/projects/climate/SHARED_CLIMATE/modulefiles/all/
module load git
if [ -f "load_compass_env.sh" ]; then
    source load_compass_env.sh
else
    source /usr/projects/climate/SHARED_CLIMATE/anaconda_envs/base/etc/profile.d/conda.sh
    conda activate compass_0.1.10
fi

module load friendly-testing
module load intel/19.0.4 intel-mpi/2019.4 hdf5-parallel/1.8.16 pnetcdf/1.11.2 netcdf-h5parallel/4.7.3 mkl/2019.0.4 scorpio/pio2/1.10.1
export I_MPI_CC=icc
export I_MPI_CXX=icpc
export I_MPI_F77=ifort
export I_MPI_F90=ifort
export CORE=ocean
export USE_PIO2=true
export AUTOCLEAN=true

export  HDF5_USE_FILE_LOCKING=FALSE

Note: the comapss version has to be updated pretty often, but I like to do that manually rather than using load_lastest_compass.sh because I don't always want the latest. You can do whatever makes sense for you.

xylar · 2020-09-04T11:49:30Z

If you want to use Gnu, I don't know what the corresponding setup looks like. I haven't been using GNU on Grizzly in awhile.

xylar · 2020-09-04T11:50:12Z

@mark-petersen, could you share the Gnu configuration you use?

mark-petersen · 2020-09-04T12:08:29Z

For intel, I use what Phil Wolfram set up just before he left:

#!/usr/bin/env bash

echo "Loading modules for grizzly intel19"
module purge
source /usr/projects/climate/SHARED_CLIMATE/anaconda_envs/load_latest_compass.sh
module use /usr/projects/climate/SHARED_CLIMATE/modulefiles/all/
module load friendly-testing
module load intel/19.0.4 intel-mpi/2019.4 hdf5-parallel/1.8.16 pnetcdf/1.11.2 netcdf-h5parallel/4.7.3 mkl/2019.0.4 scorpio/pio2/1.10.1
export I_MPI_CC=icc
export I_MPI_CXX=icpc
export I_MPI_F77=ifort
export I_MPI_F90=ifort

Currently Loaded Modules:
  1) friendly-testing   3) intel-mpi/2019.4       5) pnetcdf/1.11.2            7) mkl/2019.0.4
  2) intel/19.0.4       4) hdf5-parallel/1.8.16   6) netcdf-h5parallel/4.7.3   8) scorpio/pio2/1.10.1

Looking at my gnu libraries, they are embarrassingly old:

module load gcc/5.3.0 openmpi/1.10.5 netcdf/4.4.1 parallel-netcdf/1.5.0 pio/1.7.2;

Currently Loaded Modules:
  1) git/2.21.0   2) gcc/5.3.0   3) openmpi/1.10.5   4) netcdf/4.4.1   5) parallel-netcdf/1.5.0   6) pio/1.7.2

Since the latest on grizzly and badger is gcc/9.3.0!

On badger, we are just slightly better:

#!/usr/bin/env bash

echo "Loading modules for badger gnu"
## badger modules ###
module use /usr/projects/climate/SHARED_CLIMATE/modulefiles/spack-lmod/linux-rhel7-x86_64
module unload python; source /usr/projects/climate/SHARED_CLIMATE/anaconda_envs/load_latest_compass.sh;
# IC mods
module load gcc/6.4.0
module load openmpi/2.1.2
module load cmake/3.12.1
module load mkl

# spack mods
 module load openmpi/2.1.2-bheb4xe/gcc/6.4.0/netcdf/4.4.1.1-zei2j6r
 module load openmpi/2.1.2-bheb4xe/gcc/6.4.0/netcdf-fortran/4.4.4-v6vwmxs
 module load openmpi/2.1.2-bheb4xe/gcc/6.4.0/parallel-netcdf/1.8.0-2qwcdbn
 module load openmpi/2.1.2-bheb4xe/gcc/6.4.0/pio/1.10.0-ljj73au

# spack mods aren't setting these correctly yet...
#export NETCDF=/usr/projects/climate/SHARED_CLIMATE/software/badger/spack-install/linux-rhel7-x86_64/gcc-6.4.0/netcdf-4.4.1.1-zei2j6rxcor4chgmgbcwlx37lb3khz7t
  export NETCDF=/usr/projects/climate/SHARED_CLIMATE/software/badger/spack-install/linux-rhel7-x86_64/gcc-6.4.0/netcdf-fortran-4.4.4-v6vwmxsv33t7pmulojlijwdbikrvmwkc
  export PNETCDF=/usr/projects/climate/SHARED_CLIMATE/software/badger/spack-install/linux-rhel7-x86_64/gcc-6.4.0/parallel-netcdf-1.8.0-2qwcdbnjcq5pnkoqpx2s7um3s7ffo3xd
  export PIO=/usr/projects/climate/SHARED_CLIMATE/software/badger/spack-install/linux-rhel7-x86_64/gcc-6.4.0/pio-1.10.0-ljj73au6ctgkwmh3gbd4mleljsumijys/

Currently Loaded Modules:
  1) git/2.21.0      6) openmpi/2.1.2-bheb4xe/gcc/6.4.0/netcdf/4.4.1.1-zei2j6r
  2) gcc/6.4.0       7) openmpi/2.1.2-bheb4xe/gcc/6.4.0/netcdf-fortran/4.4.4-v6vwmxs
  3) openmpi/2.1.2   8) openmpi/2.1.2-bheb4xe/gcc/6.4.0/parallel-netcdf/1.8.0-2qwcdbn
  4) cmake/3.12.1    9) openmpi/2.1.2-bheb4xe/gcc/6.4.0/pio/1.10.0-ljj73au
  5) mkl/2019.0.4

That last set looks like what Luke is using on Grizzly.

I try to keep up the library configuration notes here: https://acme-climate.atlassian.net/wiki/spaces/ECG/pages/1054113803/MPAS+Testing+and+Machine-specific+Instructions#grizzly,-gnu

xylar · 2020-09-04T12:42:53Z

It sounds like we need to see if things were broken on Gnu well before this current issue. That's unfortunate!

mark-petersen · 2020-09-04T14:01:04Z

Ha! I was able to reproduce the error Luke found, but only with gcc/6.4.0 with debug, not intel or gcc/5.3.0. I will make an issue and look into that, but it is most likely unrelated to this PR.

xylar

Spin-up of the EC30to60km in #668 and #669 worked for me with this PR included.

mark-petersen · 2020-09-04T15:54:43Z

Tested with nightly regression suite before/after this PR with gnu 6.4.0 on badger, both debug and optimized. All tests run and are bfb with ocean/develop.

Therefore, I'm concluding that the error we see in EC60to30 spinup is unrelated to this particular PR. I just made an issue here: #685

…/develop fix Rayleigh drag logic on implicit vertical mix #676

Update MPAS-Source: fix Rayleigh damping and line continuation This PR brings in a new mpas-source submodule with changes only to the ocean core. The changes are for two bug fixes: * fix Rayleigh drag logic on implicit vertical mix, which only impacts standalone MPAS ocean spin-up simulations (MPAS-Dev/MPAS-Model/pull/676); and * fix bad continuation character in ocn init tidal boundary (MPAS-Dev/MPAS-Model/pull/675) [BFB]

…nto ocean/develop fix Rayleigh drag logic on implicit vertical mix MPAS-Dev#676 Fix if statement for Rayleigh drag introduced in MPAS-Dev#485. When Rayleigh drag was on, it turned off implicit vertical mixing. We only use Rayleigh drag for spin-up, so this does not change normal forward model runs or E3SM simulations.

fix Rayleigh drag logic on implicit vertical mix

f216f46

mark-petersen added Ocean bug fix labels Aug 29, 2020

mark-petersen requested a review from vanroekel August 29, 2020 17:37

mark-petersen commented Aug 29, 2020

View reviewed changes

vanroekel approved these changes Aug 31, 2020

View reviewed changes

xylar mentioned this pull request Sep 1, 2020

Update Earth radius to the CIME value #680

Merged

xylar approved these changes Sep 4, 2020

View reviewed changes

mark-petersen self-assigned this Sep 4, 2020

mark-petersen merged commit f241c39 into MPAS-Dev:ocean/develop Sep 4, 2020

mark-petersen deleted the ocean/fix_Rayleigh_drag_logic branch September 4, 2020 15:59

mark-petersen mentioned this pull request Sep 8, 2020

Update MPAS-Source: fix Rayleigh damping and line continuation E3SM-Project/E3SM#3817

Merged

mark-petersen added a commit that referenced this pull request Sep 8, 2020

Merge PR #676 'mark-petersen/ocean/fix_Rayleigh_drag_logic' into e3sm…

3b90ce0

…/develop fix Rayleigh drag logic on implicit vertical mix #676

mark-petersen mentioned this pull request Sep 16, 2020

runtime error in ALE routine with gnu 6.4.0 debug during EC60to30 spinup #685

Closed

matthewhoffman added bug fix Ocean labels Mar 17, 2021

fix Rayleigh drag logic on implicit vertical mix #676

fix Rayleigh drag logic on implicit vertical mix #676

Uh oh!

Conversation

mark-petersen commented Aug 29, 2020

Uh oh!

mark-petersen Aug 29, 2020

Choose a reason for hiding this comment

Uh oh!

mark-petersen commented Aug 29, 2020

Uh oh!

vanroekel left a comment

Choose a reason for hiding this comment

Uh oh!

vanroekel commented Sep 3, 2020

Uh oh!

mark-petersen commented Sep 3, 2020

Uh oh!

mark-petersen commented Sep 3, 2020

Uh oh!

vanroekel commented Sep 4, 2020

Uh oh!

vanroekel commented Sep 4, 2020

Uh oh!

xylar commented Sep 4, 2020

Uh oh!

xylar commented Sep 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xylar commented Sep 4, 2020

Uh oh!

mark-petersen commented Sep 4, 2020

Uh oh!

xylar commented Sep 4, 2020

Uh oh!

mark-petersen commented Sep 4, 2020

Uh oh!

xylar left a comment

Choose a reason for hiding this comment

Uh oh!

mark-petersen commented Sep 4, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xylar commented Sep 4, 2020 •

edited

Loading