Testing

I set up an environment on my linux laptop and on LANL IC as follows:

conda create -n compass_py3.7 -c conda-forge -c xylar python=3.7 geometric_features mpas_tools \
    jigsaw metis pyflann scikit-image basemap
conda activate compass_py3.7

Note: I have uploaded test versions of geometric_features and mpas_tools to the xylar anaconda channel. These will be deleted as soon as the packages are approved and available on conda-forge instead.

To activate this environment on LANL IC, run:

module unload python
source /usr/projects/climate/SHARED_CLIMATE/anaconda_envs/base/etc/profile.d/conda.sh
conda activate compass_py3.7

COMPASS ocean test cases that have been tested and found to run successfully in this environment:

• 40: -o ocean -c baroclinic_channel -r 10km -t decomp_test
• 41: -o ocean -c baroclinic_channel -r 10km -t default
• 42: -o ocean -c baroclinic_channel -r 10km -t restart_test
• 43: -o ocean -c baroclinic_channel -r 10km -t threads_test
• 44: -o ocean -c global_ocean -r EC60to30 -t init
• 45: -o ocean -c global_ocean -r EC60to30 -t spin_up
• 46: -o ocean -c global_ocean -r NAEC60to30cr8 -t init
• 47: -o ocean -c global_ocean -r NAEC60to30cr8 -t spin_up
• 48: -o ocean -c global_ocean -r QU240 -t init
• 49: -o ocean -c global_ocean -r QU240 -t test
• 50: -o ocean -c global_ocean -r QU60 -t init
• 51: -o ocean -c global_ocean -r QU60 -t spin_up
• 52: -o ocean -c global_ocean -r SOQU60to15 -t init
• 53: -o ocean -c global_ocean -r SOQU60to15 -t spin_up
• 54: -o ocean -c global_ocean -r USDEQU120cr10rr1 -t init
• 55: -o ocean -c global_ocean -r USDEQU300cr20rr3 -t init
• 56: -o ocean -c internal_waves -r 5km -t default
• 57: -o ocean -c internal_waves -r 5km -t ten-day
• 58: -o ocean -c isomip -r 10km -t expt1.01
• 59: -o ocean -c isomip -r 10km -t expt2.01
• 60: -o ocean -c isomip_plus -r 2km -t Ocean0
• 61: -o ocean -c isomip_plus -r 2km -t Ocean1
• 62: -o ocean -c isomip_plus -r 2km -t Ocean2
• 63: -o ocean -c isomip_plus -r 5km -t Ocean0
• 64: -o ocean -c isomip_plus -r 5km -t Ocean1
• 65: -o ocean -c isomip_plus -r 5km -t Ocean2
• 66: -o ocean -c lock_exchange -r 0.5km -t default
• 67: -o ocean -c lock_exchange -r 16km -t default
• 68: -o ocean -c overflow -r 10km -t default
• 69: -o ocean -c periodic_planar -r 20km -t default_light
• 70: -o ocean -c periodic_planar -r 20km -t region_reset_light_test
• 71: -o ocean -c periodic_planar -r 20km -t time_reset_light_test
• 72: -o ocean -c sea_mount -r 6.7km -t default
• 73: -o ocean -c single_column_model -r planar -t cvmix_test
• 74: -o ocean -c single_column_model -r sphere -t cvmix_test
• 75: -o ocean -c soma -r 16km -t 3layer
• 76: -o ocean -c soma -r 16km -t default
• 77: -o ocean -c soma -r 16km -t surface_restoring
• 78: -o ocean -c soma -r 32km -t 3layer
• 79: -o ocean -c soma -r 32km -t default
• 80: -o ocean -c soma -r 32km -t surface_restoring
• 81: -o ocean -c soma -r 32km -t time_varying_wind
• 82: -o ocean -c soma -r 4km -t 32to4km
• 83: -o ocean -c soma -r 4km -t 3layer
• 84: -o ocean -c soma -r 4km -t default
• 85: -o ocean -c soma -r 4km -t surface_restoring
• 86: -o ocean -c soma -r 8km -t 32to8km
• 87: -o ocean -c soma -r 8km -t 3layer
• 88: -o ocean -c soma -r 8km -t default
• 89: -o ocean -c soma -r 8km -t surface_restoring
• 90: -o ocean -c sub_ice_shelf_2D -r 5km -t Haney_number_init
• 91: -o ocean -c sub_ice_shelf_2D -r 5km -t Haney_number_iterative_init
• 92: -o ocean -c sub_ice_shelf_2D -r 5km -t default
• 93: -o ocean -c sub_ice_shelf_2D -r 5km -t iterative_init
• 94: -o ocean -c sub_ice_shelf_2D -r 5km -t restart_test
• 95: -o ocean -c sub_ice_shelf_2D -r 5km -t with_frazil
• 96: -o ocean -c ziso -r 10km -t default
• 97: -o ocean -c ziso -r 2.5km -t default
• 98: -o ocean -c ziso -r 20km -t default
• 99: -o ocean -c ziso -r 20km -t with_frazil
• 100: -o ocean -c ziso -r 5km -t default

Tests in bold failed, see below.

Building MPAS with python 3 will require updating to the latest releases of CVMix and Ocean-BGC after this PR is merged:
E3SM-Project/Ocean-BGC#1
and releases of CVMix and Ocean-BGC have been made with this fix included.

Failed tests

The following is a list of tests that failed (but for reasons I believe are unrelated to this PR):

• -o ocean -c global_ocean -r EC60to30 -t spin_up during simulation because of CFL violation:

CFLNumberGlobal = 0.0917271835580922, 0.122637831833784, 0.150533786272102, 
    0.17555059144412, 0.197662099329266, 0.217073518294376, 
    0.234323872726046, 0.250136386921142, 0.264790486294835, 
    0.27825253635325, 0.290323188086416, 0.301051958089178, 
    0.310209832190098, 0.3176288821503, 0.32330034866235, 0.328730874921286, 
    0.340932315339754, 0.39929006348568, 0.818147632488227, 1.93089241771095, 
    3.78167130632891, 9.49790843196592, 0.358590340863955

• -o ocean -c global_ocean -r NAEC60to30cr8 -t spin_up: the final (test_final_settings) phase seems to be misconfigured in various ways. Links to restart and Restart_timestamp are missing. timeSeriesStatsMonthly is enabled but there isn't a corresponding stream. It attempts to change to monthly restarts but the last restart is at 0001-01-07, which isn't an allowable combination in MPAS. In short, problems abound...
• -o ocean -c global_ocean -r QU60 -t spin_up during simulation for unknown reasons during the first time step
• -o ocean -c soma -r 4km -t 32to4km and -r 8km -t 32to8km because of missing MpasMeshGenerator.sh, see SOMA 32to8km and 32to4km test cases are broken #233
• -o ocean -c ziso -r 2.5km -t default runs out of memory during forward run:

Insufficient memory to allocate Fortran RTL message buffer, message #41 = hex 00000029.
Insufficient memory to allocate Fortran RTL message buffer, message #41 = hex 00000029.
Insufficient memory to allocate Fortran RTL message buffer, message #41 = hex 00000029.
slurmstepd: error: Detected 1 oom-kill event(s) in step 16129387.2 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.
srun: error: gr0510: tasks 0-2: Out Of Memory

@caozd999, could you change permissions on the 2 files above that you own (in /usr/projects/regionalclimate/COMMON_MPAS/ocean/grids/mesh_database on LANL IC) so I can read them? Probably, they should belong to the climate group. Check also for other files there than you own.

@caozd999, the 2.5 km ZISO test case now runs but crashes with a memory issue. I think it's just running on too few nodes/processors. Would you be able to update the default settings for that test case so it runs without running out of memory? Or is that something for @pwolfram?

@pwolfram, I tried increasing the ziso 2.5 km run to 144 cores (4 nodes) on Grizzly and I'm still running out of memory. Is it seeding particles in proportion to the number of cores so that adding more cores doesn't solve the memory problem?

In any case, I'm going to make a separate issue for this since it's not related to this PR.

I have now either successfully tested all test cases (55/61) or with this branch or created issues (#233, #234 and #235) to get issues that I think are unrelated to this PR addressed (6/61). I believe this branch is now ready for @mark-petersen to test and merge when ready.

@kristin-hoch and I tested together yesterday. We use the compass_py3.7 package and this PR and successfully ran QU240 init and QU240 test on grizzly and badger, compute node and front end node (except mpirun calls). This is great!

We will add and test the nightly regression suite this morning.

@xylar, @pwolfram @sbrus89 , I'd like to proceed with merge and test of several PRs. The proposed order is:

1. Adds LIGHT modes and capabilities (passive floats, improved sampling) #56 LIGHT update
2. Update sub-ice-shelf melt to remove coupled flag #164 small- ice shelf melt flag
3. merge coastal branch
4. rebase this one Update COMPASS ocean API for python 3 and using conda packages #229 and merge
5. rebase CONUS mesh update #201 CONUS mesh update

(4) and (5) will have minor conflicts. Is this OK with you both? I will ask you to rebase later today.
I commented on the MPAS-Tools PR previously. Sorry for the confusion.

@mark-petersen,

#56, #201 and ocean/coastal are all going to break python 3 support in this PR so I think they all need to be merged first, but the result is going to be another day or 2 of work for me here to add python 3 support to the scripts that are added by those and to fix outdated jigsaw calls and such. Regardless, it would be better to do this sooner rather than later.

#164 is completely independent of this work, so that one is fine.

Agreed. We will switch (4) and (5).

Okay, it's rebased. Please do some last tests and merge when ready.

@xylar thanks for your tremendous effort. This is a big accomplishment.

I tested the merged commit, all is OK. Will now make E3SM PR and then update nightly test suite.

Thanks very much for shepherding this through, @mark-petersen!