Move jigsaw_to_MPAS to mpas_tools by xylar · Pull Request #563 · MPAS-Dev/MPAS-Model

xylar · 2020-05-18T19:13:05Z

This merge uses mpas_tools.mesh.creation instead of jigsaw_to_MPAS. These tools are being moved to MPAS-Tools to allow them to be used by other workflows, see #550.

This merge includes:

removing the jigsaw_to_MPAS module from COMPASS
Updating the compass environment version to the one that includes mpas_tools with mesh.creation.
Removing links to jigsaw_to_MPAS from several test cases
Calling build_mesh and inject_bathymetry from mpas_tools in test cases that use these
Switching from mpas_tools.conversion to mpas_tools.mesh.conversion (though the former still exists for backward compatibility)
A small fix to some cartopy-related axis calls that seem to be depricated

closes #550

xylar · 2020-05-18T19:15:31Z

This PR requires the changes to MPAS-Tools in MPAS-Dev/MPAS-Tools#311

xylar · 2020-05-18T19:16:46Z

Testing

So far, I successfully tested QU240/init and QU240wISC/init test cases on my laptop, but I didn't check for bit-for-bit and such. Much more testing is obviously needed.

xylar · 2020-05-18T19:19:01Z

@mark-petersen and @pwolfram, this PR likely needs to go in at a point when ocean/develop and ocean/coastal are in sync with one another. Can we try to coordinate that sometime soon? Obviously, I can update any test cases on ocean/coastal that have been missed here, but I want to avoid any unnecessary frustration from this transition.

pwolfram · 2020-05-26T20:48:37Z

@xylar, can you please clarify what you are hoping to have in terms of coordination? This is obviously a big change. My preference would be for us to migrate all test cases from ocean/coastal over before doing this change. @caozd999 has one (#552) and I have a few too that aren't in PR stage yet.

testing_and_setup/compass/ocean/coastal/USDEQU120cr10rr2/build_mesh/config_culled_mesh.xml

testing_and_setup/compass/ocean/global_ocean/config_files/config_mesh_metrics.xml

xylar · 2020-05-26T20:51:21Z

My preference would be for us to migrate all test cases from ocean/coastal over before doing this change.

Yep, that's exactly what I mean. We should only merge this at a point where ocean/develop and ocean/coastal are identical and all test cases then get updated with these changes. I hope that can happen in the not too distant future so there isn't too much that has to be redone, but I realize this might be a bit tricky to coordinate.

pwolfram · 2020-05-26T20:51:48Z

Thanks @xylar, I understand now and agree.

pwolfram · 2020-05-26T20:52:24Z

@mark-petersen-- thoughts on when you want to do the ocean/coastal and ocean/develop merge?

xylar · 2020-08-21T11:54:36Z

This should be ready to rebase right after #485 gets merged.

xylar · 2020-08-23T09:52:47Z

Testing

I am testing with a test build of compass 0.1.9 with mpas_tools 0.0.11 (the version that includes mpas_tools.mesh.creation). I am using ocean/develop with the fixes from #662 as a baseline (meaning it will still use jigsaw_to_MPAS) and this branch for comparison.

I am testing through the initial_state (or equivalent) stage and sometimes through the test stage if there is one and it is short enough.

Baselines are here:

/lustre/scratch4/turquoise/xylar/test_fix_broken/

Runs with this branch are here:

/lustre/scratch4/turquoise/xylar/test_mpas_tools_mesh_creation/

Many, many coastal test cases are broken and I am doing my best to fix them in #662. This will become the baseline.

Baseline (from #662) finished:

Tests succeeded and bit-for-bit:

mark-petersen

@xylar I know there are some complications with timing and order here. Go ahead and merge when you are ready, and we can make additional changes if needed in another PR.

xylar · 2020-08-24T09:19:31Z

Go ahead and merge when you are ready, and we can make additional changes if needed in another PR.

I don't think that's a safe attitude for this PR. I think every affected test case needs to be checked to make sure it is bit-for-bit before and after these changes. Otherwise, we will not have confidence in such an extensive set of changes, and it will likely affect everyone working with non-idealized test cases for awhile going forward.

mark-petersen · 2020-08-26T14:25:40Z

@xylar, I'm trying to test this. I need to load compass 0.1.9 on grizzly. The load latest is 0.1.8. I hate to say it, but I don't know how to create compass 0.1.9 locally myself. Are there instructions? I should really know how to do it.

xylar · 2020-08-26T14:37:53Z

@mark-petersen, please use test_compass_0.1.9. MPAS-Dev/MPAS-Tools#311 (comment)

xylar · 2020-08-26T14:39:18Z

I hate to say it, but I don't know how to create compass 0.1.9 locally myself. Are there instructions? I should really know how to do it.

It's quite complicated. I have to make test builds of both mpas_tools and compass. I then upload them to my own anaconda channel and then install them on grizzly. Nothing I want you to worry about for now.

xylar · 2020-08-26T14:48:38Z

Just FYI, I do plan to spend a considerable amount of time in the next 3-4 weeks writing documentation, and this will be part of the process I will be documenting.

mark-petersen · 2020-08-26T17:11:01Z

Thanks @xylar . I set up and ran test cases on grizzly with test_compass_0.1.9 (I see how to query the versions now). I used the new compass to set up and run the nightly regression suite, which passed a bfb comparison with the previous ocean/develop. I also used the setup command for an EC60to30 mesh init case, which is taking a long time in the mesh culler right now.

xylar · 2020-08-26T17:31:43Z

which is taking a long time in the mesh culler right now

Like I've said, the mesh conversion tools really need to be replaced with xarray/dask versions that are faster and parallel. That would be a great project for the right student or postdoc...

xylar · 2020-08-26T17:32:51Z

And thanks very much for testing. Let me know how it goes with EC60to30.

xylar · 2020-08-26T17:33:10Z

@sbrus89, is this something you will have time to test today?

mark-petersen · 2020-08-26T17:33:55Z

Let me know how it goes with EC60to30.

It ran through init and began spin-up successfully. Also just ran QU240wISC successfully.

sbrus89 · 2020-08-26T18:03:43Z

@xylar, I'll try to run some tests this afternoon

xylar · 2020-08-26T19:34:01Z

@sbrus89, wonderful! That would be much appreciated!

mark-petersen · 2020-08-26T20:18:15Z

This one worked: USDEQU240at60cr20rr4/build_mesh. I'm done testing. Will merge after approval from @sbrus89.

xylar · 2020-08-26T20:23:03Z

Will merge after approval from @sbrus89.

The safer order of operations would be for me to create compass_0.1.9 officially and then we merge this. That would be my preference, since folks trying to use test cases after this merge will need to know that they need to load the test_compass_0.1.9 environment and I don't want people to get in the habit of using the test environments for anything but testing the PRs they were created for.

mark-petersen · 2020-08-26T20:26:01Z

OK. What if you do what's needed tomorrow morning Berlin time for compass 0.1.9, and I'll merge this tomorrow morning NM time?

xylar · 2020-08-26T20:27:15Z

Perfect! (Assuming @sbrus89's testing goes well.)

sbrus89

Approved after testing:

-o ocean -c hurricane -r USDEQU240at60cr20rr4WD -t build_mesh
-o ocean -c hurricane -r USDEQU120at30cr10rr2WD -t build_mesh
-o ocean -c tides -r USDEQU120at30cr10 -t build_mesh

Tested with branch that merges #563 and #662 on Grizzly using test_compass_0.1.9

xylar · 2020-08-27T13:14:31Z

@mark-petersen, compass_0.1.9 is now in place on LANL IC, Cori, Compy, Cooley and Anvil. You may merge this at your convenience.

Merge PR #563 'xylar/ocean/remove_jigsaw_to_mpas' into ocean/develop Merge PR #662 'xylar/ocean/fix_broken_compass_tests' into ocean/develop Merge PR #664 'xylar/ocean/fix_plot_initial_state' into ocean/develop

…3800) Update mpas-source: compass only This PR brings in a new mpas-source submodule with changes only to the ocean core. All of the changes are to the COMPASS testing code that is not used in E3SM, so this PR should have no effect on E3SM. The COMPASS commits are: * MPAS-Dev/MPAS-Model#563 'xylar/ocean/remove_jigsaw_to_mpas' * MPAS-Dev/MPAS-Model#662 'xylar/ocean/fix_broken_compass_tests' * MPAS-Dev/MPAS-Model#664 'xylar/ocean/fix_plot_initial_state' [BFB]

Update mpas-source: compass only This PR brings in a new mpas-source submodule with changes only to the ocean core. All of the changes are to the COMPASS testing code that is not used in E3SM, so this PR should have no effect on E3SM. The COMPASS commits are: * MPAS-Dev/MPAS-Model#563 'xylar/ocean/remove_jigsaw_to_mpas' * MPAS-Dev/MPAS-Model#662 'xylar/ocean/fix_broken_compass_tests' * MPAS-Dev/MPAS-Model#664 'xylar/ocean/fix_plot_initial_state' [BFB]

…develop Move jigsaw_to_MPAS to mpas_tools MPAS-Dev#563 This merge uses `mpas_tools.mesh.creation` instead of `jigsaw_to_MPAS`. These tools are being moved to MPAS-Tools to allow them to be used by other workflows, see MPAS-Dev#550. This merge includes: * removing the `jigsaw_to_MPAS` module from COMPASS * Updating the `compass` environment version to the one that includes * `mpas_tools` with `mesh.creation`. * Removing links to `jigsaw_to_MPAS` from several test cases * Calling `build_mesh` and `inject_bathymetry` from `mpas_tools` in test * cases that use these * Switching from `mpas_tools.conversion` to `mpas_tools.mesh.conversion` * (though the former still exists for backward compatibility) * A small fix to some cartopy-related axis calls that seem to be * depricated

xylar added Ocean COMPASS in progress coastal labels May 18, 2020

xylar requested review from mark-petersen, pwolfram and sbrus89 May 18, 2020 19:13

xylar assigned mark-petersen May 18, 2020

xylar mentioned this pull request May 19, 2020

Add mesh creation module to mpas_tools conda package MPAS-Dev/MPAS-Tools#311

Merged

pwolfram reviewed May 26, 2020

View reviewed changes

testing_and_setup/compass/ocean/coastal/USDEQU120cr10rr2/build_mesh/config_culled_mesh.xml Outdated Show resolved Hide resolved

pwolfram reviewed May 26, 2020

View reviewed changes

testing_and_setup/compass/ocean/global_ocean/config_files/config_mesh_metrics.xml Outdated Show resolved Hide resolved

xylar force-pushed the ocean/remove_jigsaw_to_mpas branch from 775a3ac to 669423c Compare May 27, 2020 13:36

xylar force-pushed the ocean/remove_jigsaw_to_mpas branch from 669423c to 377e8a7 Compare July 6, 2020 13:41

xylar force-pushed the ocean/remove_jigsaw_to_mpas branch from 377e8a7 to ed42331 Compare August 21, 2020 11:39

xylar mentioned this pull request Aug 21, 2020

QU240wISC test broken #658

Closed

mark-petersen mentioned this pull request Aug 21, 2020

Merge coastal to ocean/develop: COMPASS and sediment additions #485

Merged

xylar force-pushed the ocean/remove_jigsaw_to_mpas branch 2 times, most recently from 8e3c94e to e647f20 Compare August 22, 2020 18:44

mark-petersen approved these changes Aug 24, 2020

View reviewed changes

xylar mentioned this pull request Aug 24, 2020

Fix eighteen broken coastal test cases #662

Merged

xylar mentioned this pull request Aug 26, 2020

New Mesh: EC15to60kmL64E3SMv2r01: Draft of new EC mesh with refinement around Greenland #522

Closed

sbrus89 approved these changes Aug 26, 2020

View reviewed changes

mark-petersen merged commit dcfcc3f into MPAS-Dev:ocean/develop Aug 27, 2020

xylar deleted the ocean/remove_jigsaw_to_mpas branch August 27, 2020 17:31

mark-petersen mentioned this pull request Aug 27, 2020

update mpas-source: compass only E3SM-Project/E3SM#3800

Merged

xylar mentioned this pull request Nov 6, 2020

Move jigsaw_to_MPAS to MPAS-Tools #550

Closed

matthewhoffman added coastal COMPASS in progress Ocean labels Mar 17, 2021

Conversation

xylar commented May 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xylar commented May 18, 2020

Uh oh!

xylar commented May 18, 2020

Testing

Uh oh!

xylar commented May 18, 2020

Uh oh!

pwolfram commented May 26, 2020

Uh oh!

Uh oh!

Uh oh!

xylar commented May 26, 2020

Uh oh!

pwolfram commented May 26, 2020

Uh oh!

pwolfram commented May 26, 2020

Uh oh!

xylar commented Aug 21, 2020

Uh oh!

xylar commented Aug 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

mark-petersen left a comment

Choose a reason for hiding this comment

Uh oh!

xylar commented Aug 24, 2020

Uh oh!

mark-petersen commented Aug 26, 2020

Uh oh!

xylar commented Aug 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xylar commented Aug 26, 2020

Uh oh!

xylar commented Aug 26, 2020

Uh oh!

mark-petersen commented Aug 26, 2020

Uh oh!

xylar commented Aug 26, 2020

Uh oh!

xylar commented Aug 26, 2020

Uh oh!

xylar commented Aug 26, 2020

Uh oh!

mark-petersen commented Aug 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sbrus89 commented Aug 26, 2020

Uh oh!

xylar commented Aug 26, 2020

Uh oh!

mark-petersen commented Aug 26, 2020

Uh oh!

xylar commented Aug 26, 2020

Uh oh!

mark-petersen commented Aug 26, 2020

Uh oh!

xylar commented Aug 26, 2020

Uh oh!

sbrus89 left a comment

Choose a reason for hiding this comment

Uh oh!

xylar commented Aug 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

xylar commented May 18, 2020 •

edited

Loading

xylar commented Aug 23, 2020 •

edited

Loading

xylar commented Aug 26, 2020 •

edited

Loading

mark-petersen commented Aug 26, 2020 •

edited

Loading