Adds LIGHT modes and capabilities (passive floats, improved sampling) #56

pwolfram · 2018-06-19T18:19:53Z

This merge adds passive particle floats, generalizes and improves LIGHT code, and adds support for horizontal interpolation of sampling for salinity, temperature, and BGC quantities.

It also fixes the condition where partial bottom cells allows passive floats to advect out of the wetted domain. The solution, following consultation with @maltrud, is to simply ensure that particles are contained in the viable range (ssh - dz/2, bottomDepth - dz/2), which essentially returns particles to the nearest wet cell center vertically.

pwolfram · 2018-06-19T18:22:21Z

cc @bradyrx, @xylar as we have discussed 3D passive float particle tracking in the past.

pwolfram · 2018-06-19T18:22:59Z

cc @sbrus89 as an example of an MPAS-O PR that we will use for testing.

pwolfram · 2018-06-19T19:17:46Z

Particle setup files are at testing_and_setup/compass/utility_scripts/LIGHTparticles

mark-petersen

@pwolfram I don't have time to test or review in detail. Considering that you only change a LIGHT file, if it passes your tests, go ahead and merge.

mark-petersen · 2018-06-20T08:02:43Z

@pwolfram please confirm that this PR passes the nightly test suite. Only the cases with LIGHT are relevant, but it's easy to test the whole thing.

src/core_ocean/analysis_members/mpas_ocn_lagrangian_particle_tracking.F

testing_and_setup/compass/utility_scripts/LIGHTparticles/make_particle_file.py

src/core_ocean/analysis_members/Registry_lagrangian_particle_tracking.xml

This merge adds passive particle floats, generalizes and improves LIGHT code, and adds support for horizontal interpolation of sampling for salinity, temperature, and BGC quantities. * pwolfram/particlePassiveFloatVerticalTreatmentFix: (63 commits) Corrects mis-matching LIGHT top and bottom interp doh. fixed the other PO4 typo fix typo for PO4 add functionality for PO4, NO3, SiO3, NH4, Fe, O2 sensors with horizontal interpolation Fix bug for BGC segfault Fixes an index bug in particle spatial interp Switches to Wachspress approach Resolves compilation error on intel compilers Fixes boundary cell overrun Fixes failed search for nearest hex vertex Adds interp calling routines for particle tracers Cleans up LIGHT scalar interpolation Horizontally interpolates particle scalars Refractors identification of LIGHT vertical index Refractors identification of LIGHT vertical index Adds recommended fields for globalID, lat/lon Updates 20km ziso default case to test particles Fixes typo with rename of particle temp/salinity Uncomment sampling logs and change sample default in registry Fix conditional clause issues ...

pwolfram · 2018-12-19T22:20:26Z

Verified that LIGHT is BFB against fd71e645 using SOMA test case for "spot checking" (the case that failed in pwolfram#21 to find the bug fixed by e4f2dc5.

UPDATE: there was an error in the testing uncovered via the regression test and this is being looked into right now.

mark-petersen · 2018-12-20T04:25:46Z

I successfully compiled the merge into ocean/develop on grizzly with gnu and intel. Ran nightly regression with gnu, it is not bfb with ocean/develop for zLevelParticle. I got this:

 ** Running case Periodic Planar 20km - LIGHT particle region reset test
   ** FAIL (See case_outputs/Periodic_Planar_20km_-_LIGHT_particle_region_reset_test for more information)
 ** Running case Periodic Planar 20km - LIGHT particle time reset test
   ** FAIL (See case_outputs/Periodic_Planar_20km_-_LIGHT_particle_time_reset_test for more information)

Is that expected? You can see the error file here:

/lustre/scratch4/turquoise/mpeterse/runs/nightly_ocean_model_181219_0c77a78_gr_gfortran_openmp_LIGHT/case_outputs/Periodic_Planar_20km_-_LIGHT_particle_*

only error is this one:

 ** FAIL Comparison of zLevelParticle between forward/analysis_members/lagrPartTrack.0001-01-01_00.00.00.nc and ...

everything else passes.

mark-petersen · 2018-12-20T13:07:42Z

It also failed this one:

 ** Running case ZISO 20km - Smoke Test
   ** FAIL (See case_outputs/ZISO_20km_-_Smoke_Test for more information)

because of this line:

testing_and_setup/compass/utility_scripts/LIGHTparticles/make_particle_file.py:
  from pyamg.classical import split, interpolate as amginterp

I tried to install pyamg in my local python installation but couldn't get it to work. I'm sure I just installed it wrong. If you can get that ZISO smoke test case to run successfully, go ahead and merge. Then we can make a PR into E3SM.

bradyrx · 2018-12-20T17:08:40Z

testing_and_setup/compass/utility_scripts/LIGHTparticles/make_particle_file.py

+import numpy as np
+from scipy import spatial
+from scipy import sparse
+from pyamg.classical import split, interpolate as amginterp


@pwolfram, line 19 should be wrapped in a try block per our call with ANL and @mark-petersen's error. This of course will break if the user tries the --downsample flag without pyamg, but that should be fine. For whatever reason, pyamg is not an easy install (took many tries for me).

@bradyrx thanks for the suggestion. I agree, put it in a try block, and have it actually fail only when the library is needed. That will keep our nightly testing suite more accessible and portable across users and machines.

Ideally we eventually need a better solution than pyamg for coarsening, but I don't think it's crucial for this PR. It's a feasible solution for oQU240, but takes ~40min for 60to30 and ~20hr for 30to10.

Note my current hacky solution here: https://gist.github.com/bradyrx/ddb6c944e425ac5e63b8e8e7361809dd

I'm just running make_particle_file.py without downsampling, and then removing full columns with that gist by going through lat/lon with a fixed stride.

This approximately pulls columns at random due to the unstructured mesh indexing, but I don't suggest it as a formal method in ocean/develop.

Here, white dots are seeding at every grid cell and red are after running my script:

LIGHT BFB against ocean/develop Cleans up LIGHT helper functions and signatures BFB with minor clean up changes (commit f6a7ff8c8848dc02b0c028b70d58e2dfada891cd)

pwolfram · 2019-04-29T18:08:48Z

@maltrud took a look at a BGC G-case application with particles on and didn't find any problems (example below):

@mark-petersen, this is ready to merge when you have a chance.

bradyrx · 2019-04-29T20:14:54Z

@pwolfram, thanks for handling this! Testing from @maltrud confirms that passive float sensors are now good for physics and biogeochem.

pwolfram · 2019-05-01T21:38:18Z

@mark-petersen, please let me know if there is anything else you need us to do before this gets merged-- it is ready for your final approval and merge. Thanks!

bradyrx · 2019-05-01T21:49:11Z

@pwolfram, I believe that this PR still breaks if a run is restarted without all procs assigned to at least one particle. Unless you fixed that issue? @maltrud and I were going to just manually bump a particle to the empty proc(s), although that might cause load-balancing issues.

pwolfram · 2019-05-01T21:54:06Z

@bradyrx, please see #224, which is a separate and independent issue from this PR (which is nearly a year old). We should engage there so this huge chunk of work can get merged without further scope creep.

mark-petersen · 2019-05-08T13:10:17Z

Final test and merge today. Talk to me before changing anything.

@maltrud

…ix' into ocean/develop This merge adds passive particle floats, generalizes and improves LIGHT code, and adds support for horizontal interpolation of sampling for salinity, temperature, and BGC quantities. It also fixes the condition where partial bottom cells allows passive floats to advect out of the wetted domain. The solution, following consultation with @maltrud, is to simply ensure that particles are contained in the viable range (ssh - dz/2, bottomDepth - dz/2), which essentially returns particles to the nearest wet cell center vertically.

pwolfram · 2019-05-08T13:48:52Z

@mark-petersen, please merge when you get the chance, I won't make any changes-- thanks!

mark-petersen · 2019-05-08T14:59:14Z

Tested in mega-merge with MPAS nightly regression suite with gnu and intel, opt and debug, on grizzly. bfb on all tests expect were expected:

** Running case ZISO 20km - Smoke Test
   ** FAIL (See case_outputs/ZISO_20km_-_Smoke_Test for more information)
 ** Running case Periodic Planar 20km - LIGHT particle region reset test
   ** FAIL (See case_outputs/Periodic_Planar_20km_-_LIGHT_particle_region_reset_test for more information)
 ** Running case Periodic Planar 20km - LIGHT particle time reset test
   ** FAIL (See case_outputs/Periodic_Planar_20km_-_LIGHT_particle_time_reset_test for more information)

See more details here, but everything looks in order.

/lustre/scratch4/turquoise/mpeterse/runs/nightly_ocean_model_190508_a1256fb_gr_gfortran_openmp_pr_test

Testing mega-merge in E3SM now. Includes coastal branch as well.

bradyrx · 2019-05-08T15:50:26Z

@mark-petersen @pwolfram, with this going into develop now, it might be a good time to point out my repo https://github.com/bradyrx/automate-mpas. I'll be down there next week so maybe we can talk about a path forward with this. It might be nice to get some code review here and ideas.

But basically, one just needs to alter the config.sh file and run generate_e3sm_case.sh. It'll manage every step of the E3SM/MPAS case setup and launch the run. This is built mainly to manage all of the minute steps of getting LIGHT to run properly (and is especially useful if you're running a bunch of simulations for scaling tests). The python files alter all of the streams files, etc. to set up proper supercycling, sensor modes, particle initialization, I/O, and of course base settings for MPAS.

mark-petersen · 2019-05-08T21:08:29Z

test of mega-merge passed all in E3SM:

PASS SMS_D_Ln9.T62_oQU120_ais20.MPAS_LISIO_TEST.cori-knl_gnu RUN time=147
PASS SMS_D_Ln9.T62_oQU120_ais20.MPAS_LISIO_TEST.cori-knl_intel RUN time=397
PASS ERS_Ld9.T62_oEC60to30v3wLI.GMPAS-DIB-IAF-ISMF.cori-knl_gnu COMPARE_base_rest
PASS PEM_Ln9.ne30_oECv3wLI.A_WCYCL1850-DIB-ISMF_CMIP6.cori-knl_gnu COMPARE_base_modpes
PASS PET_Ln3.T62_oEC60to30v3wLI.GMPAS-DIB-IAF-ISMF.cori-knl_gnu COMPARE_base_single_thread

Includes: 1. #56 LIGHT update 2. #164 clean up ice shelf melt flag 3. merge coastal branch 4. #201 CONUS mesh update

pwolfram · 2019-05-09T14:00:24Z

Thanks @mark-petersen

Includes: 1. LIGHT update (MPAS-Ocean Lagrangian particles): MPAS-Dev/MPAS-Model#56 2. clean up ice shelf melt flag: MPAS-Dev/MPAS-Model#164 3. merge coastal branch: MPAS-Dev/MPAS-Model#54 4. CONUS mesh update: MPAS-Dev/MPAS-Model#201 5. Major COMPASS upgrade to python 3: MPAS-Dev/MPAS-Model#229 6. Eliminate outdated -DUNDERSCORE flag: MPAS-Dev/MPAS-Model#156 7. PIO Version Compatibility Check - PGI Compiler Bug Fix: MPAS-Dev/MPAS-Model#212 These changes do not alter E3SM standard cases or testing on next. COMPASS is the MPAS testing infrastructure. LIGHT and coastal changes are protected by flags and not used in normal E3SM tests.

Update mpas source This PR update the mpas-source git submodule but only includes changes that will not impact E3SM. These changes include: * LIGHT update (MPAS-Ocean Lagrangian particles): MPAS-Dev/MPAS-Model#56 * clean up ice shelf melt flag: MPAS-Dev/MPAS-Model#164 * merge coastal branch: MPAS-Dev/MPAS-Model#54 * CONUS mesh update: MPAS-Dev/MPAS-Model#201 * Major COMPASS upgrade to python 3: MPAS-Dev/MPAS-Model#229 * Eliminate outdated -DUNDERSCORE flag: MPAS-Dev/MPAS-Model#156 * PIO Version Compatibility Check - PGI Compiler Bug Fix: MPAS-Dev/MPAS-Model#212 These changes do not alter E3SM standard cases or tested configurations. COMPASS is the MPAS testing infrastructure. LIGHT and coastal changes are protected by flags and not used in normal E3SM tests. [NML] [BFB]

@maltrud

…ix' into ocean/develop This merge adds passive particle floats, generalizes and improves LIGHT code, and adds support for horizontal interpolation of sampling for salinity, temperature, and BGC quantities. It also fixes the condition where partial bottom cells allows passive floats to advect out of the wetted domain. The solution, following consultation with @maltrud, is to simply ensure that particles are contained in the viable range (ssh - dz/2, bottomDepth - dz/2), which essentially returns particles to the nearest wet cell center vertically.

@maltrud

…ix' into ocean/develop This merge adds passive particle floats, generalizes and improves LIGHT code, and adds support for horizontal interpolation of sampling for salinity, temperature, and BGC quantities. It also fixes the condition where partial bottom cells allows passive floats to advect out of the wetted domain. The solution, following consultation with @maltrud, is to simply ensure that particles are contained in the viable range (ssh - dz/2, bottomDepth - dz/2), which essentially returns particles to the nearest wet cell center vertically.

@maltrud

…ix' into ocean/develop This merge adds passive particle floats, generalizes and improves LIGHT code, and adds support for horizontal interpolation of sampling for salinity, temperature, and BGC quantities. It also fixes the condition where partial bottom cells allows passive floats to advect out of the wetted domain. The solution, following consultation with @maltrud, is to simply ensure that particles are contained in the viable range (ssh - dz/2, bottomDepth - dz/2), which essentially returns particles to the nearest wet cell center vertically.

* "GF coldpool updates and bugfix for radiation coupling" * "Update formating following Mike's suggestions"

pwolfram added bug Ocean labels Jun 19, 2018

pwolfram requested a review from mark-petersen June 19, 2018 18:22

mark-petersen approved these changes Jun 20, 2018

View reviewed changes

pwolfram commented Jun 26, 2018

View reviewed changes

src/core_ocean/analysis_members/mpas_ocn_lagrangian_particle_tracking.F Outdated Show resolved Hide resolved

pwolfram force-pushed the particlePassiveFloatVerticalTreatmentFix branch 4 times, most recently from eb96bd1 to 13e5e99 Compare July 5, 2018 20:38

pwolfram added the Don't merge label Jul 6, 2018

pwolfram commented Jul 13, 2018

View reviewed changes

testing_and_setup/compass/utility_scripts/LIGHTparticles/make_particle_file.py Outdated Show resolved Hide resolved

pwolfram force-pushed the particlePassiveFloatVerticalTreatmentFix branch from 6a645ba to 3036516 Compare July 14, 2018 16:37

bradyrx reviewed Aug 1, 2018

View reviewed changes

src/core_ocean/analysis_members/Registry_lagrangian_particle_tracking.xml Outdated Show resolved Hide resolved

pwolfram changed the title ~~Fixes vertical boundary condition for LIGHT passive floats~~ Adds LIGHT modes and capabilities (passive floats, improved sampling) Dec 19, 2018

pwolfram mentioned this pull request Dec 19, 2018

Adds scalar interpolation capability for LIGHT tracers pwolfram/MPAS-Model#21

Merged

pwolfram removed the bug label Dec 19, 2018

bradyrx suggested changes Dec 20, 2018

View reviewed changes

pwolfram added 3 commits April 20, 2019 14:04

Adds global cell ID tracking to LIGHT

1c1c46b

Adds lat / lon output to LIGHT particles

11c3876

LIGHT code cleanups

c300608

LIGHT BFB against ocean/develop Cleans up LIGHT helper functions and signatures BFB with minor clean up changes (commit f6a7ff8c8848dc02b0c028b70d58e2dfada891cd)

pwolfram force-pushed the particlePassiveFloatVerticalTreatmentFix branch 2 times, most recently from 1654b1f to fb8d7de Compare April 24, 2019 20:16

bradyrx approved these changes Apr 29, 2019

View reviewed changes

pwolfram mentioned this pull request May 1, 2019

LIGHT requires a single particle in each processor domain #224

Open

mark-petersen mentioned this pull request May 8, 2019

Update COMPASS ocean API for python 3 and using conda packages #229

Merged

mark-petersen merged commit d9c9505 into MPAS-Dev:ocean/develop May 9, 2019

mark-petersen added a commit that referenced this pull request May 9, 2019

Merge branch 'ocean/develop' into e3sm/develop

cb33532

Includes: 1. #56 LIGHT update 2. #164 clean up ice shelf melt flag 3. merge coastal branch 4. #201 CONUS mesh update

xylar mentioned this pull request May 9, 2019

Missing script from SOMA 32km default test case #239

Closed

pwolfram deleted the particlePassiveFloatVerticalTreatmentFix branch May 9, 2019 14:00

pwolfram added the LIGHT MPAS-O LIGHT tag label May 9, 2019

mark-petersen mentioned this pull request May 11, 2019

Update mpas source E3SM-Project/E3SM#2918

Merged

matthewhoffman added Don't merge Ocean labels Mar 17, 2021

barlage pushed a commit to barlage/MPAS-Model that referenced this pull request Nov 6, 2024

GF coldpool updates and bugfix for radiation coupling (MPAS-Dev#56)

3ce58de

* "GF coldpool updates and bugfix for radiation coupling" * "Update formating following Mike's suggestions"

Adds LIGHT modes and capabilities (passive floats, improved sampling) #56

Adds LIGHT modes and capabilities (passive floats, improved sampling) #56

Uh oh!

Conversation

pwolfram commented Jun 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwolfram commented Jun 19, 2018

Uh oh!

pwolfram commented Jun 19, 2018

Uh oh!

pwolfram commented Jun 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mark-petersen left a comment

Choose a reason for hiding this comment

Uh oh!

mark-petersen commented Jun 20, 2018

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pwolfram commented Dec 19, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mark-petersen commented Dec 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mark-petersen commented Dec 20, 2018

Uh oh!

bradyrx Dec 20, 2018

Choose a reason for hiding this comment

Uh oh!

mark-petersen Dec 20, 2018

Choose a reason for hiding this comment

Uh oh!

bradyrx Dec 20, 2018

Choose a reason for hiding this comment

Uh oh!

bradyrx Dec 20, 2018

Choose a reason for hiding this comment

Uh oh!

pwolfram commented Apr 29, 2019

Uh oh!

bradyrx commented Apr 29, 2019

Uh oh!

pwolfram commented May 1, 2019

Uh oh!

bradyrx commented May 1, 2019

Uh oh!

pwolfram commented May 1, 2019

Uh oh!

mark-petersen commented May 8, 2019

Uh oh!

pwolfram commented May 8, 2019

Uh oh!

mark-petersen commented May 8, 2019

Uh oh!

bradyrx commented May 8, 2019

Uh oh!

mark-petersen commented May 8, 2019

Uh oh!

pwolfram commented May 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pwolfram commented Jun 19, 2018 •

edited

Loading

pwolfram commented Jun 19, 2018 •

edited

Loading

pwolfram commented Dec 19, 2018 •

edited

Loading

mark-petersen commented Dec 20, 2018 •

edited

Loading