This was tested using a QU240 test case, and a EC60to30 test case (both generated via Compass). The results are bit-for-bit for debug cases (and should be for optimized cases as well) with Intel compiler on Cori.

As a result of the optimizations in this PR (specifically, removing the unused calculations for computeBudgets), there is a reduction of the time spent in tracer adv by 16.4% (average of 3 runs for both baseline and optimized)

I updated this PR to include the recent changes to PR #496 and the necessary updates to tracer advection resulting from the changes in #496

I ran into some issues resolving merge conflicts during rebase, so instead I started over and re-applied the relevant changes.

I had to leave the computeBudgets code in since it calculates values used elsewhere. I either missed that previously, or there were recent changes that now use the arrays. As a result, the performance improvement drops from 16.4% in tracer adv to 1.6%. This PR is still necessary, though, since it updates the tracer advection routines to use the ocn_mesh module.

This is BFB for the nightly regression suite when compiled w/ intel (both DEBUG and OPT).

I need to test this in E3SM before its deemed ready.

PASSes

SMS_D.T62_oQU120_ais20.MPAS_LISIO_TEST.cori-haswell_gnu
SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.cori-haswell_intel
PET_Ln3.T62_oEC60to30v3wLI.GMPAS-DIB-IAF-ISMF.cori-haswell_intel
PET_Ln3.T62_oEC60to30v3wLI.GMPAS-DIB-IAF-ISMF.cori-haswell_gnu
PEM_Ln9.T62_oQU240.GMPAS-IAF.cori-haswell_gnu
PEM_Ln9.T62_oQU240.GMPAS-IAF.cori-haswell_intel
PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_intel
SMS_P256x2.T62_oEC60to30v3.CMPASO-NYF.summit_pgi

So I think this is ready to go into E3SM.

Rebased to be safe. Then tested nightly regression suite. Passes all with gnu 6 debug on badger, intel 19 debug on grizzly, and is bfb in optimized with gnu 6 and intel 19 against same compilers with branch ocean/develop.

I also looked through the code. This all looks correct. Thanks @mattdturner for your work. I love the simplification of obtaining mesh variables from the module.

@jonbob this is ready and will need it's own E3SM PR. We can merge this PR when we have a free slot.