@vanroekel, this is great! I added the same fix for salinity and ecosystem tracers, as you'll see above.

@mark-petersen, thanks for reviewing and merging those other three, much appreciated! Once this one is merged, I'll rebase my various other in-progress PRs.

I rebased and added T, S, and layerthickness plots. This plot is now automatically generated in init/initial_state/initial_state.png

At least a visual check of the range could give us another way to avoid these errors, but the person generating the initial state would have to know to look at it.

When running initial_state you now see this at the end with the text output. Here is the EC60to30 64 layer:

plotting histograms of the initial condition
see: init/initial_state/initial_state.png
MPAS-Ocean initial state
date: 04/06/2020
number cells: 236771
number cells, millions:  0.237
number layers: 64

  min val   max val  variable name
  5.00e+00  6.40e+01 maxLevelCell
  1.27e+01  6.00e+03 bottomDepth
 -2.10e+00  3.05e+01 temperature
  0.00e+00  4.13e+01 salinity
  0.00e+00  2.09e+02 layerThickness
  0.00e+00  3.12e-01 rx1Edge

@mark-petersen, your plotting and stats routines don't seem to be masking out cells below maxLevelCell for T, S and layer thickness. This is leading to a bunch of zero values and an inaccurate min for salinity and layer thickness. I think we need to fix that.

But I'm very happy to have these histograms!

Thanks for the comment. I will mask those out soon.

I already did in #512

@xylar and @vanroekel, I created these, start to finish, on the current head of ocean/develop 5345dd3 after merging this PR:

EC60to30 64 layer: 
  /lustre/scratch4/turquoise/mpeterse/runs/200406_test_initial_state_od/ocean/global_ocean/EC60to30/
EC60to30wISC 60 layer:
  /lustre/scratch4/turquoise/mpeterse/runs/200406_test_initial_state_od/ocean/global_ocean/EC60to30wISC

Both got through the 20-day spin-up without any trouble.

Great! I'm going to work on the SO60to10wISC tonight or tomorrow.