plot_initial_state.py is taking forever and running out of memory #663
Description
Even for meshes that aren't particularly large by our standards (e.g. EC30to60), plot_initial_state.py is taking a very long time to run and is running out of memory:
Traceback (most recent call last):
File "./run.py", line 40, in <module>
subprocess.check_call(['./plot_initial_state.py'])
File "/usr/projects/climate/SHARED_CLIMATE/anaconda_envs/base/envs/test_compass_0.1.9/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./plot_initial_state.py']' died with <Signals.SIGKILL: 9>.
Traceback (most recent call last):
File "./run.py", line 77, in <module>
File "/usr/projects/climate/SHARED_CLIMATE/anaconda_envs/base/envs/test_compass_0.1.9/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./run.py']' returned non-zero exit status 1.
slurmstepd: error: Detected 1 oom-kill event(s) in step 17109912.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.