Quoting @philipwjones on E3SM PR 3396: "This is not likely to be an acceptable solution but can be a local mod if people need a workaround near-term."

I think a little more analysis is needed to justify the changes in this PR going onto the develop branch.

For what it's worth, there seems to be a similar issue in the flang compiler; see flang Issue 627. I think newer versions of the PGI compiler may be using LLVM, so this is perhaps not a coincidence.

@mgduda thanks for the ref to flang . This error/bug also appeared in an earlier PGI release but was fixed for a while. Now appears to be back. Regarding my statement that it might not be an acceptable solution, that referred to the fact that in the ocean all variables are default private (first private), so the extra private declaration here is probably redundant. But I didn't know what other components used as default in parallel regions. If it is also the case there, this mod is probably b4b for others too. What is the convention in the atm? Any chance you could run a quick check? I can try and look at other components. Since we need to use PGI on Summit for all GPU work, this is currently inhibiting that work. We'll keep pressing on the compiler side too.

@philipwjones and @amametjanov sorry I missed this. What is the status?

@mark-petersen PGI has reproduced the problem and it had been sent to engineers, but they warned the fix might not happen soon. Meanwhile, it's still broken on Summit. Might be nice to still have the short-term workaround in.

@mgduda could you also review this with the PR bunch today? The makefile flag does not affect others, and is a way for us to run with the PGI compiler and threading until PGI fixes the root problem.

The -DCPRPGI is added here to compiler sets pgi, pgi-llnl, pgi-nersc.

@mgduda can we merge this?

Sorry, all, for holding this up. I have no objections, and I'll leave an approving review; but first, I think it would be good if the merge commit message (which is usually taken to be the original description of the PR) contained a more detailed description of the problem that this PR is addressing, how the changes in the PR fix the problem, etc. So, I'd recommend describing:

1. What is the error this PR fixes (including a compiler error message, if applicable)?
2. Which versions of the PGI compiler are known to be broken (and which are known to work)?
3. How the error is fixed (in this case, with the addition of a new preprocessing macro in all pgi build targets, which activates different code in such-and-such code module)?

@amametjanov The description of the PR right now states that it "Remove[s] private attribute for pointers", but it looks like the entire private clause has been removed. As a non-expert in OpenMP, isn't it problematic that non-pointer, non-loop variables like nAdded are now no longer private? Are additional modifications needed in other cores (besides the ocean core) to make all variables default private?

@amametjanov if you could edit the first comment on this PR to include more detail and address these comments, then we can merge it in. I always use that first comment as the merge message.

PGI Update - was informed today (6/4) that the PGI fix for this has been implemented and should appear in the 20.7 (July) release. Think it's still useful to keep this fix in for when we may need to run with previous compiler versions.

Keeping non-pgi modules constant at

module load DefApps python/3.5.2 subversion/1.9.3 git/2.13.0 cmake/3.13.4 essl/6.1.0-2
 netlib-lapack/3.8.0 netcdf/4.6.1 netcdf-fortran/4.4.4 spectrum-mpi/10.3.1.2-20200121
 parallel-netcdf/1.8.1 hdf5/1.10.4

and varying 5 available pgi modules on Summit, the errors are as follows.
For pgi/18.7 and pgi/18.10, there is a build error:

PGF90-F-0000-Internal compiler error. print_token(): missing token       0
  (/gpfs/alpine/cli115/proj-shared/azamat/e3sm_scratch/e3sm-tests/20200604-cprpgi-macro/SMS_P42x2.T62_oQU120.CMPASO-NYF.summit_pgi.20200604_pgi18-7/bld/cmake-bld/framework/mpas_dmpar.f90: 8236)

with line 8236 having:

8235       !$omp do private(commListPtr, listPosition, bufferOffset, nAdded, fieldCursor, exchListPtr, iExch, iBuffer) 
8236       do listItem = 1, commListSize
8237         commListPtr => exchangeGroup % sendList

For pgi/19.9 (D), pgi/19.10 and pgi/20.1, runs are aborted during initialization:

Program terminated with signal 5, Trace/breakpoint trap.
#0  mpas_dmpar::mpas_dmpar_exch_group_pack_buffer_field1d_real ()
    at /gpfs/alpine/cli115/proj-shared/azamat/e3sm_scratch/e3sm-tests/20200604-cprpgi-macro/SMS_P42x2.T62_oQU120.CMPASO-NYF.summit_pgi.20200604_pgi20-1/bld/cmake-bld/framework/mpas_dmpar.f90:8448
8448              exchListPtr => fieldCursor % sendList % halos(haloLayer) % exchList

Regarding

Are additional modifications needed in other cores (besides the ocean core) to make all variables default private?

I don't think so. This is PGI- and pointer-specific.

Updated PR description above for a merge commit message.

@mgduda, the PR description in the commit message is now improved. Can you approve?