Fix bug setting destination grid object sigma vector passed to MESSy

[lizziel]

Name and Institution (Required)

Name: Lizzie Lundgren
Institution: Harvard University

Describe the update

While investigating differences in CESM output when using HEMCO with different core counts I discovered a bug in vertical regridding using MESSy. The MESSy regridding scheme is used by models that input 3D atmospheric data on a different vertical grid than model grid, e.g. CESM and WRF-GC but not GEOS-Chem Classic or GCHP.

The collapse of the HEMCO grid 3D pressure sigma array into a 1D vector passed to the MESSy regridding scheme is at odds with how it is used within MESSy. Swapping two dimensions of the collapse (lat and lon) fixes the bug. This bug is not part of MESSy (written in 2002) but the interface to MESSy within HEMCO (written in 2014). It has been present since MESSy was originally implemented into HEMCO.

The bug causes the wrong horizontal layout of sigma pressure profiles during vertical regridding. In MPI models, the profiles within a given core are correct but are distributed across the regional grid of that core incorrectly.

Expected changes

This update only affects models which use HEMCO to vertically regrid input data that are not on the model vertical grid, i.e. CESM and WRF-GC. The extent of the impact is dependent on the model decomposition and how many CPUs are used.

To assess the impact I ran a 3-hr CESM run using HEMCO with CAM-chem. All emissions were turned off except for AEIC CO emissions, which were computed and output as a diagnostic every timestep. The following results show the impact of the bug at hour 3 of the run (instantaneous output). All plots show a single level where CO emissions from aviation tracks are visible.

Before and after the fix at 72 cores:

Before and after the fix at 144 cores:

72 cores vs 144 cores with the bug present shows the core-dependency:

72 cores vs 144 cores with bug fix applied shows that the core-dependency is no longer there:

Reference(s)

None

Related Github Issue

closes #315
ESCOMP/CAM#856

[yantosca]

Thanks @lizziel. Such a complex bug caused by swapping 2 lines of code! Good to merge.

[lizziel]

I should add that "round psfc" appears in the plot titles because I applied rounding to surface pressure. This is due to a second source of core-dependency differences in 3D emissions, although one with less impact. The current implementation of ESMF regridding from the CAM grid to the HEMCO grid in the HEMCO_CESM interface introduces numerical noise. To get around this I rounded surface pressure post-regrid to achieve identical surface pressure in both the 72 cores and 144 cores runs. To follow progress on that bug see ESCOMP/CAM#856.