Add support for CMake to GCHP (updated)

[LiamBindle]

Hi everyone,

This is an updated PR (replacing PR #23) adding CMake support to GCHP.

After the changes in 12.4.0, I did a bit of an overhaul of the GC-Classic build system. The main goal of the overhaul was simplifying things and to cleaning things up. Because of the changes in 12.4.0 (especially the removal of compiler definitions by FlexGrid), I was able to make some nice simplifications to the build system.

As a part of the overhaul, I took some time to come up with rules for controlling the GEOS-Chem build from an external project (e.g. GCHP), and this new PR uses those rules to control how the code in GC-Classic is built. This means the GC-Classic build system is almost unaware of GCHP (and GCHP builds the code in GC-Classic similar to how any other project would build GEOS-Chem).

The GC-Classic overhaul ended up making the GCHP build a lot easier too (I only added 5 files to GCHP besides the cmake/ directory which is copied from ESMA, and ESMA's CMakeLists are almost stock). And because of this, I decided to rewrite the GCHP build system yesterday afternoon (so that the commit history doesn't have my development history).

I'll update this PR when it's ready to be tried out!

Liam

[lizziel]

Thanks Liam. I am very excited to try this out! Just let me know when it's ready.

[LiamBindle]

Hi Lizzie. I think it's ready to go! I haven't validated the executable, but everything should compile fine.

Also, if anyone else would like to try it out, please do!

Where to start

The instructions for building GCHP with CMake are here. You'll have to have ESMF and gFTL built and installed somewhere on your machine (as a part of a different GCHP build is fine).

Notes

These are a few things that might be helpful to know ahead of time. They should be covered in the instructions, but I think it might be helpful to go in knowing these.

• Run cmake as many times as you want. Users should think of it as an interactive tool for configuring the build. Use the -D<var>=<value> option to set variables.
• You'll get "errors" saying ESMF and gFTL can't be found. These are expected, and it's just how CMake tells you it couldn't find a dependency. Unless you install ESMF and GFTL to a standard location (or set ESMF_ROOT and GFTL_ROOT environment variables), CMake won't find these on it's own.
  • You resolve these errors with the CMAKE_PREFIX_PATH variable. This is nice because you always use this variable regardless of the dependency (i.e. point CMake to missing dependencies is consistent regardless of the dependency)
  • So you'll run cmake -DCMAKE_PREFIX_PATH="<path to install>" . a few times pointing CMake to different places
• It looks like ESMF doesn't install it's .inc files, so you'll have to point CMake to ESMF/src/include (again using the CMAKE_PREFIX_PATH variable)
• I wrote a tutorial that introduces the cmake command to users which is here. It cover's things like what the cmake command actually does, how to set variables, how to set the build type to Debug or Release, and how to resolve missing dependencies (using the CMAKE_PREFIX_PATH variable). It's isn't necessary, but it might be helpful to anyone that's brand new to CMake
• Compiler errors are not expected. In fact, you shouldn't even be able to get compiler errors if you try (because cmake should identify the problem and it won't generate build files if there's a problem).
• If you get any warning or policy messages, please let me know.
• Lastly, please stress test it! Let me know what you can break!

Feedback

I'd really appreciate any feedback/ideas that you have about how it could be improved (anything and everything)! If anything doesn't work, or if anything is confusing, please let me know! 😄

Also, if anyone wants help, let me know when you plan to try it out and I'll make sure I'm on Slack!

Note: I'll be away on June 7-11.

--Liam

[lizziel]

Hi Liam,
Here are my comments which we can go over when you get back.

1. The instructions assume ESMF is downloaded and built outside of GCHP, but ESMF will still be part of GCHP in 12.5. I got around this by copying in a version of ESMF I had already compiled using GNU Make in GCHP from elsewhere. Is the idea that people should use GNU to build ESMF?

2. I think the step of making the run directory from the source code is missing.

3. As you said, I had to do cmake -DCMAKE_PREFIX_PATH="" for ESMF, ESMF/src/include, and gFTL. It took me a bit to figure it out since I had to use your notes above as well as the tutorial instructions. Best put all the information in one place in the instructions.

4. My cmake success message is not the same as yours. My GEOS-Chem and GCHP version strings are different, and I get warnings that nf-config is not found. I am using netcdf 4.5.0 and nf-config is not in this version (see this issue on unidata/netcdf-fortran github). Full message below:

/local/elundgren/rundirs/gchp_standard/build $ cmake -DCMAKE_PREFIX_PATH="/local/elundgren/gFTL" .
CMake Warning at CMakeScripts/FindNetCDF.cmake:30 (message):
Couldn't find nf-config. Is it in your PATH?
Call Stack (most recent call first):
CMakeLists.txt:30 (find_package)
-- GEOS-Chem version: 12.0.0-591-gd597567e
-- Useful CMake variables:

• CMAKE_PREFIX_PATH: /local/elundgren/gFTL

• CMAKE_BUILD_TYPE: Release Debug
  -- Run directory setup:

• RUNDIR: ..
  -- Bootstrapping /local/elundgren/rundirs/gchp_standard/build/..
  -- GCHP version: 12.0.3-288-g08d12227
  CMake Warning at CMakeScripts/FindNetCDF.cmake:30 (message):
  Couldn't find nf-config. Is it in your PATH?
  Call Stack (most recent call first):
  GCHP/CMakeLists.txt:23 (find_package)
  CMake Warning at CMakeScripts/FindNetCDF.cmake:30 (message):
  Couldn't find nf-config. Is it in your PATH?
  Call Stack (most recent call first):
  GCHP/cmake/FindESMF.cmake:86 (find_package)
  GCHP/CMakeLists.txt:24 (find_package)
  -- Found GFTL: /local/elundgren/gFTL/include
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /local/elundgren/rundirs/gchp_standard/build

5. I went ahead and compiled anyway to see what would happen. I got a bunch of these errors:

gfortran: error: unrecognized command line option ‘-Wno-’; did you mean ‘-gno-z’?
make[3]: *** [GCHP/Shared/GFDL_fms_r4/CMakeFiles/GFDL_fms_r4.dir/shared/mpp/mpp_parameter.F90.o] Error 1

But it did not actually stop until here:
[ 16%] Building C object GCHP/Shared/GMAO_pFIO/CMakeFiles/GMAO_pFIO.dir/pFIO_ShaveMantissa.c.o
[ 16%] Linking Fortran static library libGMAO_pFIO.a
[ 16%] Built target GMAO_pFIO
make: *** [all] Error 2

I am not sure if having nf-config would have affected this.

---

That is it for comments on the tutorial. But I have several other general questions.

1. I would still like to use a GNU Makefile in the run directory to compile. It is used as a utility file that is not related to actual makefiles in the source code. It allows flexibility for compiling sub-components of GEOS-Chem, cleaning sourcing code, and cleaning up the run directory, among other things. It is also a huge time-saver and avoids error since many commands are bundled into one. Will you incorporate compilation with cmake into the existing GCHP run directory Makefile?

2. How does cleaning source code fit into compiling with cmake?

3. Will there be tiered options for compilation such as there currently is?

Thanks so much for your work on this!!!
Lizzie

[LiamBindle]

Hi Lizzie,

Thanks for trying it out and for your comments!

Oh no! That was my bad. As of last Thursday, I still hadn't tried the GCHP build with GNU compilers. Sorry, I completely blanked on communicating that.

A bug in my get_warning_suppression_flags function caused your build to fail. This should be fixed in c509876.

Yesterday morning I tried the build with GNU compilers and it looks like it's working now. I had to make two small fixes (for some missing -ffree-line-length-none flags), one to my GEOS-Chem fork and one to my GCHP fork.

Easy things

2. I think the step of making the run directory from the source code is missing.

I can add these to the final set of instructions. For demoing the CMake builds, I'm just assuming a run directory was already created (using createRunDir.sh).

3. As you said, I had to do cmake -DCMAKE_PREFIX_PATH="" for ESMF, ESMF/src/include, and gFTL. It took me a bit to figure it out since I had to use your notes above as well as the tutorial instructions. Best put all the information in one place in the instructions.

Sounds good, I'll try to clarify step 4 of my instructions.

PS: See my "Improving usability" below. If you set ESMF_ROOT and GFTL_ROOT environment variables you won't get these errors when you initialize your build directory (it'll work out of the box). I think I'll make this the recommended approach rather than getting users to use CMAKE_PREFIX_PATH to resolve these errors.

4. My cmake success message is not the same as yours. My GEOS-Chem and GCHP version strings are different, and I get warnings that nf-config is not found. I am using netcdf 4.5.0 and nf-config is not in this version (see this issue on unidata/netcdf-fortran github). Full message below:

This is okay. I think the language I used here is confusing. I'll change "version" to "repo description". These are the output of git describe so when the repo's are ahead of a tagged version, a more verbose description is printed (also, some of the tags might be missing on my branch so you might get something like 12.0.0-593-gXXXXXXX which means 593 commits ahead of 12.0.0 at commit XXXXXX).

ESMF

1. The instructions assume ESMF is downloaded and built outside of GCHP, but ESMF will still be part of GCHP in 12.5. I got around this by copying in a version of ESMF I had already compiled using GNU Make in GCHP from elsewhere. Is the idea that people should use GNU to build ESMF?

I thought that ESMF was going to be made an external dependency. Is that happening in a later version? Right now my CMake scripts are expecting ESMF to be built and installed in a separate step, using ESMF's instructions. I was expecting users to do this locally or to get their sysadmin to install ESMF system-wide, and then point CMake to their install with ESMF_ROOT or CMAKE_PREFIX_PATH.

CMake doesn't need much from ESMF. In fact, the only reference to ESMF in our CMakeLists is a call to find_package(ESMF REQUIRED) which calls the FindESMF.cmake module I wrote. This FindESMF.cmake module sets up the compiler and linker flags for including and linking against ESMF. The way this module works is it searches for specific files that are unique to ESMF's install. Specifically, it looks for ESMF_ErrReturnCodes.inc , ESMC.h, esmf.mod, andlibesmf.a . These files are used as "hooks" for the paths are used in compiler and link flags.

Note: In the v8.0.0 snapshot, ESMF's .inc files aren't installed by make install. But, some of these .inc files are needed to build GCHP. This means ESMF's src/include directory needs to be included for the time being.

So, because of this, it doesn't really matter how ESMF is built. It's easiest and safest if ESMF is an external dependency (then we just call FindESMF.cmake), but if ESMF needs to remain in GCHP/, then I could write a wrapper with ExternalProject that builds and installs ESMF to your build directory. The downsides of using ExternalProject are:

• We are responsible for the ESMF build
• Wrappers are kind of evil. IMO they make it hard for end-users to troubleshoot and resolve errors
• The value for ESMF_COMM is difficult to determine automatically. I'd probably have to add a variable and get users to set which one they're using.
• Errors will show up during the compile step rather than the configure step

---

General things

2. How does cleaning source code fit into compiling with cmake?
3. Will there be tiered options for compilation such as there currently is?

Note: The following is CMake's default behavior (i.e. these aren't decision I've made)

The Makefiles that CMake generates have all, install, and clean targets, as well as targets for every component (e.g. HCOI , MAPL_Base, GIGC, etc). To compile a specific component like HCOI you'd do make HCOI.

Cleaning is somewhat different though. Before explaining the difference, I should mention that cmake generates a dependency tree of source files and components. When you run a command like make HCOI, which files need to be compiled/recompile is determined automatically at using this dependency tree. Initially, none of HCOI's dependencies exist so all of HCOI's dependencies will be compiled. Then, if you were to modify a source file that HCOI depends on (e.g a file in Headers/) and rerun make HCOI , the Makefiles would automatically figure out what all needs to be recompiled. The same goes for changing compiler options and definitions (e.g. if you were to build GC-Classic with OpenMP on and then turn it off which set's the NO_OMP definition).

This means that after modifying source code there's no need to clean before recompiling. The clean target cleans everything (including everything in Shared/) and is really only meant for exceptional circumstances.

1. I would still like to use a GNU Makefile in the run directory to compile. It is used as a utility file that is not related to actual makefiles in the source code. It allows flexibility for compiling sub-components of GEOS-Chem, cleaning sourcing code, and cleaning up the run directory, among other things. It is also a huge time-saver and avoids error since many commands are bundled into one. Will you incorporate compilation with cmake into the existing GCHP run directory Makefile?

I'll defer to your judgement on this because you have a better understanding of users' workflow than I do.

I do, however, have concerns about this. IMO a benefit of using CMake to build GEOS-Chem is that CMake has a somewhat standard way of building software. This means that building GEOS-Chem with CMake is similar to building any other software project with CMake. This makes troubleshooting easier (errors aren't unique to GEOS-Chem so they're easier to google), improves familiarity for new users (or familiarity for GEOS-Chem users building other software; also, someone that's familiar with CMake probably doesn't even need to read the build instructions), and ultimately it should break GEOS-Chem's build procedure into transparent steps.

I know I haven't quite got things right from the usability standpoint yet, but I think that with a little more playing around I can get it right.

I think some of the things that made the run directory's Makefile so useful are included in CMake's default behavior (recompiling and tiered build options with the dependency solving). For things that aren't related to building GEOS-Chem, though, like cleaning up the run directory, I totally get that a utility script is helpful for that.

Improving usability

I think that I could make things easier by getting users to set environment variables along side CC, CXX, and FC that specify where dependencies are installed.

If you set the ESMF_ROOT and GFTL_ROOT environment variables, my FindESMF.cmake and FindGFTL.cmake modules will automatically use them and there is no need to play around with CMAKE_PREFIX_PATH. In short the instructions would be:

1. Set CC, CXX, FC, ESMF_ROOT and GFTL_ROOT
   • ESMF_ROOT needs to be a semicolon separated list of the install prefix and src/include until the dependency on ESMF's internal .inc files are removed or ESMF includes the .inc files in their install
2. Go to your run directory
3. Run cmake <path to source code>
   • This will work out of the box if ESMF_ROOT and GFTL_ROOT are set
4. Run make
5. Run make install

Previously, I said that the CMake build system didn't require any environment setup, so I've shied away from using these. But, I think that this would make the build system easier to use because it would just work out of the box.

Thanks again for your help! Let me know what you think.

Liam

[LiamBindle]

I just updated the GCHP build instructions (here). Now when you initialize your build directory you shouldn't get any errors (because the ESMF and GFTL install locations are specified with the ESMF_ROOT and GFTL_ROOT environment variables).

[lizziel]

Hi Liam,
Thanks for your very detailed response. For now I am only going to address questions about separating ESMF as an external library. GCHP 12.5 will still include ESMF in its current form as a subdirectory in the repository. This is because big changes will happen to the repo very soon, with GMAO releasing MAPL and other components (largely restructured compared to what we use now), on a public GitHub in July and ESMF publicly releasing v8 in September. Removing ESMF as an external library will be part of a larger change of making a lightweight GCHP repo with advection, MAPL, and other needed GMAO components included as git submodules. Doing this restructuring will begin as soon as the public github repos are available, in a few weeks.

I wonder if it makes the most sense to include CMake capability for GCHP in the version that includes the restructuring rather than 12.5? Development of it will start very soon so it seems logical to direct work towards that version rather than one that is both more difficult to work with and very short-lived.

[LiamBindle]

Hi Lizzie,

Okay, I see, that makes sense. I thought that ESMF removal was more immediate.

I wonder if it makes the most sense to include CMake capability for GCHP in the version that includes the restructuring rather than 12.5? Development of it will start very soon so it seems logical to direct work towards that version rather than one that is both more difficult to work with and very short-lived.

I agree. The payoff for getting CMake into 12.5 rather than the version with restructuring is probably pretty slim, and there's a lot more potential for problems if ESMF is built as an ExternalProject.

For the time being, it's fine to use an ESMF build from another instance of GCHP as your ESMF installation (i.e. something like export ESMF_ROOT="$ESMF_DIR/DEFAULTINSTALLDIR;$ESMF_DIR").

Thanks,
Liam