Compiling CMAQ on Cheyenne at NCAR

CMAQ Compiling
21

You are not using mpifort -show, you are using ifort -show.
In the config_cmaq.csh the myFC should be set to mpifort, rather than ifort.
That will allow the compiler to use and find the mpi library -lmpi

setenv myFC mpifort
22

If I used mpifort -show, it shows
NCAR_ERROR: wrapper cannot locate path to mpifort

Should I contact NCAR helpdesk for this wrapper issue?

23

Hi,

Are you on a system that uses a module environment?
Please check the output of the following command:
module avail openmpi

24

The output is here
----------------------- /glade/u/apps/ch/modulefiles/default/intel/19.0.5 ----------------
openmpi/3.1.4 openmpi/4.0.3 openmpi/4.0.5 (D)
Where:
D: Default Module

Now I need to confirm 2 problems with NCAR help desk:
(1) Did CMAQ accept mpt/2.22 compiled IOAPI3.2? If no, can we change to another mpi such as OpenMPI, MPICH, MVAPICH2, or the Intel MPI?
(2) The myFC in config_cmaq.csh should be changed to mpifort, but now Cheyenne has a wrapper error.

Thanks for your replying. @lizadams

25

I solved the problem.

For IOAPI3.2, I used Linux2_x86_64ifortmpi. Here are the steps:

  1. Modify /ioapi-3.2/ioapi/Makeinclude.Linux2_x86_64ifortmpi
    cd /…/ioapi-3.2/ioapi
    vi Makeinclude.Linux2_x86_64ifortmpi
    Line 23: OMPFLAGS = #-openmp
    Line 24: OMPLIBS = #-openmp
    Add a line after Line 33: -DIOAPI_NCF4=1
    Line 41: #ARCHLIB = -Bstatic
    :wq
    Here’ s my reference for Line 33: Availability/Download of the BAMS/Models-3 I/O API

  2. Modify /ioapi-3.2/Makefile
    cd …/
    vi Makefile
    Line 193: NCFLIBS = -L/glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/lib -lnetcdff -lnetcdf -qopenmp
    :wq

  3. Set environment
    export BIN=Linux2_x86_64ifortmpi
    export BASEDIR=$PWD
    export CPLMODE=nocpl
    mkdir $BIN

  4. compiling IOAPI
    make configure
    make

The environment settings in config_cmaq.sh is:
case intel:

    #> I/O API, netCDF, and MPI library locations
    setenv IOAPI_INCL_DIR   /glade/scratch/chengpin/ioapi-3.2/ioapi/fixed_src #ioapi_inc_intel    #> I/O API include header files
    setenv IOAPI_LIB_DIR    /glade/scratch/chengpin/ioapi-3.2/Linux2_x86_64ifortmpi #ioapi_lib_intel    #> I/O API libraries
    setenv NETCDF_LIB_DIR   /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/lib #netcdf_lib_intel   #> netCDF C directory path
    setenv NETCDF_INCL_DIR  /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/include #netcdf_inc_intel   #> netCDF C directory path
    setenv NETCDFF_LIB_DIR  /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/lib #netcdff_lib_intel  #> netCDF Fortran directory path
    setenv NETCDFF_INCL_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/include #netcdff_inc_intel  #> netCDF Fortran directory path
    setenv MPI_LIB_DIR      /glade/u/apps/opt/intel/2019u5/impi/2019.6.154/intel64  #mpi_lib_intel      #> MPI directory path

    #> Compiler Aliases and Flags
    #> set the compiler flag -qopt-report=5 to get a model optimization report in the build directory with the optrpt extension
    setenv myFC mpiifort
    setenv myCC mpicc
    setenv myFSTD "-O3 -fno-alias -mp1 -fp-model source -ftz -simd -align all -xHost -vec-guard-write -unroll-aggressive"
    setenv myDBG  "-O0 -g -check bounds -check uninit -fpe0 -fno-alias -ftrapuv -traceback"
    setenv myLINK_FLAG "" #"-qopenmp-simd" openMP not supported w/ CMAQ
    setenv myFFLAGS "-fixed -132"
    setenv myFRFLAGS "-free"
    setenv myCFLAGS "-O2"
    setenv extra_lib "-lcurl"
    #setenv extra_lib ""
    setenv mpi_lib ""    #> No Library specification needed for mpiifort
                         #> -lmpich for mvapich
                         #> -lmpi for openmpi

    breaksw

Hope it will help other users on NCAR Cheyenne.
Thank you, guys.