Compilation flags for CMAQ using gcc 10.2.0 and openMPI

Hello,

I am trying to compile CCTM from CMAQ v5.3.2.1. For this version, I want to read emissions compressed in netCDF-4 (.nc4) format. I am using precompiled libraries already available on my HPC system. These are: gcc 10.2.0, openmpi 4.1.3, zlib 1.2.11, hdf5 1.10.7, netcdf-c 4.8.1 and netcdf-fortran 4.5.3. All of them where built with the same gcc compiler. With them, I have been able to compile I/O API 3.2 (special version 20200828) with sucess - or I least I believe so since after adding some flags, such as -DIOAPI_NCF4=1 and -fallow-argument-mismatch, I was able to compile without errors.

The problem now appears when compiling CCTM. Sadly, as a new user I am not allowed to upload the log file. I have properly defined the directories to my openMPI, however, the error shows that the mpi.h file does not exist:

distr_env.c:36:10: fatal error: mpi.h: No such file or directory
   36 | #include "mpi.h"
      |          ^~~~~~~
compilation terminated.

On my CMAQ_HOME/lib/gcc/mpi the file mpi.h indeed does not exist. Checking the config_cmaq.csh I see that the code links the openmpi/lib to my CMAQ/lib/gcc/mpi. However, after inspecting I see that the mpi.h file exist on the openmpi directory, but under openmpi/include rather than openmpi/lib. Should I manually change this?

I dug into several threads on this forum and I could not find someone with a similar error. How can I solve this? Also, I am a bit confused since I have seen some threads recommending to use the “-fopenmp” flag but at the same time, the config_cmaq.csh file expresses that it should not be used (?):

setenv myLINK_FLAG # "-fopenmp" openMP not supported w/ CMAQ

Any feedback would be much appreciated.
Thanks.

===================================
build_cctm.log: (I removed all the intro to be able to post this topic).

Compiler is set to gcc

*****(many lines here were removed)*****
  
  USE/MODULE dependencies defined

Makefile generated
mpifort -c -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 -fallow-argument-mismatch -fallow-argument-mismatch -O3 -funroll-loops -finit-character=32 -Wtabs -Wsurprising -march=native -ftree-vectorize  -ftree-loop-if-convert -finline-limit=512 -I /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/ioapi/lib -I /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/ioapi/include_files -I /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/mpi -I. -fallow-argument-mismatch -Dparallel -Dstage_opt -DSUBST_BARRIER=SE_BARRIER -DSUBST_GLOBAL_MAX=SE_GLOBAL_MAX -DSUBST_GLOBAL_MIN=SE_GLOBAL_MIN -DSUBST_GLOBAL_MIN_DATA=SE_GLOBAL_MIN_DATA -DSUBST_GLOBAL_TO_LOCAL_COORD=SE_GLOBAL_TO_LOCAL_COORD -DSUBST_GLOBAL_SUM=SE_GLOBAL_SUM -DSUBST_GLOBAL_LOGICAL=SE_GLOBAL_LOGICAL -DSUBST_LOOP_INDEX=SE_LOOP_INDEX -DSUBST_SUBGRID_INDEX=SE_SUBGRID_INDEX -DSUBST_HI_LO_BND_PE=SE_HI_LO_BND_PE -DSUBST_SUM_CHK=SE_SUM_CHK -DSUBST_SE_INIT=SE_INIT -DSUBST_INIT_ARRAY=SE_INIT_ARRAY -DSUBST_COMM=SE_COMM -DSUBST_MY_REGION=SE_MY_REGION -DSUBST_SLICE=SE_SLICE -DSUBST_GATHER=SE_GATHER -DSUBST_DATA_COPY=SE_DATA_COPY -DSUBST_IN_SYN=SE_IN_SYN -DSUBST_PE_COMM=\"./PE_COMM.EXT\" -DSUBST_CONST=\"./CONST.EXT\" -DSUBST_FILES_ID=\"./FILES_CTM.EXT\" -DSUBST_EMISPRM=\"./EMISPRM.EXT\" -DSUBST_MPI=\"/lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/mpi/include/mpif.h\" BEIS_DEFN.F
mpifort -c -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 -fallow-argument-mismatch -fallow-argument-mismatch -O3 -funroll-loops -finit-character=32 -Wtabs -Wsurprising -march=native -ftree-vectorize  -ftree-loop-if-convert -finline-limit=512 -I /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/ioapi/lib -I /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/ioapi/include_files -I /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/mpi -I. -fallow-argument-mismatch -Dparallel -Dstage_opt -DSUBST_BARRIER=SE_BARRIER -DSUBST_GLOBAL_MAX=SE_GLOBAL_MAX -DSUBST_GLOBAL_MIN=SE_GLOBAL_MIN -DSUBST_GLOBAL_MIN_DATA=SE_GLOBAL_MIN_DATA -DSUBST_GLOBAL_TO_LOCAL_COORD=SE_GLOBAL_TO_LOCAL_COORD -DSUBST_GLOBAL_SUM=SE_GLOBAL_SUM -DSUBST_GLOBAL_LOGICAL=SE_GLOBAL_LOGICAL -DSUBST_LOOP_INDEX=SE_LOOP_INDEX -DSUBST_SUBGRID_INDEX=SE_SUBGRID_INDEX -DSUBST_HI_LO_BND_PE=SE_HI_LO_BND_PE -DSUBST_SUM_CHK=SE_SUM_CHK -DSUBST_SE_INIT=SE_INIT -DSUBST_INIT_ARRAY=SE_INIT_ARRAY -DSUBST_COMM=SE_COMM -DSUBST_MY_REGION=SE_MY_REGION -DSUBST_SLICE=SE_SLICE -DSUBST_GATHER=SE_GATHER -DSUBST_DATA_COPY=SE_DATA_COPY -DSUBST_IN_SYN=SE_IN_SYN -DSUBST_PE_COMM=\"./PE_COMM.EXT\" -DSUBST_CONST=\"./CONST.EXT\" -DSUBST_FILES_ID=\"./FILES_CTM.EXT\" -DSUBST_EMISPRM=\"./EMISPRM.EXT\" -DSUBST_MPI=\"/lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/mpi/include/mpif.h\" centralized_io_module.F
mpifort -c -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 -fallow-argument-mismatch -fallow-argument-mismatch -O3 -funroll-loops -finit-character=32 -Wtabs -Wsurprising -march=native -ftree-vectorize  -ftree-loop-if-convert -finline-limit=512 -I /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/ioapi/lib -I /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/ioapi/include_files -I /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/mpi -I. -fallow-argument-mismatch -Dparallel -Dstage_opt -DSUBST_BARRIER=SE_BARRIER -DSUBST_GLOBAL_MAX=SE_GLOBAL_MAX -DSUBST_GLOBAL_MIN=SE_GLOBAL_MIN -DSUBST_GLOBAL_MIN_DATA=SE_GLOBAL_MIN_DATA -DSUBST_GLOBAL_TO_LOCAL_COORD=SE_GLOBAL_TO_LOCAL_COORD -DSUBST_GLOBAL_SUM=SE_GLOBAL_SUM -DSUBST_GLOBAL_LOGICAL=SE_GLOBAL_LOGICAL -DSUBST_LOOP_INDEX=SE_LOOP_INDEX -DSUBST_SUBGRID_INDEX=SE_SUBGRID_INDEX -DSUBST_HI_LO_BND_PE=SE_HI_LO_BND_PE -DSUBST_SUM_CHK=SE_SUM_CHK -DSUBST_SE_INIT=SE_INIT -DSUBST_INIT_ARRAY=SE_INIT_ARRAY -DSUBST_COMM=SE_COMM -DSUBST_MY_REGION=SE_MY_REGION -DSUBST_SLICE=SE_SLICE -DSUBST_GATHER=SE_GATHER -DSUBST_DATA_COPY=SE_DATA_COPY -DSUBST_IN_SYN=SE_IN_SYN -DSUBST_PE_COMM=\"./PE_COMM.EXT\" -DSUBST_CONST=\"./CONST.EXT\" -DSUBST_FILES_ID=\"./FILES_CTM.EXT\" -DSUBST_EMISPRM=\"./EMISPRM.EXT\" -DSUBST_MPI=\"/lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/mpi/include/mpif.h\" PHOT_MOD.F
gcc -c -O2 -fallow-argument-mismatch -DFLDMN -I/lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/mpi/include -I. distr_env.c
mpifort -c -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 -fallow-argument-mismatch -fallow-argument-mismatch -O3 -funroll-loops -finit-character=32 -Wtabs -Wsurprising -march=native -ftree-vectorize  -ftree-loop-if-convert -finline-limit=512 -I /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/ioapi/lib -I /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/ioapi/include_files -I /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/mpi -I. -fallow-argument-mismatch -Dparallel -Dstage_opt -DSUBST_BARRIER=SE_BARRIER -DSUBST_GLOBAL_MAX=SE_GLOBAL_MAX -DSUBST_GLOBAL_MIN=SE_GLOBAL_MIN -DSUBST_GLOBAL_MIN_DATA=SE_GLOBAL_MIN_DATA -DSUBST_GLOBAL_TO_LOCAL_COORD=SE_GLOBAL_TO_LOCAL_COORD -DSUBST_GLOBAL_SUM=SE_GLOBAL_SUM -DSUBST_GLOBAL_LOGICAL=SE_GLOBAL_LOGICAL -DSUBST_LOOP_INDEX=SE_LOOP_INDEX -DSUBST_SUBGRID_INDEX=SE_SUBGRID_INDEX -DSUBST_HI_LO_BND_PE=SE_HI_LO_BND_PE -DSUBST_SUM_CHK=SE_SUM_CHK -DSUBST_SE_INIT=SE_INIT -DSUBST_INIT_ARRAY=SE_INIT_ARRAY -DSUBST_COMM=SE_COMM -DSUBST_MY_REGION=SE_MY_REGION -DSUBST_SLICE=SE_SLICE -DSUBST_GATHER=SE_GATHER -DSUBST_DATA_COPY=SE_DATA_COPY -DSUBST_IN_SYN=SE_IN_SYN -DSUBST_PE_COMM=\"./PE_COMM.EXT\" -DSUBST_CONST=\"./CONST.EXT\" -DSUBST_FILES_ID=\"./FILES_CTM.EXT\" -DSUBST_EMISPRM=\"./EMISPRM.EXT\" -DSUBST_MPI=\"/lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/lib/x86_64/gcc/mpi/include/mpif.h\" hrsolver.F
cc1: warning: command-line option ‘-fallow-argument-mismatch’ is valid for Fortran but not for C
distr_env.c:36:10: fatal error: mpi.h: No such file or directory
   36 | #include "mpi.h"
      |          ^~~~~~~
compilation terminated.
make: *** [Makefile:462: distr_env.o] Error 1
make: *** Waiting for unfinished jobs....
centralized_io_module.F:1882:0:

 1882 | #endif   ! end if stage option
      | 
Warning: extra tokens at end of #endif directive
centralized_io_module.F:1883:0:

 1883 | #endif   ! end if m3dry option
      | 
Warning: extra tokens at end of #endif directive
centralized_io_module.F:3784:51:

 3784 |              WRITE( LOGDEV, "(5x,A,/,5x,A,/,5x,A)"),
      |                                                   1
Warning: Legacy Extension: Comma before i/o item list at (1)
centralized_io_module.F:3803:37:

 3803 |              WRITE( XMSG, '(A,A,A)' ),'ERROR: Could not read ',
      |                                     1
Warning: Legacy Extension: Comma before i/o item list at (1)
centralized_io_module.F:3812:59:

 3812 |              WRITE( LOGDEV, "(5x,A,/,5x,A,/,5x,A,/,5x,A)" ),
      |                                                           1
Warning: Legacy Extension: Comma before i/o item list at (1)
centralized_io_module.F:3828:59:

 3828 |              WRITE( LOGDEV, "(5x,A,/,5x,A,/,5x,A,/,5x,A)" ),
      |                                                           1
Warning: Legacy Extension: Comma before i/o item list at (1)
centralized_io_module.F:3850:59:

 3850 |              WRITE( LOGDEV, "(5x,A,/,5x,A,/,5x,A,/,5x,A)" ),
      |                                                           1
Warning: Legacy Extension: Comma before i/o item list at (1)
centralized_io_module.F:3872:59:

 3872 |              WRITE( LOGDEV, "(5x,A,/,5x,A,/,5x,A,/,5x,A)" ),
      |                                                           1
Warning: Legacy Extension: Comma before i/o item list at (1)
centralized_io_module.F:3887:59:

 3887 |              WRITE( LOGDEV, "(5x,A,/,5x,A,/,5x,A,/,5x,A)" ),
      |                                                           1
Warning: Legacy Extension: Comma before i/o item list at (1)
centralized_io_module.F:3901:59:

 3901 |              WRITE( LOGDEV, "(5x,A,/,5x,A,/,5x,A,/,5x,A)" ),
      |                                                           1
Warning: Legacy Extension: Comma before i/o item list at (1)
centralized_io_module.F:3915:59:

 3915 |              WRITE( LOGDEV, "(5x,A,/,5x,A,/,5x,A,/,5x,A)" ),
      |                                                           1
Warning: Legacy Extension: Comma before i/o item list at (1)
centralized_io_module.F:2361:53:

 2361 |              WRITE( LOGDEV, '(/,5x,A,/,5x,A,/5x,A)' ),
      |                                                     1
Warning: Legacy Extension: Comma before i/o item list at (1)
**ERROR** while running make command

mv Makefile Makefile.gcc
mv: 'Makefile' and 'Makefile.gcc' are the same file
if ( -e Makefile.gcc && -e Makefile ) rm Makefile
rm Makefile
ln -s Makefile.gcc Makefile
if ( 0 != 0 ) then
if ( -e /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/CCTM/scripts/BLD_CCTM_v5321_gcc_STAGE/CCTM_v5321.cfg ) then
echo    >>> previous CCTM_v5321.cfg exists, re-naming to CCTM_v5321.cfg.old <<<
   >>> previous CCTM_v5321.cfg exists, re-naming to CCTM_v5321.cfg.old <<<
mv /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/CCTM/scripts/BLD_CCTM_v5321_gcc_STAGE/CCTM_v5321.cfg /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/CCTM/scripts/BLD_CCTM_v5321_gcc_STAGE/CCTM_v5321.cfg.old
endif
mv CCTM_v5321.cfg.bld /lustre/isaac/proj/UTK0221/NIH/CONUS/CMAQ/5.3.2.1-gcc/CCTM/scripts/BLD_CCTM_v5321_gcc_STAGE/CCTM_v5321.cfg
exit

===================================
My config_cmaq.csh: (I kept only the settings relevant to gcc)

#!/bin/csh -f

# ================= CMAQv5.3.X Configuration Script ================= #
# Requirements: I/O API & netCDF libraries                            #
#               PGI, Intel, or Gnu Fortran compiler                   #
#               MPICH for multiprocessor computing                    #
# Optional:     Git for GitHub source code repository                 #
#                                                                     #
# Note that this script was configured/tested on Red Hat Linux O/S    #
#                                                                     #
# To report problems or request help with this script/program:        #
#             http://www.cmascenter.org/help-desk.cfm                 #
# =================================================================== #

#> Critical Folder Locations
 # CMAQ_HOME - this is where the config_cmaq.csh script is located. It
 # is also the root directory for all the executables. It may include 
 # the repository if the user is building CMAQ inside the repository. It
 # may, on the other hand, be outside the repository if the user has 
 # created a separate project directory where they wish to put build-
 # and run-scripts as well as executables.
 setenv CMAQ_HOME $cwd

 # CMAQ_REPO - this is always the location of the CMAQ repository that
 # the user will pull from to create exectuables. If the user is building
 # CMAQ inside the repository then it will be equal to CMAQ_HOME. If not,
 # the user must supply an alternative folder locaiton.
 setenv CMAQ_REPO /lustre/isaac/proj/UTK0221/andrew/CMAQ/5.3.2.1

 # CMAQ_DATA - this may be where the input data are located. It may be a 
 # symbolic link to another location on the system, but it should be
 # provided here 
 setenv CMAQ_DATA $CMAQ_HOME/data
 if ( ! -d $CMAQ_DATA ) mkdir -p $CMAQ_DATA

 cd $CMAQ_HOME

#===============================================================================
#> architecture & compiler specific settings
#===============================================================================

#> Set the compiler option
 if ( $#argv == 1 ) then
    #> Use the user's input to set the compiler parameter
    setenv compiler $1
    setenv compilerVrsn Empty
 else if ( $#argv == 2 ) then
    #> Compiler Name and Version have been provided
    setenv compiler $1
    setenv compilerVrsn $2
 else if ( $#argv == 0 ) then
    #> If config.cmaq is called from Bldit.cctm or run.cctm, then this 
    #> variable is already defined
    if ( ! $?compiler ) then
      echo "Error: 'compiler' should be set either in the"
      echo "       environment or as input to config.cmaq"
      echo "       Example:> ./config.cmaq [compiler]"
      echo "       Options: intel | gcc | pgi"
      exit
    else if ( ! $?compilerVrsn ) then
      setenv compilerVrsn Empty
    endif
 else
    #> More than two inputs were given. Exit this script just to
    #> be on the safe side.
    echo "Error: Too many inputs to config.cmaq. This script"
    echo "       is expecting one input (the name of the"
    echo "       desired compiler. In some installations, you "
    echo "       may also be able to specify the compiler version "
    echo "       as the second input, but this is not by default."
    exit
 endif
 echo "Compiler is set to $compiler"
 
 # GAF:
setenv NCDIR /spack/spack-0.17.2/apps/linux-rhel8-cascadelake/gcc-10.2.0/netcdf-c-4.8.1-7dwpqiw3lhygretuczntsfnug27ark67
setenv NFDIR /spack/spack-0.17.2/apps/linux-rhel8-cascadelake/gcc-10.2.0/netcdf-fortran-4.5.3-slv6qm3skovi7vcwvnveuvliajniqovh
setenv IOAPI_BLD /lustre/isaac/proj/UTK0221/andrew/LIBRARIES/ioapi-3.2/Linux2_x86_64gfort_openmpi_4.1.3_gcc_10.2.0
setenv IOAPI_SRC /lustre/isaac/proj/UTK0221/andrew/LIBRARIES/ioapi-3.2
setenv OPENMPIDIR /sw/isaac/applications/openmpi/4.1.3/rhel8.7-gcc10.2.0/openmpi-4.1.3


#> Compiler flags and settings
 switch ( $compiler )
    
#>  gfortran compiler............................................................
    case gcc:
  
        #> I/O API, netCDF, and MPI library locations
        setenv IOAPI_INCL_DIR   ${IOAPI_SRC}/ioapi/fixed_src   #> I/O API include header files
        setenv IOAPI_LIB_DIR    ${IOAPI_BLD}   #> I/O API libraries
        
        setenv NETCDF_LIB_DIR   ${NCDIR}/lib            #> netCDF C directory path
        setenv NETCDF_INCL_DIR  ${NCDIR}/include        #> netCDF C directory path
        setenv NETCDFF_LIB_DIR  ${NFDIR}/lib            #> netCDF Fortran directory path
        setenv NETCDFF_INCL_DIR ${NFDIR}/include        #> netCDF Fortran directory path
        
        setenv MPI_LIB_DIR      ${OPENMPIDIR}/lib       #> MPI directory path
        setenv MPI_INCL_DIR     ${OPENMPIDIR}/include
    
        #> Compiler Aliases and Flags
        #> set the compiler flag -fopt-info-missed to generate a missed optimization report in the bldit logfile
        setenv myFC mpifort
        setenv myCC gcc
        setenv myFSTD "-O3 -funroll-loops -finit-character=32 -Wtabs -Wsurprising -march=native -ftree-vectorize  -ftree-loop-if-convert -finline-limit=512"
        setenv myDBG  "-Wall -O0 -g -fcheck=all -ffpe-trap=invalid,zero,overflow -fbacktrace"
        setenv myFFLAGS "-ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 -fallow-argument-mismatch"
        setenv myFRFLAGS "-ffree-form -ffree-line-length-none -funroll-loops -finit-character=32"
        setenv myCFLAGS "-O2 -fallow-argument-mismatch"
        setenv myLINK_FLAG # "-fopenmp" openMP not supported w/ CMAQ
        setenv extra_lib ""
        #setenv mpi_lib "-lmpi_mpifh"   #> -lmpich for mvapich or -lmpi for openmpi
        setenv mpi_lib ""   #> -lmpich for mvapich or -lmpi for openmpi
    
        breaksw

    default:
        echo "*** Compiler $compiler not found"
        exit(2)
        breaksw

 endsw
 
#> Apply Specific Module and Library Location Settings for those working inside EPA
 # source /work/MOD3DEV/cmaq_common/cmaq_env.csh  #>>> UNCOMMENT if at EPA

#> Add The Complier Version Number to the Compiler String if it's not empty
 setenv compilerString ${compiler}
 if ( $compilerVrsn != "Empty" ) then
    setenv compilerString ${compiler}${compilerVrsn}
 endif

#===============================================================================
 
#> I/O API, netCDF, and MPI libraries
 setenv netcdf_lib "-lnetcdf"  #> -lnetcdff -lnetcdf for netCDF v4.2.0 and later
 setenv netcdff_lib "-lnetcdff"
 setenv ioapi_lib "-lioapi"
 setenv pnetcdf_lib "-lpnetcdf"

#> Query System Info and Current Working Directory
 setenv system "`uname -m`"
 setenv bld_os "`uname -s``uname -r | cut -d. -f1`"
 setenv lib_basedir $CMAQ_HOME/lib

#> Generate Library Locations
 setenv CMAQ_LIB    ${lib_basedir}/${system}/${compilerString}
 setenv MPI_DIR     $CMAQ_LIB/mpi
 setenv NETCDF_DIR  $CMAQ_LIB/netcdf
 setenv NETCDFF_DIR $CMAQ_LIB/netcdff
 setenv PNETCDF_DIR $CMAQ_LIB/pnetcdf
 setenv IOAPI_DIR   $CMAQ_LIB/ioapi

#> Create Symbolic Links to Libraries
 if ( ! -d $CMAQ_LIB ) mkdir -p $CMAQ_LIB
 if (   -e $MPI_DIR  ) rm -rf $MPI_DIR
     # ln -s $MPI_LIB_DIR $MPI_DIR
     ln -s $MPI_INCL_DIR $MPI_DIR
 if ( ! -d $NETCDF_DIR )  mkdir $NETCDF_DIR
 if ( ! -e $NETCDF_DIR/lib ) ln -sfn $NETCDF_LIB_DIR $NETCDF_DIR/lib
 if ( ! -e $NETCDF_DIR/include ) ln -sfn $NETCDF_INCL_DIR $NETCDF_DIR/include
 if ( ! -d $NETCDFF_DIR )  mkdir $NETCDFF_DIR
 if ( ! -e $NETCDFF_DIR/lib ) ln -sfn $NETCDFF_LIB_DIR $NETCDFF_DIR/lib
 if ( ! -e $NETCDFF_DIR/include ) ln -sfn $NETCDFF_INCL_DIR $NETCDFF_DIR/include
 if ( ! -d $IOAPI_DIR ) then
    mkdir $IOAPI_DIR
    ln -sfn $IOAPI_INCL_DIR $IOAPI_DIR/include_files
    ln -sfn $IOAPI_LIB_DIR  $IOAPI_DIR/lib
 endif

#> Check for netcdf and I/O API libs/includes, error if they don't exist
 if ( ! -e $NETCDF_DIR/lib/libnetcdf.a ) then 
    echo "ERROR: $NETCDF_DIR/lib/libnetcdf.a does not exist in your CMAQ_LIB directory!!! Check your installation before proceeding with CMAQ build."
    exit
 endif
if ( ! -e $NETCDFF_DIR/lib/libnetcdff.a ) then
    echo "ERROR: $NETCDFF_DIR/lib/libnetcdff.a does not exist in your CMAQ_LIB directory!!! Check your installation before proceeding with CMAQ build."
    exit
 endif
 if ( ! -e $IOAPI_DIR/lib/libioapi.a ) then 
    echo "ERROR: $IOAPI_DIR/lib/libioapi.a does not exist in your CMAQ_LIB directory!!! Check your installation before proceeding with CMAQ build."
    exit
 endif
 if ( ! -e $IOAPI_DIR/lib/m3utilio.mod ) then 
    echo "ERROR: $IOAPI_MOD_DIR/m3utilio.mod does not exist in your CMAQ_LIB directory!!! Check your installation before proceeding with CMAQ build."
    exit
 endif

#> Set executable id
 setenv EXEC_ID ${bld_os}_${system}${compilerString}

The standard UNIX/Linux conventions are that mpi.h should be in a something/include directory; it is an artifact of the way that CMAQ build system attempts to support multiple compilers and MPI systems that you need to sym-link it to (CMAQ’s notion of version-specific) CMAQ/lib/gcc/mpi. Your experience with -fallow-argument-mismatch is one that shows their methods are inadequate, and don’t allow for Fortran-compiler specifics the way I/O API binary-types such as Linux2_x86_64gfort10 and Linux2_x86pg_gcc_mp do.

The config_cmaq.csh file claim that -fopenmp not be be used because CMAQ does not use OpenMP is based upon a failure to understand that including OpenMP support in the compile-and-link does not cost anything: including this flag does not affect the performance of CMAQ in any way. Many other programs do use OpenMP and want this support; if the I/O API library is compiled to support OpenMP, then the CMAQ build-process needs this flag.

Thanks for the feedback. Although I am not completely sure what is the solution or workaround to this.

Let me clarify, for I/O API I used the Makeinclude.Linux2_x86_64gfort10 as base. I had to include the --fallow-argument-mismatch flag when building it since after a Google search, I found that adding it will remove the errors I was getting. The errors were the same as the ones shown in this other CMAS thread. However, the solution given there did not solve my problem. But adding this flag did. My guess that this issue was caused by the newer gcc version I am using.
Now, I made a test without using this flag when building CCTM and also linking the openmpi/include directory as CMAQ_HOME/lib/gcc/mpi. Therefore, the mpi.h file is present there now. However, I get the same error as before. The code complains that it cannot find that file. What should I do now?

Snapshot of the error:

distr_env.c:36:10: fatal error: mpi.h: No such file or directory
   36 | #include "mpi.h"
      |          ^~~~~~~
compilation terminated.
make: *** [Makefile:461: distr_env.o] Error 1
make: *** Waiting for unfinished jobs....

Note 1: I am not using the -fopenmp flag when building CCTM, since I also did not use it for I/O API. However, I am not sure whether I should specify something in other flags in my config_cmaq.csh, such as extra_lib, mpi_lib or myLINK_FLAG.

Note 2: I noticed that the Makefile created defines:

FC = mpifort
CC = gcc

Is this correct? Should not it be FC = gfortran or something similar?
Before running the bldit_cctm.csh script I have set setenv FC gfortran but then it is overwritten in the Makefile.

mpifort is actually a script (of sorts) that is built specifically for some C-compiler/Fortran-compiler set (e.g., gcc and gfortran); it “knows” from the MPI install-and-build-process how to invoke the Fortran compiler in such a way as to “find” its own include-files and library-files.

The way the CMAQ build-process overrides that internal “knowledge” about a particular MPI installation actually reflects a rather serious misunderstanding about the entire MPI-system characteristics ;-(