MPI undefined error when installing CCTM using gcc and gfortran

Hi everyone,
I successfully compiled MCIP BCON and ICON when configuring CMAQv5.3.3 using mpich4.0 ioapi-3.2-large. However, I encountered an error in the mpi library link during the installation of CCTM. I would like to know the reason for this. Below is the log file when I make CCTM
/usr/bin/ld: cmaq_main.o: in function MAIN__': cmaq_main.F:(.text+0x2ea): undefined reference to mpi_wtime_’
/usr/bin/ld: cmaq_main.F:(.text+0x3dd): undefined reference to mpi_finalize_' /usr/bin/ld: se_data_send_module.o: in function _se_data_send_module_MOD_se_5d_data_send’:
se_data_send_module.f:(.text+0x5a9): undefined reference to mpi_send_' /usr/bin/ld: se_data_send_module.f:(.text+0x62d): undefined reference to mpi_send

/usr/bin/ld: se_data_send_module.f:(.text+0x953): undefined reference to mpi_send_' /usr/bin/ld: se_data_send_module.o: in function __se_data_send_module_MOD_se_4de_data_send’:se_data_send_module.f:(.text+0xb3a): undefined reference to mpi_send_' /usr/bin/ld: se_data_send_module.o: in function se_data_send_module_MOD_se_4d_data_send’:
se_data_send_module.f:(.text+0x15f3): undefined reference to mpi_send_' /usr/bin/ld: se_data_send_module.o:se_data_send_module.f:(.text+0x1674): more undefined references to mpi_send
’ follow
/usr/bin/ld: se_internal_util_module.o: in function __se_internal_util_module_MOD_se_generate_map': se_internal_util_module.f:(.text+0xa81): undefined reference to mpi_gather

/usr/bin/ld: se_internal_util_module.f:(.text+0xaab): undefined reference to mpi_bcast_' /usr/bin/ld: se_data_recv_module.o: in function _se_data_recv_module_MOD_se_5d_data_recv’:
se_data_recv_module.f:(.text+0x1c4): undefined reference to mpi_recv_' /usr/bin/ld: se_data_recv_module.o: in function se_data_recv_module_MOD_se_4de_data_recv’:se_data_recv_module.f:(.text+0xb3c): undefined reference to mpi_recv_' /usr/bin/ld: se_data_recv_module.f:(.text+0x11ee): undefined reference to mpi_recv
/usr/bin/ld: se_data_recv_module.o: in function __se_data_recv_module_MOD_se_4d_data_recv': se_data_recv_module.f:(.text+0x13c1): undefined reference to mpi_recv

/usr/bin/ld: se_data_recv_module.o: in function __se_data_recv_module_MOD_se_3de_data_recv':se_data_recv_module.f:(.text+0x1b77): undefined reference to mpi_recv

/usr/bin/ld: se_data_recv_module.o:se_data_recv_module.f:(.text+0x2153): more undefined references to mpi_recv_' follow /usr/bin/ld: se_global_sum_module.o: in function _se_global_sum_module_MOD_se_global_rasum’:
se_global_sum_module.f:(.text+0xc8): undefined reference to mpi_reduce_' /usr/bin/ld: se_global_sum_module.f:(.text+0x119): undefined reference to mpi_bcast

/usr/bin/ld: se_global_sum_module.f:(.text+0x187): undefined reference to mpi_reduce_' /usr/bin/ld: se_global_sum_module.o: in function _se_global_sum_module_MOD_se_global_iasum’:
se_global_sum_module.f:(.text+0x678): undefined reference to mpi_reduce_' /usr/bin/ld: se_global_sum_module.f:(.text+0x6c9): undefined reference to mpi_bcast

/usr/bin/ld: se_global_sum_module.f:(.text+0x737): undefined reference to mpi_reduce_' /usr/bin/ld: se_global_sum_module.o: in function _se_global_sum_module_MOD_se_global_rsum’:se_global_sum_module.f:(.text+0xafe): undefined reference to mpi_reduce_' /usr/bin/ld: se_global_sum_module.f:(.text+0xb37): undefined reference to mpi_bcast
/usr/bin/ld: se_global_sum_module.o: in function `__se_global_sum_module_MOD_se_global_isum’:

I set the gcc section in my config_cmaq.csh file as follows:
#> set the compiler flag -fopt-info-missed to generate a missed optimization report in the bldit logfile
setenv myFC gfortran
setenv myCC gcc
setenv myFSTD “-O3 -funroll-loops -finit-character=32 -Wtabs -Wsurprising -march=native -ftree-vectorize -ftree-loop-if-convert -finline-limit=512”
setenv myDBG “-Wall -O0 -g -fcheck=all -ffpe-trap=invalid,zero,overflow -fbacktrace” setenv myFFLAGS “-ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32”
setenv myFRFLAGS “-ffree-form -ffree-line-length-none -funroll-loops -finit-character=32”
setenv myCFLAGS “-O2”
setenv myLINK_FLAG “-fopenmp” # “-fopenmp” # openMP may be required if I/O API was built using this link flag.
setenv extra_lib “”
setenv mpi_lib “-lmpich”

My Makefile as follows:
EXEC = CCTM_v533.exe

FC = gfortran
CC = gcc

LIB = /home/mapuni/WRF/CMAQ-5.3.3/lib/x86_64/gcc
include_path = -I (LIB)/ioapi/lib \ -I (LIB)/ioapi/include_files
-I (LIB)/mpi/lib -I.  WARN = FSTD = -O3 -funroll-loops -finit-character=32 -Wtabs -Wsurprising -march=native -ftree-vectorize -ftree-loop-if-convert -finline-limit=512 DBG = -Wall -O0 -g -fcheck=all -ffpe-trap=invalid,zero,overflow -fbacktrace  ifneq (,(filter (debug), TRUE true True T )) DEBUG = TRUE endif  ifneq (,(filter (DEBUG), TRUE true )) f_FLAGS = -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 (DBG) (include_path) f90_FLAGS = -ffree-form -ffree-line-length-none -funroll-loops -finit-character=32 (DBG) (include_path) else f_FLAGS = -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 (FSTD) (include_path) f90_FLAGS = -ffree-form -ffree-line-length-none -funroll-loops -finit-character=32 (FSTD) (include_path) endif  F_FLAGS = (f_FLAGS) -fallow-argument-mismatch
F90_FLAGS = (f90_FLAGS) -fallow-argument-mismatch C_FLAGS = -O2 -DFLDMN -I(LIB)/mpi/include -I.

LINK_FLAGS = -fopenmp

REPOROOT = /home/mapuni/WRF/CMAQ-5.3.3/CCTM/src

Thank you very much for your help. I have been stuck for many days and even after changing to the mpich version, I still cannot generate CCTMv533.exe properly

If you change MPI versions, then you need to carefully recreate the hard coded links to the MPI libraries.
The links are created when you source the config_cmaq.csh

These are the environment variables that you likely modified in the config_cmaq.csh

        setenv MPI_INCL_DIR           /<your_path>/include   #> MPI Include directory pat
        setenv MPI_LIB_DIR             /<your_path>/lib    #> MPI Lib directory path

Later in the config_cmaq.csh the links to these paths are created:

#> Create Symbolic Links to Libraries
 if ( ! -d $CMAQ_LIB ) mkdir -p $CMAQ_LIB
 if (   -e $MPI_DIR  ) rm -rf $MPI_DIR
     mkdir $MPI_DIR
     ln -s $MPI_LIB_DIR $MPI_DIR/lib
     ln -s $MPI_INCL_DIR $MPI_DIR/include

I recommend that you verify the linked path has been updated.

Look in the following directory if you are using the gcc compiler:

ls -lrt CMAQ/lib/x86_64/gcc/mpi

Output example:

lrwxrwxrwx 1 lizadams rc_cep-emc_psx 58 Dec 22 00:40 lib -> /nas/longleaf/apps-dogwood/mpi/gcc_9.1.0/openmpi_4.0.1/lib
lrwxrwxrwx 1 lizadams rc_cep-emc_psx 62 Dec 22 00:40 include -> /nas/longleaf/apps-dogwood/mpi/gcc_9.1.0/openmpi_4.0.1/include

If these links are not updated then change to the directory and remove the links and then resource the config_cmaq.csh

Thank you for your reply, dear lizadams

I have ensured that there are no issues with my mpich and the lib link path of CMAQ when raising the questions you mentioned.

In the end, I modified the Makefile of CCTM by changing FC=gfortran to FC=mpifort, which solved my compilation problem.

EXEC = CCTM_v533.exe

FC = mpifort
CC = gcc

LIB = /home/mapuni/WRF/CMAQ-5.3.3/lib/x86_64/gcc
include_path = -I $(LIB)/ioapi/lib \
               -I $(LIB)/ioapi/include_files \
               -I $(LIB)/mpi/lib -I.

I am not sure what the reason is.
Thank you again for your patient reply and answer!