MPI undefined error when installing CCTM using gcc and gfortran

CMAQ Compiling
1

Hi everyone,
I successfully compiled MCIP BCON and ICON when configuring CMAQv5.3.3 using mpich4.0 ioapi-3.2-large. However, I encountered an error in the mpi library link during the installation of CCTM. I would like to know the reason for this. Below is the log file when I make CCTM
/usr/bin/ld: cmaq_main.o: in function MAIN__': cmaq_main.F:(.text+0x2ea): undefined reference to mpi_wtime_’
/usr/bin/ld: cmaq_main.F:(.text+0x3dd): undefined reference to mpi_finalize_' /usr/bin/ld: se_data_send_module.o: in function _se_data_send_module_MOD_se_5d_data_send’:
se_data_send_module.f:(.text+0x5a9): undefined reference to mpi_send_' /usr/bin/ld: se_data_send_module.f:(.text+0x62d): undefined reference to mpi_send’
/usr/bin/ld: se_data_send_module.f:(.text+0x953): undefined reference to mpi_send_' /usr/bin/ld: se_data_send_module.o: in function __se_data_send_module_MOD_se_4de_data_send’:se_data_send_module.f:(.text+0xb3a): undefined reference to mpi_send_' /usr/bin/ld: se_data_send_module.o: in function se_data_send_module_MOD_se_4d_data_send’:
se_data_send_module.f:(.text+0x15f3): undefined reference to mpi_send_' /usr/bin/ld: se_data_send_module.o:se_data_send_module.f:(.text+0x1674): more undefined references to mpi_send’ follow
/usr/bin/ld: se_internal_util_module.o: in function __se_internal_util_module_MOD_se_generate_map': se_internal_util_module.f:(.text+0xa81): undefined reference to mpi_gather’
/usr/bin/ld: se_internal_util_module.f:(.text+0xaab): undefined reference to mpi_bcast_' /usr/bin/ld: se_data_recv_module.o: in function _se_data_recv_module_MOD_se_5d_data_recv’:
se_data_recv_module.f:(.text+0x1c4): undefined reference to mpi_recv_' /usr/bin/ld: se_data_recv_module.o: in function se_data_recv_module_MOD_se_4de_data_recv’:se_data_recv_module.f:(.text+0xb3c): undefined reference to mpi_recv_' /usr/bin/ld: se_data_recv_module.f:(.text+0x11ee): undefined reference to mpi_recv’
/usr/bin/ld: se_data_recv_module.o: in function __se_data_recv_module_MOD_se_4d_data_recv': se_data_recv_module.f:(.text+0x13c1): undefined reference to mpi_recv’
/usr/bin/ld: se_data_recv_module.o: in function __se_data_recv_module_MOD_se_3de_data_recv':se_data_recv_module.f:(.text+0x1b77): undefined reference to mpi_recv’
/usr/bin/ld: se_data_recv_module.o:se_data_recv_module.f:(.text+0x2153): more undefined references to mpi_recv_' follow /usr/bin/ld: se_global_sum_module.o: in function _se_global_sum_module_MOD_se_global_rasum’:
se_global_sum_module.f:(.text+0xc8): undefined reference to mpi_reduce_' /usr/bin/ld: se_global_sum_module.f:(.text+0x119): undefined reference to mpi_bcast’
/usr/bin/ld: se_global_sum_module.f:(.text+0x187): undefined reference to mpi_reduce_' /usr/bin/ld: se_global_sum_module.o: in function _se_global_sum_module_MOD_se_global_iasum’:
se_global_sum_module.f:(.text+0x678): undefined reference to mpi_reduce_' /usr/bin/ld: se_global_sum_module.f:(.text+0x6c9): undefined reference to mpi_bcast’
/usr/bin/ld: se_global_sum_module.f:(.text+0x737): undefined reference to mpi_reduce_' /usr/bin/ld: se_global_sum_module.o: in function _se_global_sum_module_MOD_se_global_rsum’:se_global_sum_module.f:(.text+0xafe): undefined reference to mpi_reduce_' /usr/bin/ld: se_global_sum_module.f:(.text+0xb37): undefined reference to mpi_bcast’
/usr/bin/ld: se_global_sum_module.o: in function `__se_global_sum_module_MOD_se_global_isum’:

————————————————————————
I set the gcc section in my config_cmaq.csh file as follows：
#> set the compiler flag -fopt-info-missed to generate a missed optimization report in the bldit logfile
setenv myFC gfortran
setenv myCC gcc
setenv myFSTD “-O3 -funroll-loops -finit-character=32 -Wtabs -Wsurprising -march=native -ftree-vectorize -ftree-loop-if-convert -finline-limit=512”
setenv myDBG “-Wall -O0 -g -fcheck=all -ffpe-trap=invalid,zero,overflow -fbacktrace” setenv myFFLAGS “-ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32”
setenv myFRFLAGS “-ffree-form -ffree-line-length-none -funroll-loops -finit-character=32”
setenv myCFLAGS “-O2”
setenv myLINK_FLAG “-fopenmp” # “-fopenmp” # openMP may be required if I/O API was built using this link flag.
setenv extra_lib “”
setenv mpi_lib “-lmpich”

My Makefile as follows:
EXEC = CCTM_v533.exe

FC = gfortran
CC = gcc

LIB = /home/mapuni/WRF/CMAQ-5.3.3/lib/x86_64/gcc
include_path = -I (LIB)/ioapi/lib \ -I (LIB)/ioapi/include_files
-I (LIB)/mpi/lib -I.  WARN = FSTD = -O3 -funroll-loops -finit-character=32 -Wtabs -Wsurprising -march=native -ftree-vectorize -ftree-loop-if-convert -finline-limit=512 DBG = -Wall -O0 -g -fcheck=all -ffpe-trap=invalid,zero,overflow -fbacktrace  ifneq (,(filter (debug), TRUE true True T )) DEBUG = TRUE endif  ifneq (,(filter (DEBUG), TRUE true )) f_FLAGS = -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 (DBG) (include_path) f90_FLAGS = -ffree-form -ffree-line-length-none -funroll-loops -finit-character=32 (DBG) (include_path) else f_FLAGS = -ffixed-form -ffixed-line-length-132 -funroll-loops -finit-character=32 (FSTD) (include_path) f90_FLAGS = -ffree-form -ffree-line-length-none -funroll-loops -finit-character=32 (FSTD) (include_path) endif  F_FLAGS = (f_FLAGS) -fallow-argument-mismatch
F90_FLAGS = (f90_FLAGS) -fallow-argument-mismatch C_FLAGS = -O2 -DFLDMN -I(LIB)/mpi/include -I.

LINKER = $(FC)
LINK_FLAGS = -fopenmp

REPOROOT = /home/mapuni/WRF/CMAQ-5.3.3/CCTM/src

Thank you very much for your help. I have been stuck for many days and even after changing to the mpich version, I still cannot generate CCTMv533.exe properly

2

If you change MPI versions, then you need to carefully recreate the hard coded links to the MPI libraries.
The links are created when you source the config_cmaq.csh

These are the environment variables that you likely modified in the config_cmaq.csh

        setenv MPI_INCL_DIR           /<your_path>/include   #> MPI Include directory pat
        setenv MPI_LIB_DIR             /<your_path>/lib    #> MPI Lib directory path

Later in the config_cmaq.csh the links to these paths are created:

#> Create Symbolic Links to Libraries
 if ( ! -d $CMAQ_LIB ) mkdir -p $CMAQ_LIB
 if (   -e $MPI_DIR  ) rm -rf $MPI_DIR
     mkdir $MPI_DIR
     ln -s $MPI_LIB_DIR $MPI_DIR/lib
     ln -s $MPI_INCL_DIR $MPI_DIR/include

I recommend that you verify the linked path has been updated.

Look in the following directory if you are using the gcc compiler:

ls -lrt CMAQ/lib/x86_64/gcc/mpi

Output example:

lrwxrwxrwx 1 lizadams rc_cep-emc_psx 58 Dec 22 00:40 lib -> /nas/longleaf/apps-dogwood/mpi/gcc_9.1.0/openmpi_4.0.1/lib
lrwxrwxrwx 1 lizadams rc_cep-emc_psx 62 Dec 22 00:40 include -> /nas/longleaf/apps-dogwood/mpi/gcc_9.1.0/openmpi_4.0.1/include

If these links are not updated then change to the directory and remove the links and then resource the config_cmaq.csh

3

Thank you for your reply, dear lizadams

I have ensured that there are no issues with my mpich and the lib link path of CMAQ when raising the questions you mentioned.

In the end, I modified the Makefile of CCTM by changing FC=gfortran to FC=mpifort, which solved my compilation problem.

EXEC = CCTM_v533.exe

FC = mpifort
CC = gcc

LIB = /home/mapuni/WRF/CMAQ-5.3.3/lib/x86_64/gcc
include_path = -I $(LIB)/ioapi/lib \
               -I $(LIB)/ioapi/include_files \
               -I $(LIB)/mpi/lib -I.

I am not sure what the reason is.
Thank you again for your patient reply and answer!