I am trying to compile MCIP, however I keep getting the following undefined references:
/home/colethatcher/ioapi-3.2/Linux4_x86_64gfortmpi/libioapi.a(wrmpigrd.o): In function wrmpigrd_': wrmpigrd.F90:(.text+0x52): undefined reference tompi_bcast_’
/home/colethatcher/ioapi-3.2/Linux4_x86_64gfortmpi/libioapi.a(modpdata.o): In function __modpdata_MOD_pn_flag': modpdata.F90:(.text+0x350): undefined reference tompi_allreduce_’
/home/colethatcher/ioapi-3.2/Linux4_x86_64gfortmpi/libioapi.a(modpdata.o): In function __modpdata_MOD_pn_setup': modpdata.F90:(.text+0xeb0): undefined reference tompi_comm_rank_’
modpdata.F90:(.text+0xf0b): undefined reference to mpi_comm_group_' modpdata.F90:(.text+0xf35): undefined reference tompi_group_incl_’
modpdata.F90:(.text+0xf5a): undefined reference to mpi_comm_create_' modpdata.F90:(.text+0xf82): undefined reference tompi_group_excl_’
modpdata.F90:(.text+0xfa9): undefined reference to mpi_comm_create_' modpdata.F90:(.text+0xfc9): undefined reference tompi_comm_dup_’
modpdata.F90:(.text+0x1c3f): undefined reference to mpi_comm_dup_' /home/colethatcher/ioapi-3.2/Linux4_x86_64gfortmpi/libioapi.a(pn_wrvars.o): In functionpn_wrint_’:
pn_wrvars.F90:(.text+0xc49): undefined reference to mpi_recv_' pn_wrvars.F90:(.text+0x1474): undefined reference tompi_send_’
/home/colethatcher/ioapi-3.2/Linux4_x86_64gfortmpi/libioapi.a(pn_wrvars.o): In function pn_wrreal_': pn_wrvars.F90:(.text+0x2109): undefined reference tompi_recv_’
pn_wrvars.F90:(.text+0x2934): undefined reference to mpi_send_' /home/colethatcher/ioapi-3.2/Linux4_x86_64gfortmpi/libioapi.a(pn_wrvars.o): In functionpn_wrdble_’:
pn_wrvars.F90:(.text+0x35c9): undefined reference to mpi_recv_' pn_wrvars.F90:(.text+0x3df4): undefined reference tompi_send_’
/home/colethatcher/ioapi-3.2/Linux4_x86_64gfortmpi/libioapi.a(pn_wrvars.o): In function pn_wrint8_': pn_wrvars.F90:(.text+0x4a89): undefined reference tompi_recv_’
pn_wrvars.F90:(.text+0x52b4): undefined reference to `mpi_send_’
collect2: error: ld returned 1 exit status
Makefile:182: recipe for target ‘mcip.exe’ failed
This implies that the appropriate libraries are not being linked in the makefile. Here is the relevant portion of my makefile;
LIBS = -L$(IOAPI_ROOT)/Linux4_x86_64gfortmpi -lioapi
-L$(NETCDF)/lib -lnetcdff -lnetcdf -lpnetcdf -lmpich -lgomp
I added -lpnetcdf, -lmpich, and -lgomp which corrected a bunch of other undefined references. I’ve only included the remaining errors after linking those 3 libraries. My problem is that the undefined subroutines should be linked in -lmpich library, but they’re not.
Any suggestions?
Thanks in advance.
You don’t say what other undefined references you were getting, but since MCIP is an entirely serial code, there is no need for MPICH, openMP, or pnetCDF.
1 Like
I had a similar problem with the ifort compiler.
I was getting the following
ld: /proj/ie/proj/CMAS/CMAQ/CMAQv5.3_branch_UNC3/mvapich2_2.3rc1_intel_17.2/lib/x86_64/intel/ioapi/lib/libioapi.a(m3msg2.o): in function m3msg2_': m3msg2.F:(.text+0x20): undefined reference to__kmpc_global_thread_num’
ld: m3msg2.F:(.text+0x37): undefined reference to __kmpc_critical' ld: m3msg2.F:(.text+0x9e): undefined reference to__kmpc_end_critical’
I solved it by adding -qopenmp to the LIBS section of the Makefile
LIBS = -L$(IOAPI_ROOT)/lib -lioapi
-L$(NETCDF)/lib -lnetcdff -lnetcdf -qopenmp
I think the problem is that whatever flags are used to build ioapi need to be used for compiling MCIP and CMAQ. In my case I took a look at whether -qopenmp was used in the Makeinclude.* file for ioapi build, and found:
>grep openmp Makeinclude.Linux2_x86_64ifort
OMPFLAGS = -qopenmp # for Intel compilers, version 16 or later
OMPLIBS = -qopenmp
2 Likes
PnetCDF/MPI enabled I/O API should only be used with CMAQ; all others should use “vanilla” I/O API.
This is documented in the I/O API installation instructions, https://cjcoats.github.io/ioapi/AVAIL.html#build
though perhaps the wording might be improved (and strengthened).
In principle, a lot of effort has been expended to make it work anyway, but it does require substantial changes to compiler, compile-flags, and link-flags in Makefiles, etc.
@cothatch and @lizadams –
As was noted by @cgnolte and @cjcoats, MCIP is a serial code, so there is no dependency on parallel processing. MCIP will not need a version of I/O API that is built for MPI or openMP. There are no plans to make MCIP into a parallelized code.
However, it seems like there is another disconnect somewhere in the build scripts for the entire CMAQ system, so this will need to be addressed. I don’t know if it’s possible to (at least temporarily) maintain both a parallel build of I/O API and a serial build of I/O API. I realize this is suboptimal.
I’m open to suggestions on how to make this work more seamlessly for the user community.
Tanya
There are also dependencies on the netCDF build. You can use commands such as
nc-config --all
nf-config --all
To see how the C and Fortran versions of the netCDF library are built, respectively.
I have found that gfortran will not compile if there is a symbol that is unresolved in the *.a files, even if the code doesn’t use it. Try adding the following
-Wl,--unresolved-symbols=ignore-in-object-files
For example:
LIBS = -L$(IOAPI_ROOT) -lioapi \
-L$(NETCDF)/lib -lnetcdff -lnetcdf -lnetcdf -Wl,--unresolved-symbols=ignore-in-object-files
For more information see: https://stackoverflow.com/questions/5555632/can-gcc-not-complain-about-undefined-references
1 Like
The solution here is that you have different $BIN-directories for the two different library-versions. And you use one $BIN for CMAQ, and the other for everything else.
– Carlie
1 Like
It looks like you are using IOAPI 3.2. Since MCIP is strictly a serial code (it can be parallelized easily), you can only link with regular IOAPI (@cjoats has mentioned it as vanilla IOAPI). In other words, when you build your IOAPI library, set BIN Linux2_x86_64ifort not Linux2_x86_64ifortmpi (assume you were using ifort compiler and same things applies to gcc or pgi compiler).
Thank you all for your help.
I made another a serial build of I/O API as many of you suggested this resulted in the many of the same undefined references. However, I then included the -fopenmp library as @lizadams suggested and MCIP compiled correctly.
Thanks again
1 Like
Thanks to @cgnolte for pointing us in the correct direction that MCIP should not be pulling in parallel dependencies.
Unfortunately, trying to work around building MCIP with IOAPI compiled with parallel NetCDF using -Wl,--unresolved-symbols=ignore-in-object-files or -fopenmp doesn’t work for me either because I still get OpenMPI related errors:
gfortran -o mcip.exe -O3 -I/home/yul18051/CMAQ_Project/netcdf_both/include -I/home/yul18051/src/cmaq/5.2.1/spack/opt/spack/linux-rhel6-x86_64/gcc-9.1.0/ioapi-3.2-3emuycdwvyttrvy2i5fgnce26avrn7tm/include/fixed132 -I/home/yul18051/src/cmaq/5.2.1/spack/opt/spack/linux-rhel6-x86_64/gcc-9.1.0/ioapi-3.2-3emuycdwvyttrvy2i5fgnce26avrn7tm/lib -Wl,--unresolved-symbols=ignore-in-object-files mcip.o alloc_ctm.o alloc_met.o alloc_x.o bcldprc_ak.o blddesc.o chkwpshdr.o chkwrfhdr.o close_files.o collapx.o comheader.o comheader_lufrac.o comheader_mos.o comheader_soi.o cori.o dealloc_ctm.o dealloc_met.o dealloc_x.o detangle_soil_px.o e_aerk.o dynflds.o getgist.o getluse.o getmet.o getpblht.o getpsih.o getsdt.o getversion.o graceful_stop.o gridout.o init_io.o init_met.o init_x.o julian.o layht.o ll2xy_lam.o ll2xy_lam_sec.o ll2xy_lam_tan.o ll2xy_merc.o ll2xy_ps.o locate.o lucro.o mapfac_lam.o mapfac_merc.o mapfac_ps.o metcro.o metdot.o metgrid2ctm.o metvars2ctm.o mm5v3opts.o moscro.o pblsup.o ptemp.o pvs.o qsat.o rdmm5v3.o rdwrfem.o readnml.o readter.o refstate.o resistcalc.o setgriddefs.o setup.o setup_mm5v3.o setup_wrfem.o sfclayer.o soilcro.o statflds.o vertarys.o vertnhy.o vertnhy_wrf.o vstamp.o vtemp.o wind.o wrfemopts.o wrgdesc.o xy2ll_lam.o xy2ll_merc.o const_mod.o const_pbl_mod.o coord_mod.o date_time_mod.o date_pack_mod.o files_mod.o groutcom_mod.o luoutcom_mod.o luvars_mod.o mcipparm_mod.o mcoutcom_mod.o mdoutcom_mod.o metinfo_mod.o metvars_mod.o mosoutcom_mod.o soioutcom_mod.o vgrd_mod.o wrf_netcdf_mod.o xvars_mod.o sat2mcip_mod.o -L/home/yul18051/src/cmaq/5.2.1/spack/opt/spack/linux-rhel6-x86_64/gcc-9.1.0/ioapi-3.2-3emuycdwvyttrvy2i5fgnce26avrn7tm/lib -lioapi -L/home/yul18051/CMAQ_Project/netcdf_both/lib -lnetcdff -lnetcdf
/home/yul18051/src/cmaq/5.2.1/spack/opt/spack/linux-rhel6-x86_64/gcc-9.1.0/openmpi-3.1.4-kjjd2yn2ppk3ra6amsukrdtsen6tmubo/lib/libopen-pal.so.40: undefined reference to `hwloc_topology_dup'
collect2: error: ld returned 1 exit status
make[1]: *** [mcip.exe] Error 1
make[1]: Leaving directory `/home/yul18051/src/cmaq/5.2.1/CMAQ-5.2.1/PREP/mcip/src'
make: *** [all] Error 2
make: Leaving directory `/home/yul18051/src/cmaq/5.2.1/CMAQ-5.2.1/PREP/mcip/src'
To make people’s lives easier, I’ve contributed an IOAPI package to spack @cjcoats so it’s trivial to maintain two separate and consistent IOAPI dependency tree installations with and without parallel-netcdf, namely:
# Parallel IOAPI for CMAQ
spack install ioapi ^netcdf+parallel-netcdf
# Serial IOAPI for MCIP
spack install ioapi
I’m going to try compiling MCIP against non-parallel NETCDF dependent IOAPI and see if solves my MCIP linking failure.
1 Like
Not sure, what you mean by others in ’ PnetCDF/MPI enabled I/O API should only be used with CMAQ ; all others should use “vanilla” I/O API.’, does that also include the following case?
I have built the ioapi3.2 without any mpi or parallel netcdf, and using it for making MCIP work to prepare CMAQ ready data from WRF. I am not using CMAQ (at this point), but only MCIP (v4.3) program on its own (git clone -b 4.3 https://github.com/CMASCenter/MCIP.git).
The mcip makefile for gfortran includes the lines [ I am using gcc]
#…gfortran
FC = gfortran
FFLAGS = -O3 -I$(NETCDF)/include -I$(IOAPI_ROOT)/Linux2_x86_64gfort
LIBS = -L$(IOAPI_ROOT)/Linux2_x86_64gfort -lioapi
-L$(NETCDF)/lib -lnetcdf -lnetcdff
[note the order of lnetcdf and lnetcdff]
With this, issuing ‘make’, I get
gfortran -o mcip.exe -O3 -I/home/somedir/cmaq_user/model_libs/library_cmaq/netcdf/include -I/home/somedir/cmaq_user/model_libs/library_cmaq/ioapi3.2/Linux2_x86_64gfort mcip.o alloc_ctm.o alloc_met.o alloc_x.o bcldprc_ak.o blddesc.o chkwpshdr.o chkwrfhdr.o close_files.o collapx.o comheader.o cori.o dealloc_ctm.o dealloc_met.o dealloc_x.o e_aerk.o dynflds.o getgist.o getluse.o getmet.o getpblht.o getsdt.o getversion.o graceful_stop.o gridout.o init_io.o init_met.o init_x.o julian.o layht.o ll2xy_lam.o ll2xy_lam_sec.o ll2xy_lam_tan.o ll2xy_merc.o ll2xy_ps.o locate.o mapfac_lam.o mapfac_merc.o mapfac_ps.o metcro.o metdot.o metgrid2ctm.o metvars2ctm.o mm5v3opts.o pblsup.o ptemp.o pvs.o qsat.o rdmm5v3.o rdwrfem.o readnml.o readter.o refstate.o resistcalc.o setgriddefs.o setup.o setup_mm5v3.o setup_wrfem.o sfclayer.o statflds.o vertarys.o vertnhy.o vertnhy_wrf.o vstamp.o vtemp.o wind.o wrfemopts.o wrgdesc.o xy2ll_lam.o xy2ll_merc.o const_mod.o const_pbl_mod.o coord_mod.o date_pack_mod.o date_time_mod.o files_mod.o groutcom_mod.o luvars_mod.o mcipparm_mod.o mcoutcom_mod.o mdoutcom_mod.o metinfo_mod.o metvars_mod.o vgrd_mod.o wrf_netcdf_mod.o xvars_mod.o sat2mcip_mod.o -L/home/somedir/cmaq_user/model_libs/library_cmaq/ioapi3.2/Linux2_x86_64gfort -lioapi -L/home/somedir/cmaq_user/model_libs/library_cmaq/netcdf/lib -lnetcdf -lnetcdff
/home/somedir/cmaq_user/model_libs/library_cmaq/ioapi3.2/Linux2_x86_64gfort/libioapi.a(m3msg2.o): In function m3msg2_': m3msg2.F:(.text+0x1e): undefined reference toGOMP_critical_name_start’
m3msg2.F:(.text+0x8d): undefined reference to GOMP_critical_name_end' /home/somedir/cmaq_user/model_libs/library_cmaq/ioapi3.2/Linux2_x86_64gfort/libioapi.a(m3msg2.o): In functionm3mesg_’:
m3msg2.F:(.text+0xd4): undefined reference to GOMP_critical_name_start' m3msg2.F:(.text+0x14d): undefined reference toGOMP_critical_name_end’
/home/somedir/cmaq_user/model_libs/library_cmaq/ioapi3.2/Linux2_x86_64gfort/libioapi.a(m3msg2.o): In function m3prompt_': m3msg2.F:(.text+0x1a4): undefined reference toGOMP_critical_name_start’
m3msg2.F:(.text+0x29e): undefined reference to GOMP_critical_name_end' . . . nf_varsio.F90:(.text+0x1714): undefined reference tonc_inq_varndims’
nf_varsio.F90:(.text+0x1816): undefined reference to nc_get_vars_double' /home/somedir/cmaq_user/model_libs/library_cmaq/ioapi3.2/Linux2_x86_64gfort/libnetcdff.a(nf_varsio.o): In functionnf_get_vars_’:
nf_varsio.F90:(.text+0x18a5): undefined reference to nc_inq_varndims' nf_varsio.F90:(.text+0x19a7): undefined reference tonc_get_vars’
collect2: error: ld returned 1 exit status
make[1]: *** [mcip.exe] Error 1
make[1]: Leaving directory /home/somedir/cmaq_user/mcip_test/MCIP/src' make[1]: Leaving directory/home/somedir/cmaq_user/mcip_test/MCIP/src’
make: *** [all] Error 2
However, if the order of lnetcdf and lnetcdff is just reversed, the I am getting the following:
gfortran -o mcip.exe -O3 -I/home/somedir/cmaq_user/model_libs/library_cmaq/netcdf/include -I/home/somedir/cmaq_user/model_libs/library_cmaq/ioapi3.2/Linux2_x86_64gfort mcip.o alloc_ctm.o alloc_met.o alloc_x.o bcldprc_ak.o blddesc.o chkwpshdr.o chkwrfhdr.o close_files.o collapx.o comheader.o cori.o dealloc_ctm.o dealloc_met.o dealloc_x.o e_aerk.o dynflds.o getgist.o getluse.o getmet.o getpblht.o getsdt.o getversion.o graceful_stop.o gridout.o init_io.o init_met.o init_x.o julian.o layht.o ll2xy_lam.o ll2xy_lam_sec.o ll2xy_lam_tan.o ll2xy_merc.o ll2xy_ps.o locate.o mapfac_lam.o mapfac_merc.o mapfac_ps.o metcro.o metdot.o metgrid2ctm.o metvars2ctm.o mm5v3opts.o pblsup.o ptemp.o pvs.o qsat.o rdmm5v3.o rdwrfem.o readnml.o readter.o refstate.o resistcalc.o setgriddefs.o setup.o setup_mm5v3.o setup_wrfem.o sfclayer.o statflds.o vertarys.o vertnhy.o vertnhy_wrf.o vstamp.o vtemp.o wind.o wrfemopts.o wrgdesc.o xy2ll_lam.o xy2ll_merc.o const_mod.o const_pbl_mod.o coord_mod.o date_pack_mod.o date_time_mod.o files_mod.o groutcom_mod.o luvars_mod.o mcipparm_mod.o mcoutcom_mod.o mdoutcom_mod.o metinfo_mod.o metvars_mod.o vgrd_mod.o wrf_netcdf_mod.o xvars_mod.o sat2mcip_mod.o -L/home/somedir/cmaq_user/model_libs/library_cmaq/ioapi3.2/Linux2_x86_64gfort -lioapi -L/home/somedir/cmaq_user/model_libs/library_cmaq/netcdf/lib -lnetcdff -lnetcdf
/home/somedir/cmaq_user/model_libs/library_cmaq/ioapi3.2/Linux2_x86_64gfort/libioapi.a(m3msg2.o): In function m3msg2_': m3msg2.F:(.text+0x1e): undefined reference toGOMP_critical_name_start’
m3msg2.F:(.text+0x8d): undefined reference to GOMP_critical_name_end' /home/somedir/cmaq_user/model_libs/library_cmaq/ioapi3.2/Linux2_x86_64gfort/libioapi.a(m3msg2.o): In functionm3mesg_’:
m3msg2.F:(.text+0xd4): undefined reference to GOMP_critical_name_start' m3msg2.F:(.text+0x14d): undefined reference toGOMP_critical_name_end’
.
.
.
/home/somedir/cmaq_user/model_libs/library_cmaq/ioapi3.2/Linux2_x86_64gfort/libioapi.a(wratt3.o): In function master.0.wratt3_': wratt3.F90:(.text+0x114): undefined reference toGOMP_critical_name_start’
wratt3.F90:(.text+0x191): undefined reference to GOMP_critical_name_end' wratt3.F90:(.text+0xa47): undefined reference toGOMP_critical_name_end’
wratt3.F90:(.text+0xe0b): undefined reference to GOMP_critical_name_end' collect2: error: ld returned 1 exit status make[1]: *** [mcip.exe] Error 1 make[1]: Leaving directory/home/somedir/cmaq_user/mcip_test/MCIP/src’
make: *** [all] Error 2
Any hints to get out of it?
Thank you.
You are correct to switch the order of the netcdf libraries to: lnetcdff and lnetcdf (note the order is dependent on what version of netcdf you are using)
The next step is to add the flag -qopenmp to the link step, and that will resolve the GOMP_critical_name_end and other undefined references.
That’s -fopenmp for gfortran; -qopenmp is for Intel ifort version 16 or later.
In case it’s useful to anyone trying to compile MCIP, I’ve uploaded my 2 shell scripts here:
Thank you, but where should I put the ‘-fopenmp’, There is no place withing MCIP for that? The Makefile from MCIP/src has the following main part
.SUFFIXES:
.SUFFIXES: .o .f90 .F90
MODEL = mcip.exe
#NETCDF = /home/somedir/cmaq_user/model_libs/library_cmaq/netcdf
NETCDF = /home/somedir/cmaq_user/model_libs/library_cmaq/netcdf
#NETCDF = /home/somedir/cmaq_user/model_libs/library_cmaq/netcdf
IOAPI_ROOT = /home/somedir/cmaq_user/model_libs/library_cmaq/ioapi3.2
#…gfortran
FC = gfortran
FFLAGS = -O3 -I$(NETCDF)/include -I$(IOAPI_ROOT)/Linux2_x86_64gfort
LIBS = -L$(IOAPI_ROOT)/Linux2_x86_64gfort -lioapi
-L$(NETCDF)/lib -lnetcdff -lnetcdf
-L$(PVM)/lib -lpvm3
DEFS =
MODULES =
const_mod.o
.
.
.
sat2mcip_mod.o
OBJS =
mcip.o
valloc_ctm.o
.
.
.
xy2ll_merc.o
all:
@(MAKE) (MODULES)
@(MAKE) (MODEL)
(MODEL): (OBJS)
(FC) -o (MODEL) (FFLAGS) (OBJS) (MODULES) (LIBS)
(OBJS): (MODULES)
.f90.o:
(FC) (FFLAGS) (INCLUDES) -c <
.F90.o:
(FC) (FFLAGS) (INCLUDES) -c < $(DEFS)
clean:
rm -f *.o *.mod *.il $(MODEL)
So, where does that linker flag (-qopenmp or -fopenmp) go? [ just a reminder, I am using MCIP on its own, no relation with CMAQ at this point, and I don’t see any place for mpi too!]
Thank you.
at the start of the right-hand-side for LIBS
Thanks for the info, but ‘module purge’ not working in our linux cluster. No command ‘module’, and not sure how to build it from source!
Thank you for the quick reply, really appreciate!! Seems working and will come back if any issue arises.
I got passed that step, but now stuck at running mcip; The error is
MCIP>./run.mcip
Tue Jul 30 20:07:37 UTC 2019
rm: No match.
This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3
I/O Applications Programming Interface, [I/O API] which is
built on top of the netCDF I/O library (Copyright 1993, 1996
University Corporation for Atmospheric Research/Unidata
Program) and the PVM parallel-programming library (from
Oak Ridge National Laboratory).
Copyright (C) 1992-2002 MCNC,
(C) 1992-2013 Carlie J. Coats, Jr.,
(C) 2003-2012 Baron Advanced Meteorological Systems, LLC, and
(C) 2014-2019 UNC Institute for the Environment.
Released under the GNU LGPL License, version 2.1. See URL
https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html
for conditions of use.
ioapi-3.2: $Id: init3.F90 120 2019-06-21 14:18:20Z coats $
Version with PARMS3.EXT/PARAMETER::MXVARS3= 2048
netCDF version 4.4.1 of Jul 28 2019 23:01:39 $
EXECUTION_ID: mcip
==============================================================================
US EPA COMMUNITY MULTISCALE AIR QUALITY MODEL
METEOROLOGY-CHEMISTRY INTERFACE PROCESSOR
MCIP V4.3 FROZEN 11/06/2015
==============================================================================
*** SUBROUTINE: SETUP
*** UNKNOWN OR UNSUPPORTED WRF OUTPUT VERSION
*** VERSION = OUTPUT FROM WRF V4.0 MODEL
*** ERROR ABORT in subroutine SETUP
ABNORMAL TERMINATION IN SETUP
Date and time 0:00:00 June 1, 2019 (2019152:000000)
Error running mcip
Can anybody hint? Is WRF v 4.0 not supported?
Thank you.