CMAQ Compiling Issue

deeksha · April 30, 2020, 2:27pm

Hi,

I compiled ioapi3.2 with gnu8.1.0 compilers. Now, when I try to compile CMAQ, I run into following error. Let me know if I can provide more information that may help. Thanks

$HOME/CMAQ_install/lib/x86_64/gcc/ioapi/lib/libioapi.a(wratt3.o): In function `master.0.wratt3_’:

wratt3.F90:(.text+0x646): undefined reference to `GOMP_critical_name_start’

wratt3.F90:(.text+0x6c6): undefined reference to `GOMP_critical_name_end’

wratt3.F90:(.text+0xa47): undefined reference to `GOMP_critical_name_end’

wratt3.F90:(.text+0x12de): undefined reference to `GOMP_critical_name_end’

collect2: error: ld returned 1 exit status

make: *** [CCTM_v531.exe] Error 1

ERROR while running make command

mv Makefile Makefile.gcc

if ( -e Makefile.gcc && -e Makefile ) rm Makefile

ln -s Makefile.gcc Makefile

if ( 0 != 0 ) then

if ( -e $HOME/CMAQ_install/CCTM/scripts/BLD_CCTM_v531_gcc/CCTM_v531.cfg ) then

mv CCTM_v531.cfg.bld $HOME/CMAQ_install/CCTM/scripts/BLD_CCTM_v531_gcc/CCTM_v531.cfg

exit

cjcoats · April 30, 2020, 3:57pm

Your compile and link flags for CMAQ should include the contents of the OMPFLAGS variable from your ioapi/Makeinclude.$BIN, in this case, -fopenmp.

deeksha · May 1, 2020, 1:41pm

Hello, That worked. Now when I try to run the 2016 test case, I get a few errors. I am pasting some parts of the log below. Any help is appreciated

ln: failed to create symbolic link ‘/home/user/cmaq_build/CMAQ_install/lib/x86_64/gcc/mpi’: File exists
ln: failed to create symbolic link ‘/home/user/cmaq_build/CMAQ_install/lib/x86_64/gcc/mpi/centos7.5_gnu8.1.0’: Permission denied
ln: failed to create symbolic link ‘/home/user/cmaq_build/CMAQ_install/lib/x86_64/gcc/mpi’: File exists
ln: failed to create symbolic link ‘/home/user/cmaq_build/CMAQ_install/lib/x86_64/gcc/mpi’: File exists
ln: failed to create symbolic link ‘/home/user/cmaq_build/CMAQ_install/lib/x86_64/gcc/mpi’: File exists
.
.
.
.

An ORTE daemon has unexpectedly failed after launch and before
communicating back to mpirun. This could be caused by a number
of factors, including an inability to create a connection back
to mpirun due to a lack of common network interfaces and/or no
route found between them. Please check network connectivity
(including firewalls and network routing requirements).
.
.
.
.
.
.
ln: cannot remove ‘/home/user/cmaq_build/CMAQ_install/lib/x86_64/gcc/netcdff/includ’: No such file or directory
ln: cannot remove ‘/home/user/cmaq_build/CMAQ_install/lib/x86_64/gcc/netcdff/includ’: No such file or directory
ln: cannot remove ‘/home/user/cmaq_build/CMAQ_install/lib/x86_64/gcc/netcdff/includ’: No such file or directory

cjcoats · May 1, 2020, 2:11pm

This looks like a spelling error: includ should probably be include

deeksha · May 1, 2020, 2:24pm

Could it be character length issue given the path is long?

cjcoats · May 1, 2020, 2:41pm

You might try going to the BLD… directory and seeing if you can fix the Makefile by hand.

deeksha · May 1, 2020, 7:26pm

Thanks. Now, I am getting another error. It looks like a memory related issue. However, I am requesting for enough memory I think. For testing, I am running on a single node with 32 CPUS (requesting >100gb memory) and I use srun to run the job. Thanks so much for your help.

[1588360333.428623] [or-condo-c44:101973:0] ib_md.c:1758 UCX ERROR ibv_open_device(mlx4_0) failed: Cannot allocate memory
[1588360333.431491] [or-condo-c44:101684:0] ib_md.c:1758 UCX ERROR ibv_open_device(mlx4_0) failed: Cannot allocate memory
[1588360333.443975] [or-condo-c44:101576:0] ib_md.c:1758 UCX ERROR ibv_open_device(mlx4_0) failed: Cannot allocate memory

 ***  ERROR in INIT3/INITLOG3  ***
 Error opening log file on unit        99
 I/O STATUS =        17
 File: CTM_LOG_000.v531_gcc_Bench_2016_12SE1_20160701

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Process name: [[22616,1],15]
Exit code: 2

deeksha · May 2, 2020, 4:03am

I checked one of the cctm.log file and it has the following error. The MET_CRO_3D file looks fine under the input Met path. The variable in the netcdf file is “TFLAG” and not “FLAG”. This is in the 2016 SE test case input. That could be the issue. Have you seen this before? Any suggestions will be helpful. Thanks so much for all the help!

ioapi-3.2: Id: init3.F90 150 2020-04-11 17:51:44Z coats
Version with PARMS3.EXT/PARAMETER::MXVARS3= 2048
netCDF version 4.6.1 of Sep 8 2018 14:57:57 $

 EXECUTION_ID: CMAQ_CCTMv531_dry_20200502_035708_452510341
 Error reading file attribute FLAG
 netCDF error number  -49  processing file "MET_CRO_3D"
 NetCDF: Variable not found

 *** ERROR ABORT in subroutine RUNTIME_VARS
 Could not open MET_CRO_3D       file

cjcoats · May 2, 2020, 11:11am

This is a malformed input file: this message happens only for files of type DCTNRY3 = 0 and a properly formed MET_CRO_3D should be of type GRDDED3 = 1.

Did the script that caused this file to be created ignore exit-failure for the program that created it?

deeksha · May 4, 2020, 8:27pm

Thank you for your response. I am running the 2016 SE test case. So, I downloaded this input file from CMAQ drive.
https://drive.google.com/drive/folders/10wFNch1MkI49ZjD2XD6wK2xzDWOav2zY

deeksha · May 6, 2020, 12:54pm

Hi,

Do you have any suggestions that I can try to fix this issue. Thank you.

cjcoats · May 6, 2020, 1:29pm

What does ncdump -h say about this file?
If it shows FTYPE = 1 then you have a corrupted build, in which case you should re-build everything – loibraries and model – from scratch (making sure that the INCLUDE-files all match (you haven’t been changing any of the INCLUDE-files, have you?)).

deeksha · May 6, 2020, 5:58pm

Thanks for your response. I am not changing any INCLUDE-files. I re-build everything again. Same problem persists. I am using pre installed netcdf/4.6.1 libraries. Could the problem be related to that? Any idea! Again appreciate your help!

cjcoats · May 6, 2020, 10:55pm

What does ncdump -h say about this file?

deeksha · May 6, 2020, 11:16pm

ncdump -h METCRO2D_160701.nc
netcdf METCRO2D_160701 {
dimensions:
TSTEP = UNLIMITED ; // (25 currently)
DATE-TIME = 2 ;
LAY = 1 ;
VAR = 35 ;
ROW = 80 ;
COL = 100 ;
variables:
int TFLAG(TSTEP, VAR, DATE-TIME) ;
TFLAG:units = “<YYYYDDD,HHMMSS>” ;
TFLAG:long_name = "TFLAG " ;
TFLAG:var_desc = "Timestep-valid flags: (1) YYYYDDD or (2) HHMMSS " ;
float PRSFC(TSTEP, LAY, ROW, COL) ;
PRSFC:long_name = "PRSFC " ;
PRSFC:units = "Pa " ;
PRSFC:var_desc = "surface pressure " ;
float USTAR(TSTEP, LAY, ROW, COL) ;
USTAR:long_name = "USTAR " ;
USTAR:units = "m s-1 " ;
USTAR:var_desc = "cell averaged friction velocity " ;
float WSTAR(TSTEP, LAY, ROW, COL) ;
WSTAR:long_name = "WSTAR " ;
WSTAR:units = "m s-1 " ;
WSTAR:var_desc = "convective velocity scale " ;
float PBL(TSTEP, LAY, ROW, COL) ;
PBL:long_name = "PBL " ;
PBL:units = "m " ;
PBL:var_desc = "PBL height " ;
float ZRUF(TSTEP, LAY, ROW, COL) ;
ZRUF:long_name = "ZRUF " ;
ZRUF:units = "m " ;
ZRUF:var_desc = "surface roughness length " ;
float MOLI(TSTEP, LAY, ROW, COL) ;
MOLI:long_name = "MOLI " ;
MOLI:units = "m-1 " ;
MOLI:var_desc = "inverse of Monin-Obukhov length " ;
float HFX(TSTEP, LAY, ROW, COL) ;
HFX:long_name = "HFX " ;
HFX:units = "W m-2 " ;
HFX:var_desc = "sensible heat flux " ;
float LH(TSTEP, LAY, ROW, COL) ;
LH:long_name = "LH " ;
LH:units = "W m-2 " ;
LH:var_desc = "latent heat flux " ;
float RADYNI(TSTEP, LAY, ROW, COL) ;
RADYNI:long_name = "RADYNI " ;
RADYNI:units = "m s-1 " ;
RADYNI:var_desc = "inverse of aerodynamic resistance " ;
float RSTOMI(TSTEP, LAY, ROW, COL) ;
RSTOMI:long_name = "RSTOMI " ;
RSTOMI:units = "m s-1 " ;
RSTOMI:var_desc = "inverse of stomatic resistance " ;
float TEMPG(TSTEP, LAY, ROW, COL) ;
TEMPG:long_name = "TEMPG " ;
TEMPG:units = "K " ;
TEMPG:var_desc = "skin temperature at ground " ;
float TEMP2(TSTEP, LAY, ROW, COL) ;
TEMP2:long_name = "TEMP2 " ;
TEMP2:units = "K " ;
TEMP2:var_desc = "temperature at 2 m " ;
float Q2(TSTEP, LAY, ROW, COL) ;
Q2:long_name = "Q2 " ;
Q2:units = "kg kg-1 " ;
Q2:var_desc = "mixing ratio at 2 m " ;
float WSPD10(TSTEP, LAY, ROW, COL) ;
WSPD10:long_name = "WSPD10 " ;
WSPD10:units = "m s-1 " ;
WSPD10:var_desc = "wind speed at 10 m " ;
float WDIR10(TSTEP, LAY, ROW, COL) ;
WDIR10:long_name = "WDIR10 " ;
WDIR10:units = "degree " ;
WDIR10:var_desc = "wind direction at 10 m " ;
float GLW(TSTEP, LAY, ROW, COL) ;
GLW:long_name = "GLW " ;
GLW:units = "W m-2 " ;
GLW:var_desc = "longwave radiation at ground " ;
float GSW(TSTEP, LAY, ROW, COL) ;
GSW:long_name = "GSW " ;
GSW:units = "W m-2 " ;
GSW:var_desc = "solar radiation absorbed at ground " ;
float RGRND(TSTEP, LAY, ROW, COL) ;
RGRND:long_name = "RGRND " ;
RGRND:units = "W m-2 " ;
RGRND:var_desc = "solar radiation reaching ground " ;
float RN(TSTEP, LAY, ROW, COL) ;
RN:long_name = "RN " ;
RN:units = "cm " ;
RN:var_desc = "nonconvective precipitation in interval " ;
float RC(TSTEP, LAY, ROW, COL) ;
RC:long_name = "RC " ;
RC:units = "cm " ;
RC:var_desc = "convective precipitation in interval " ;
float CFRAC(TSTEP, LAY, ROW, COL) ;
CFRAC:long_name = "CFRAC " ;
CFRAC:units = "1 " ;
CFRAC:var_desc = "total cloud fraction " ;
float CLDT(TSTEP, LAY, ROW, COL) ;
CLDT:long_name = "CLDT " ;
CLDT:units = "m " ;
CLDT:var_desc = "cloud top layer height " ;
float CLDB(TSTEP, LAY, ROW, COL) ;
CLDB:long_name = "CLDB " ;
CLDB:units = "m " ;
CLDB:var_desc = "cloud bottom layer height " ;
float WBAR(TSTEP, LAY, ROW, COL) ;
WBAR:long_name = "WBAR " ;
WBAR:units = "g m-3 " ;
WBAR:var_desc = "average liquid water content of cloud " ;
float SNOCOV(TSTEP, LAY, ROW, COL) ;
SNOCOV:long_name = "SNOCOV " ;
SNOCOV:units = "1 " ;
SNOCOV:var_desc = "snow cover " ;
float VEG(TSTEP, LAY, ROW, COL) ;
VEG:long_name = "VEG " ;
VEG:units = "1 " ;
VEG:var_desc = "vegetation coverage " ;
float LAI(TSTEP, LAY, ROW, COL) ;
LAI:long_name = "LAI " ;
LAI:units = "m2 m-2 " ;
LAI:var_desc = "leaf-area index " ;
float SEAICE(TSTEP, LAY, ROW, COL) ;
SEAICE:long_name = "SEAICE " ;
SEAICE:units = "1 " ;
SEAICE:var_desc = "sea ice " ;
float SNOWH(TSTEP, LAY, ROW, COL) ;
SNOWH:long_name = "SNOWH " ;
SNOWH:units = "m " ;
SNOWH:var_desc = "snow height " ;
float WR(TSTEP, LAY, ROW, COL) ;
WR:long_name = "WR " ;
WR:units = "m " ;
WR:var_desc = "canopy moisture content " ;
float SOIM1(TSTEP, LAY, ROW, COL) ;
SOIM1:long_name = "SOIM1 " ;
SOIM1:units = "m3 m-3 " ;
SOIM1:var_desc = "volumetric soil moisture in top cm " ;
float SOIM2(TSTEP, LAY, ROW, COL) ;
SOIM2:long_name = "SOIM2 " ;
SOIM2:units = "m3 m-3 " ;
SOIM2:var_desc = "volumetric soil moisture in top m " ;
float SOIT1(TSTEP, LAY, ROW, COL) ;
SOIT1:long_name = "SOIT1 " ;
SOIT1:units = "K " ;
SOIT1:var_desc = "soil temperature in top cm " ;
float SOIT2(TSTEP, LAY, ROW, COL) ;
SOIT2:long_name = "SOIT2 " ;
SOIT2:units = "K " ;
SOIT2:var_desc = "soil temperature in top m " ;
float SLTYP(TSTEP, LAY, ROW, COL) ;
SLTYP:long_name = "SLTYP " ;
SLTYP:units = "1 " ;
SLTYP:var_desc = "soil texture type by USDA category " ;

// global attributes:
:IOAPI_VERSION = "ioapi-3.2: Id: init3.F90 98 2018-04-05 14:35:07Z coats " ;
:EXEC_ID = "mcip " ;
:FTYPE = 1 ;
:CDATE = 2019357 ;
:CTIME = 150547 ;
:WDATE = 2019357 ;
:WTIME = 150547 ;
:SDATE = 2016183 ;
:STIME = 0 ;
:TSTEP = 10000 ;
:NTHIK = 1 ;
:NCOLS = 100 ;
:NROWS = 80 ;
:NLAYS = 1 ;
:NVARS = 35 ;
:GDTYP = 2 ;
:P_ALP = 33. ;
:P_BET = 45. ;
:P_GAM = -97. ;
:XCENT = -97. ;
:YCENT = 40. ;
:XORIG = 792000. ;
:YORIG = -1080000. ;
:XCELL = 12000. ;
:YCELL = 12000. ;
:VGTYP = 7 ;
:VGTOP = 5000.f ;
:VGLVLS = 1.f, 0.9975f ;
:GDNAM = “2016_12SE1_CROSS” ;
:UPNAM = "OUTCM3IO " ;
:VAR-LIST = "PRSFC USTAR WSTAR PBL ZRUF MOLI HFX LH RADYNI RSTOMI TEMPG TEMP2 Q2 WSPD10 WDIR10 GLW GSW RGRND RN RC CFRAC CLDT CLDB WBAR SNOCOV VEG LAI SEAICE SNOWH WR SOIM1 SOIM2 SOIT1 SOIT2 SLTYP " ;
:FILEDESC = "US EPA COMMUNITY MULTISCALE AIR QUALITY MODEL METEOROLOGY-CHEMISTRY INTERFACE PROCESSOR MCIP V5.1 FROZEN 11/21/2019 INPUT METEOROLOGY DATA FROM WRF ARW V3.8 INPUT RUN INITIALIZED: 2015-12-21-00:00:00.0000 CUMULUS PARAMETERIZATION: KF with radiative feedback SHALLOW CONVECTION: No shallow convection MICROPHYSICS: Morrison 2-moment LONGWAVE RADIATION: RRTMg SHORTWAVE RADIATION: RRTMg PBL SCHEME: ACM2 (Pleim) SURFACE LAYER SCHEME: Pleim LAND-SURFACE SCHEME: Pleim-Xiu Land-Surface Model URBAN MODEL: No urban physics LAND USE CLASSIFICATION: NLCD40 3D ANALYSIS NUDGING: GRID WIND COEFF: 1.000E-04 s-1 TEMP COEFF: 1.000E-04 s-1 MOIS COEFF: 1.000E-05 s-1 GEOP COEFF: not applicable SFC ANALYSIS NUDGING: STANDARD WIND COEFF: 0.000E+00 s-1 TEMP COEFF: 0.000E+00 s-1 MOIS COEFF: 0.000E+00 s-1 OBS NUDGING: OFF WIND COEFF: not applicable TEMP COEFF: not applicable MOIS COEFF: not applicable EARTH RADIUS ASSUMED IN MCIP: 6370000.000 m " ;
:HISTORY = “” ;
}

cjcoats · May 6, 2020, 11:23pm

So ncdump says FTYPE=1 (as it should be). However, your I/O API OPEN3 call thinks FTYPE=0.

This says some part of your CMAQ or I/O API installation is corrupted.

deeksha · May 7, 2020, 12:56am

Thanks. The executables look fine. I could not locate any error in the build log file either. So, not sure what’s happening. Any suggestions on debugging this? Thank you!

deeksha · May 8, 2020, 4:19am

Hi,

Just want to mention the issue was due to the version of netcdf I was using. Now, the test case is running. Thank you.

cjcoats · May 8, 2020, 11:18am

Would you tell us which netCDF version had problems? – and then I’ll put it into the I/O API Troubleshooting page https://cjcoats.github.io/ioapi/ERRORS.html, and credit you for it.

– Carlie

deeksha · May 8, 2020, 3:48pm

Sure. I was using preinstalled netcdf/4.6.1 that has zlib and HDF5 support, which was causing the above error. Switching to netcdf-c-4.7.4 and netcdf-fortran-4.5.2 fixed it.

Topic		Replies	Views
CMAQ_5.3.2 CCTM compile error CMAQ	2	489	February 16, 2021
CCTM Compile Error Compiling	3	1641	February 4, 2020
Error when compile create_omi CMAQ	5	861	September 17, 2021
CMAQ v5.3 compiling error Compiling	4	2113	June 6, 2021
Build CCTM CMAQv5.4 Error Compiling	2	371	January 2, 2023

CMAQ Compiling Issue

Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted.

Related topics

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.