Icon and bcon get stuck while running for the single day benchmark


#1

Hello, I’m using the CMAQ5.2. When I run the run_icon.csh and run_bcon.csh for the single day benchmark, the program get stuck with no specific error. Would you please help me with that problem? Last part of the run_icon.log is as follow:

/public/software/CMAQ5.2/new-bulid/PREP/icon/scripts/BLD_ICON_v52_profile_intel/ICON_v52_profile.exe

 >>--->> WARNING in subroutine NCPOPT
 notCDF call to NCPOPT

 
 >>--->> WARNING in subroutine NF_INQ_LIBVERS
 notCDF call to NF_INQ_LIBVERS
 
                                                                 
 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).  Copyright (C) 1992-2002 MCNC,  
 (C) 1992-2013 Carlie J. Coats, Jr., and (C) 2003-2012 Baron     
 Advanced Meteorological Systems, LLC and released under the     
 GNU LGPL  License, version 2.1.  See URL                        
                                                                 
     https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html     
                                                                 
 for conditions of use.                                          
                                                                 
 ioapi-3.1: $Id:: init3.F 321 2016-02-26 16:00:20Z coats                $
 Version with PARMS3.EXT/PARAMETER::MXVARS3= 2048
 netCDF version notCDF version 1.0
  
  
 EXECUTION_ID: ICON_v52_profile.exe
 Value for GRID_NAME:  'SE52BENCH'
 Value for GRID_NAME:  'SE52BENCH'

 File "GRIDDESC" opened for input on unit:  99
 /public/software/CMAQ5.2/new-bulid/data/SE52BENCH/single_day/cctm_input/met/mcip/GRIDDESC

 Value for NOTCDF_FATAL not defined;returning default:   TRUE

The output of bcon is similar.


#2

It looks like you have selected the “notCDF” version of netCDF, and the default value of the environment variable “NOTCDF_FATAL” is TRUE. That suggests to me the program is aborting, though there should be an error message indicating this.

I recommend you recompile IOAPI with a suitable version of the netCDF library.


#3

thank you for your reply! I have recompiled the IOAPI and now the problem of NOTCDF seems to have been solved. However, another problem emerged, and the program is still stuck. Last part of the run_icon.log is as follow:

 ioapi-3.2: $Id: init3.F90 98 2018-04-05 14:35:07Z coats $
 Version with PARMS3.EXT/PARAMETER::MXVARS3= 2048
 MPI/PnetCDF parallel-I/O enabled
 PnetCDF Library Version "1.10.0 of  2 Jul 2018"
 netCDF version 4.6.1 of Sep 15 2018 15:08:23 $

 EXECUTION_ID: ICON_v52_profile.exe
 Value for GRID_NAME:  'SE52BENCH'
 Value for GRID_NAME:  'SE52BENCH'

 File "GRIDDESC" opened for input on unit:  99
 /public/software/CMAQ5.2/new-bulid/data/SE52BENCH/single_day/cctm_input/met/mcip/GRIDDESC

 "MODPDATA:PN_SETUP":  Version
 $Id: modpdata.F90 1 2017-06-10 18:05:20Z coats $
 Value for NPCOL_NPROW not defined; returning defaultval ':  '                '
 Value for NPCOL_NPROW not defined; returning defaultval ':  '                '

I add setenv NPCOL_NPROW “1 1” to the run script and try again, then the program runs and exits, with no data in the output directory. Last part of the run_icon.log is as follow:

ioapi-3.2: Id: init3.F90 98 2018-04-05 14:35:07Z coats
Version with PARMS3.EXT/PARAMETER::MXVARS3= 2048
MPI/PnetCDF parallel-I/O enabled
PnetCDF Library Version “1.10.0 of 2 Jul 2018”
netCDF version 4.6.1 of Sep 15 2018 15:08:23 $

 EXECUTION_ID: ICON_v52_profile.exe
 Value for GRID_NAME:  'SE52BENCH'
 Value for GRID_NAME:  'SE52BENCH'

 File "GRIDDESC" opened for input on unit:  99
 /public/software/CMAQ5.2/new-bulid/data/SE52BENCH/single_day/cctm_input/met/mcip/GRIDDESC

 "MODPDATA:PN_SETUP":  Version
 $Id: modpdata.F90 1 2017-06-10 18:05:20Z coats $
 Value for NPCOL_NPROW:  '1 1'
 Value for NPCOL_NPROW:  '1 1'
 Processor-grid dimensions:  COLS=      1  ROWS=      1
 Value for GRID_NAME:  'SE52BENCH'
 Value for GRID_NAME:  'SE52BENCH'
 MODPDATA:PN_SETUP  Model-grid dimensions:  NCOLS=    100  NROWS=     80

Attempting to use an MPI routine before initializing MPI
0.021u 0.031s 0:00.07 71.4% 0+0k 0+0io 1pf+0w
exit ( )

Both ICON and BCON are single-processor program, but the libraries I compiled them with, such as netCDF, IOAPI and HDF5, are all parallel-enabled, and even ICON and BCON themselves are compiled using mpiifort to keep consistent with the libraries. Does this cause the problem? If this is indeed the cause of the problem, then I have to recompile all the libraries, from hdf5 to IOAPI, with no MPI, but what about the CCTM? Do I have to compile two kind of libraries, with and without MPI, and compile ICON and BCON with seriel libraries and CCTM with parallel ones?
Thank you in advance for your help!


#4

When you say netCDF is parallel-enabled, do you mean you are using pNetCDF and IOAPI compiled with pNetCDF.
I have not used that. I am not sure to what extent the CMAQ programs (ICON, BCON, CCTM) work with pNetCDF. I would think they are compatible, but relatively few people have done that.
To clarify: it is possible to run CMAQ in parallel using IOAPI and regular netCDF.

For a new user, I would recommend compiling a plain vanilla (“no coupling”) version of IOAPI. That can be challenging enough… pay special attention to the notes about compiling with netCDF4.
https://www.cmascenter.org/ioapi/documentation/all_versions/html/AVAIL.html#build


#5

Thank you very much! I did use PnetCDF. I have recompiled all the relative libraries and now ICON and BCON work normally. There are still some error in the compilation of CCTM, I’d continue to try it tomorror.
Thank you again!


#6

From the I/O API manual (https://www.cmascenter.org/ioapi/documentation/all_versions/html/AVAIL.html#build or https://cjcoats.github.io/ioapi/AVAIL.html#build):

  • I/O API 3.2 only: If you want to enable the PnetCDF distributed I/O configuration (for CMAQ), cd ioapi and then cp Makefile.pncf Makefile and setenv environment variable BIN to one of the supported distributed-I/O types Linux2_x86_64gfortmpi, Linux2_x86_64ifortmpi,Linux2_x86_64pgmpi, Linux2_x86_64sunmpi
    NOTES
  1. PnetCDF enabled I/O API libraries are not compatible with “normal” libraries. Make sure you keep them separate.
    You will want to build both, since the PnetCDF enabled libraries are for CMAQ only.
  2. What is called mpif90 is really a script that understands about an underlying compiler-set (e.g., Intel ifort/icc, PGI pgf90/pgcc, etc.), a set of (MPI-installation specific) INCLUDE files, and a set of $quot;.a" library-files. These are not compatible from MPI-implementation to MPI-implementation (OpenMPI, MVAPICH, etc.) nor from underlying compiler-set to underlying compiler-set .
  3. There may even be compatibility problems between different versions of the “same” implementation: This author has found that OpenMPI-1.4.x is not even source-code compatible with OpenMPI-1.6.x ;-(
  4. This part of the I/O API source code had to be hacked badly to deal with the above.

So you want separate MPI enabled libraries for CMAQ, and non-MPI libraries for everything else…

Carlie J. Coats, Jr., Ph.D. cjcoats@email.unc.edu
Senior Software Engineer I/O API Author/Maintainer
Center for Environmental Modeling for Policy Development,
UNC Institute for the Environment www.ie.unc.edu
100 Europa Dr., Suite 490 Rm 4051 / Campus Box 1105 919.843.5951
Chapel Hill, NC 27599-1105 Fax 919.966.9920