Hi everyone
Currently I’m trying to run CCTM using mpich-3.2, and I have this error in processors’ logs (CTM_LOG_001,… etc)
*** ERROR ABORT in subroutine AQCHEM on PE 001
99
Maximum AQCHEM total iterations exceeded
Does Anybody know how I could fix this issue?
Thanks
Edison
Hi Edison,
Unfortunately, this error could come from a number of sources. Could you please reply with more information about your CMAQ model version, simulation domain, chemical mechanism, etc? COuld you send a copy of your runscript and the full log file?
Best regards,
Ben Murphy
Hi Ben
Currently I’m using CMAQ 5.0.2 with cb05tucl_ae6_aq mechanism and compiled with gcc-gfortran. Here’s my runscript:
#! /bin/csh -f
# ====================== CCTMv5.0.2 Run Script ====================== #
# Usage: run.cctm >&! cctm_D502a.log & #
# The following environment variables must be set for this script to #
# execute properly: #
# setenv M3DATA = input/output data directory #
# To report problems or request help with this script/program: #
# http://www.cmascenter.org/html/help.html #
# =================================================================== #
#> Source the config.cmaq file to set the run environment
#source ../config.cmaq
# Set Input variables from config.cmaq
set M3HOME=/SIMCAB/transporte/BenchMark/CMAQ_v5.0.2
set M3MODEL=$M3HOME/models
set M3CCTM=$M3HOME/scripts/cctm
set M3DATA=$M3HOME/data
set system="`/bin/uname -i`"
set bld_os="`/bin/uname -s``/bin/uname -r | cut -d. -f1`"
set lib_basedir=$M3HOME/lib
set M3LIB=$M3HOME/lib/x86_64/gcc
set WORK=$M3HOME/scripts
set compiler=gcc
set EXEC_ID=${bld_os}_${system}${compiler}
setenv MPI_INC $M3LIB/mpich/include
# Set External Variables
set SDATE=2018-06-02
set initj=`date -d $SDATE +%j`
#export tdyj=`date -d 2018-01-01 +%j`
set i=0
set today=`date -d "$SDATE +${i}days" +%Y%m%d`
set tomorrow=`date -d "$today +1day" +%Y%m%d`
set yesterday=`date -d "$today -1day" +%Y%m%d`
set jdate=`date -d "$SDATE +${i}days" +%Y%j`
set gdate=`date -d "$SDATE +${i}days" +%Y%m%d`
#> Check that M3DATA is set:
if ( ! -e $M3DATA ) then
echo " $M3DATA path does not exist"
exit 1
endif
echo " "; echo " Input data path, M3DATA set to $M3DATA"; echo " "
set PROC = mpi #> serial or mpi
set APPL = D502a
set CFG = CMAQ-BENCHMARK
set MECH = cb05tucl_ae6_aq
set EXEC = CCTM_${APPL}_$EXEC_ID
#> horizontal domain decomposition
if ( $PROC == serial ) then
setenv NPCOL_NPROW "1 1"; set NPROCS = 1 # single processor setting
else
setenv NPCOL_NPROW "4 4"; set NPROCS = 16
endif
#> Set the working directory:
set BASE = $M3HOME/scripts/cctm
set BLD = ${BASE}/BLD_${APPL}
cd $BASE; date; cat $BASE/cfg.$CFG; echo " "; set echo
#> timestep run parameters
#Calling initial simulation date
set SDATEini = $SDATE
#> timestep run parameters
set STDATE = `date -d "$today" +%Y%j` # beginning date
set STDATEm1 = `date -d "$today -1day" +%Y%j`
set STTIME = 000000 # beginning GMT time (HHMMSS)
set NSTEPS = 240000 # time duration (HHMMSS) for this run
set TSTEP = 010000 # output time step interval (HHMMSS) #changed 010000 to 000000
set YEAR = `date -d "$today" +%Y`
set YR = `date -d "$today" +%y`
set MONTH = `date -d "$today" +%m`
set DAY = `date -d "$today" +%d`
set yesterday = $yesterday
set YMD = ${YEAR}${MONTH}${DAY}
# =====================================================================
# CCTM Configuration Options
# =====================================================================
#setenv LOGFILE $BASE/$APPL.log #> log file name; uncomment to write standard output to a log, otherwise write to screen
setenv GRIDDESC $M3DATA/mcip/GRIDDESC #> horizontal grid defn
setenv GRID_NAME CMAQBogota #> check GRIDDESC file for GRID_NAME options
#setenv CONC_SPCS "O3 NO ANO3I ANO3J NO2 FORM ISOP ANH4J ASO4I ASO4J" #> CONC file species; comment or set to "ALL" to write all species to CONC
#setenv CONC_BLEV_ELEV " 1 4" #> CONC file layer range; comment to write all layers to CONC
#setenv AVG_CONC_SPCS "O3 NO CO NO2 ASO4I ASO4J NH3" #> ACONC file species; comment or set to "ALL" to write all species to ACONC
setenv AVG_CONC_SPCS "ALL"
setenv CTM_EMLAYS 1
setenv ACONC_BLEV_ELEV " 1 1" #> ACONC file layer range; comment to write all layers to ACONC
#setenv ACONC_END_TIME Y #> override default beginning ACON timestamp [ default: N ]
setenv CTM_MAXSYNC 720 #> max sync time step (sec) [default: 720]
setenv CTM_MINSYNC 60 #> min sync time step (sec) [default: 60]
setenv CTM_CKSUM Y #> write cksum report [ default: Y ]
setenv CLD_DIAG N #> write cloud diagnostic file [ default: N ]
setenv CTM_AERDIAG N #> aerosol diagnostic file [ default: N ]
setenv CTM_PHOTDIAG N #> photolysis diagnostic file [ default: N ]
setenv CTM_SSEMDIAG N #> sea-salt emissions diagnostic file [ default: N ]
setenv CTM_WB_DUST N #> use inline windblown dust emissions [ default: Y ]
setenv CTM_ERODE_AGLAND N #> use agricultural activity for windblown dust [ default: N ]; ignore if CTM_WB_DUST = N
setenv CTM_DUSTEM_DIAG N #> windblown dust emissions diagnostic file [ default: N ]; ignore if CTM_WB_DUST = N
setenv CTM_LTNG_NO N #> turn on lightning NOx [ default: N ]
setenv CTM_WVEL N #> save derived vertical velocity component to conc file [ default: N ]
#setenv KZMIN Y #> use Min Kz option in edyintb [ default: Y ], otherwise revert to Kz0UT
setenv CTM_ILDEPV Y #> calculate in-line deposition velocities [ default: Y ]
setenv CTM_MOSAIC N #> landuse specific deposition velocities [ default: N ]
setenv CTM_ABFLUX N #> Ammonia bi-directional flux for in-line deposition velocities [ default: N ]; ignore if CTM_ILDEPV = N
setenv CTM_HGBIDI N #> Mercury bi-directional flux for in-line deposition velocities [ default: N ]; ignore if CTM_ILDEPV = N
setenv CTM_SFC_HONO Y #> Surface HONO interaction [ default: Y ]; ignore if CTM_ILDEPV = N
setenv CTM_DEPV_FILE N #> write diagnostic file for deposition velocities [ default: N ]
setenv CTM_BIOGEMIS N #> calculate in-line biogenic emissions [ default: N ]
#setenv B3GTS_DIAG Y #> write biogenic mass emissions diagnostic file [ default: N ]; ignore if CTM_BIOGEMIS = N
setenv CTM_PT3DEMIS N #> calculate in-line plume rise for elevated point emissions [ default: N ]
#setenv PT3DDIAG Y #> optional 3d point source emissions diagnostic file [ default: N]; ignore if CTM_PT3DEMIS = N
#setenv PT3DFRAC N #> optional layer fractions diagnostic (play) file(s) [ default: N]; ignore if CTM_PT3DEMIS = N
setenv IOAPI_LOG_WRITE F #> turn on excess WRITE3 logging [ options: T | F ]
setenv FL_ERR_STOP N #> stop on inconsistent input files
setenv PROMPTFLAG F #> turn on I/O-API PROMPT*FILE interactive mode [ options: T | F ]
setenv IOAPI_OFFSET_64 NO #> support large timestep records (>2GB/timestep record) [ options: YES | NO ]
setenv EXECUTION_ID $EXEC #> define the model execution id
set DISP = delete #> [ delete | update | keep ] existing output files
# =====================================================================
#> Input/Output Directories
# =====================================================================
set ICpath = $M3DATA/icbc/icon_geos/d01 #> initial conditions input directory
set BCpath = $M3DATA/icbc/bcon_geos/d01 #> boundary conditions input directory
set EMISpath = $M3DATA/emis #> surface emissions input directory
set IN_PTpath = $M3DATA/emis #> elevated emissions input directory (in-line point only)
set IN_LTpath = $M3DATA/lightning #> lightning NOx input directory
set METpath = $M3DATA/mcip #> meteorology input directory
set JVALpath = $M3DATA/jproc #> offline photolysis rate table directory
set OMIpath = $M3DATA/raw/phot #> ozone columne data for the photolysis model
set LUpath = $M3DATA/dust #> BELD landuse data for windblown dust model
set SZpath = $M3DATA/ocean #> Surf zone file for in-line seasalt emissions
set OUTDIR = $M3DATA/cctm_out #> output file directory
# =====================================================================
#> Input Files
# =====================================================================
#> Initial conditions
set ICFILE = ts${YEAR}${MONTH}${DAY}.ICON.d01.nc
#> Boundary conditions
set BCFILE = ts.proncat.BCON.d01.nc
#> Off-line photolysis rates
set JVALfile = JTABLE_${STDATE}
#> Ozone column data
set OMIfile = OMI.dat
#> MCIP meteorology files
set EXTN = WRFd01_${YEAR}_${MONTH}_${DAY}
setenv GRID_DOT_2D $METpath/GRIDDOT2D_${EXTN}
setenv GRID_CRO_2D $METpath/GRIDCRO2D_${EXTN}
setenv MET_CRO_2D $METpath/METCRO2D_${EXTN}
setenv MET_CRO_3D $METpath/METCRO3D_${EXTN}
setenv MET_DOT_3D $METpath/METDOT3D_${EXTN}
setenv MET_BDY_3D $METpath/METBDY3D_${EXTN}
#> Emissions files
setenv EMISDATE ${YEAR}${MONTH}${DAY}
if ( $CTM_PT3DEMIS == 'N' ) then
set EMISfile = emis-d01-CMAQ-MERGED-${STDATE}-mol-diurnal.nc #> Offline 3d emissions file name
else
#> In-line emissions configuration
set CASE1 = 12US1_C25_2006am
set CASE2 = 12US1_cmaq_cb05_tx_C25_2006am
setenv CASE ${EMISDATE}_$CASE2
set EMISfile = emis_mole_all_${CASE}.ncf #> Surface emissions
setenv NPTGRPS 5 #> Number of elevated source groups
setenv STK_GRPS_01 $IN_PTpath/stack_groups_ptnonipm_${CASE1}.ncf
setenv STK_GRPS_02 $IN_PTpath/stack_groups_ptipm_${CASE1}.ncf
setenv STK_GRPS_03 $IN_PTpath/stack_groups_othpt_${CASE1}.ncf
setenv STK_GRPS_04 $IN_PTpath/stack_groups_seca_c3_${CASE1}.ncf
setenv STK_GRPS_05 $IN_PTpath/stack_groups_ptfire_${EMISDATE}_${CASE1}.ncf
setenv LAYP_STTIME $STTIME
setenv LAYP_NSTEPS $NSTEPS
setenv STK_EMIS_01 $IN_PTpath/inln_mole_ptnonipm_${CASE}.ncf
setenv STK_EMIS_02 $IN_PTpath/inln_mole_ptipm_${CASE}.ncf
setenv STK_EMIS_03 $IN_PTpath/inln_mole_othpt_${CASE}.ncf
setenv STK_EMIS_04 $IN_PTpath/inln_mole_seca_c3_${CASE}.ncf
setenv STK_EMIS_05 $IN_PTpath/inln_mole_ptfire_${CASE}.ncf
setenv LAYP_STDATE $STDATE
endif
#> Lightning NOx configuration
if ( $CTM_LTNG_NO == 'Y' ) then
setenv LTNGNO $IN_LTpath/nox_CMAQ-BENCHMARK.35L.$EMISDATE #> offline calculated lightning NOx
# setenv LTNGNO "InLine" #> set LTNGNO to "Inline" to activate in-line calculation
#> In-line lightning NOx options
setenv LTNGPARAM N #> use lightning parameter file? [ default: Y ]
setenv LTNGPARM_FILE $M3DATA/params/LTNG_RATIO.2004.$MONTH.ioapi #> lightning parameter file; ignore if LTNGPARAM = N
setenv LTNGDIAG N #> write lightning diagnostic file [ default: N ]
setenv LTNGOUT $OUTDIR/$EXEC.LTNGDIAG.${CFG}_${EMISDATE} #> lightning diagnostic file; ignore if LTNGDIAG = N
endif
#> In-line biogenic emissions configuration
if ( $CTM_BIOGEMIS == 'Y' ) then
set GSPROpath = ${M3DATA}/emis
setenv GSPRO $GSPROpath/gspro_cb05soa_notoxics_cmaq_poc_09nov2007.txt
set IN_BEISpath = ${M3DATA}/emis
setenv B3GRD $IN_BEISpath/b3grd_CMAQ-BENCHMARK_C70_2006am_Fulltox.ncf
setenv BIOG_SPRO B10C5 # speciation profile to use for biogenics
setenv BIOSW_YN N # use frost date switch [ default: Y ]
setenv BIOSEASON $IN_BEISpath/bioseason.cmaq.2002_02b_CMAQ-BENCHMARK_v31.ncf #> ignore season switch file if BIOSW_YN = N
setenv SUMMER_YN Y # Use summer normalized emissions? [ default: Y ]
setenv PX_VERSION N # MCIP is PX version? [ default: N ]
setenv B3GTS_DIAG Y #> beis mass emissions diagnostic file [ default: N ]
setenv INITIAL_RUN Y # non-existent or not using SOILINP [ default: N ]; default uses SOILINP
setenv SOILINP $OUTDIR/$EXEC.SOILINP.${CFG}_${EMISDATE} # Biogenic NO soil input file; ignore if INITIAL_RUN = Y
endif
#> Windblown dust emissions configuration
if ( $CTM_WB_DUST == 'Y' ) then
setenv DUST_LU_1 $LUpath/beld3_CMAQ-BENCHMARK_output_a.ncf
setenv DUST_LU_2 $LUpath/beld3_CMAQ-BENCHMARK_output_tot.ncf
if ( $CTM_ERODE_AGLAND == 'Y' ) then
setenv CROPMAP01 ${M3DATA}/crop/BeginPlanting_12km_CMAQ-BENCHMARK
setenv CROPMAP04 ${M3DATA}/crop/EndPlanting_12km_CMAQ-BENCHMARK
setenv CROPMAP08 ${M3DATA}/crop/EndHarvesting_12km_CMAQ-BENCHMARK
endif
endif
#> In-line sea salt emisisions configuration
setenv OCEAN_1 ${EMISpath}/CMAQ-BOGOTA_d01_surf_${STDATE}.nc #> horizontal grid-dependent surf zone file
# =====================================================================
#> Output Files
# =====================================================================
#> set output file name extensions
setenv CTM_APPL ${CFG}_${YMD}
#> set output file names
set CONCfile = $EXEC.CONC.${CTM_APPL} # CTM_CONC_1
set ACONCfile = $EXEC.ACONC.${CTM_APPL} # CTM_ACONC_1
set CGRIDfile = $EXEC.CGRID.${CTM_APPL} # CTM_CGRID_1
set DD1file = $EXEC.DRYDEP.${CTM_APPL} # CTM_DRY_DEP_1
set DV1file = $EXEC.DEPV.${CTM_APPL} # CTM_DEPV_DIAG
set PT1file = $EXEC.PT3D.${CTM_APPL} # CTM_PT3D_DIAG
set BIO1file = $EXEC.B3GTS_S.${CTM_APPL} # B3GTS_S
set SOIL1file = $EXEC.SOILOUT.${CTM_APPL} # SOILOUT
set WD1file = $EXEC.WETDEP1.${CTM_APPL} # CTM_WET_DEP_1
set WD2file = $EXEC.WETDEP2.${CTM_APPL} # CTM_WET_DEP_2
set AV1file = $EXEC.AEROVIS.${CTM_APPL} # CTM_VIS_1
set AD1file = $EXEC.AERODIAM.${CTM_APPL} # CTM_DIAM_1
set RJ1file = $EXEC.PHOTDIAG1.${CTM_APPL} # CTM_RJ_2
set RJ2file = $EXEC.PHOTDIAG2.${CTM_APPL} # CTM_RJ_2
set SSEfile = $EXEC.SSEMIS.$CTM_APPL # CTM_SSEMIS_1
set DSEfile = $EXEC.DUSTEMIS.$CTM_APPL # CTM_DUST_EMIS_1
set PA1file = $EXEC.PA_1.${CTM_APPL} # CTM_IPR_1
set PA2file = $EXEC.PA_2.${CTM_APPL} # CTM_IPR_2
set PA3file = $EXEC.PA_3.${CTM_APPL} # CTM_IPR_3
set IRR1file = $EXEC.IRR_1.${CTM_APPL} # CTM_IRR_1
set IRR2file = $EXEC.IRR_2.${CTM_APPL} # CTM_IRR_2
set IRR3file = $EXEC.IRR_3.${CTM_APPL} # CTM_IRR_3
set DEPVFSTfile = $EXEC.DEPVFST.${CTM_APPL} # CTM_DEPV_FST
set DEPVMOSfile = $EXEC.DEPVMOS.${CTM_APPL} # CTM_DEPV_MOS
set DDFSTfile = $EXEC.DDFST.${CTM_APPL} # CTM_DRY_DEP_FST
set DDMOSfile = $EXEC.DDMOS.${CTM_APPL} # CTM_DRY_DEP_MOS
#> In-line biogenic emissions output files
if ( $CTM_BIOGEMIS == 'Y' ) then
setenv B3GTS_S $OUTDIR/$EXEC".B3GTS_S".${CTM_APPL}
setenv SOILOUT $OUTDIR/$EXEC".SOILOUT".${CTM_APPL} # Biogenic NO soil output file
endif
#> set floor file (neg concs)
setenv FLOOR_FILE $BASE/FLOOR_${CTM_APPL}
#> create output directory
if ( ! -d "$OUTDIR" ) mkdir -p $OUTDIR
#> look for existing log files
set test = `ls CTM_LOG_???.${CTM_APPL}`
if ( "$test" != "" ) then
if ( $DISP == 'delete' ) then
echo " ancillary log files being deleted"
foreach file ( $test )
echo " deleting $file"
rm $file
end
else
echo "*** Logs exist - run ABORTED ***"
exit 1
endif
endif
#> for the run control ...
setenv CTM_STDATE $STDATE
setenv CTM_STTIME $STTIME
setenv CTM_RUNLEN $NSTEPS
setenv CTM_TSTEP $TSTEP
setenv EMIS_1 $EMISpath/$EMISfile
setenv INIT_GASC_1 $ICpath/$ICFILE
setenv INIT_AERO_1 $INIT_GASC_1
setenv INIT_NONR_1 $INIT_GASC_1
setenv INIT_TRAC_1 $INIT_GASC_1
setenv BNDY_GASC_1 $BCpath/$BCFILE
setenv BNDY_AERO_1 $BNDY_GASC_1
setenv BNDY_NONR_1 $BNDY_GASC_1
setenv BNDY_TRAC_1 $BNDY_GASC_1
setenv OMI $OMIpath/$OMIfile
setenv XJ_DATA $JVALpath/$JVALfile
set TR_DVpath = $METpath
set TR_DVfile = $MET_CRO_2D
#> species defn & photolysis
setenv gc_matrix_nml ${BLD}/GC_$MECH.nml
setenv ae_matrix_nml ${BLD}/AE_$MECH.nml
setenv nr_matrix_nml ${BLD}/NR_$MECH.nml
setenv tr_matrix_nml ${BLD}/Species_Table_TR_0.nml
#> check for photolysis input data
setenv CSQY_DATA ${BLD}/CSQY_DATA_$MECH
if (! (-e $CSQY_DATA ) ) then
echo " $CSQY_DATA not found "
exit 1
endif
#>- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
source $BASE/outck.q
ls -l $BLD/$EXEC; size $BLD/$EXEC
unlimit
#limit
#> Executable call for single PE, uncomment to invoke
# /usr/bin/time $BLD/$EXEC
#> Executable call for multi PE, configure for your system
set MPI = /usr/bin
set MPIRUN = $MPI/mpirun
time $MPIRUN -np $NPROCS $BLD/$EXEC
date
exit
Thanks for your help
Edison
Hi Edison,
Did you figure out the issues about this error? I’m having the same one, and could not fixed.
Yasmin,
As alluded to above, this could be several different things. Look in your log files for NaN or Infinity. If you still cannot determine what is wrong, please start a new thread.
State what version of the model you are running. Are you getting this error with the benchmark case, or is your own particular case? When does the error occur relative to the start of the model run? Is it essentially immediate, or after some number of hours or days?
Hi, Have you figured out the issue? I am facing the same problems but not able to solve it. I appreciate your advise.
Thank you
Jie
Hi Mr. Nolte,
I figured out that issue, the problem was in the emissions file, where there were some concentrations extremely high (which were wrong due to a problem relate to SMOKE processor and temporal profiles).
The version I have been working is the 5.2.1, and I have run my own case.
Hi Yasmin, Thank you for your answer, In my case, bcon and icon were the problem, but, in another answer in this, this problem could occur in mcip data too.
What is the problem with your icon and bcon? How did you solve it?