CMAQ-5.4 ISAM when I run ISAM, I meet a problem, please help me to solve it.when I set 32 regions to trace PM_TOT ,my progress stop.follow is my log

CMAQ-ISAM
1

when I run ISAM, I meet a problem, please help me to solve it.when I set 32 regions to trace PM_TOT ,my progress stop.follow is my log.The error is very strange.
Added an Accumulation Mode for Aerosol Species ASVPO2J
Added an Accumulation Mode for Aerosol Species ASVPO3J
Added an Accumulation Mode for Aerosol Species AIVPO1J
Added an Aitken Mode for Aerosol Species ALVOO1I
Added an Accumulation Mode for Aerosol Species ALVOO1J
Added an Aitken Mode for Aerosol Species ALVOO2I
Added an Accumulation Mode for Aerosol Species ALVOO2J
Added an Aitken Mode for Aerosol Species ASVOO1I
Added an Accumulation Mode for Aerosol Species ASVOO1J
Added an Aitken Mode for Aerosol Species ASVOO2I
Added an Accumulation Mode for Aerosol Species ASVOO2J
Added an Accumulation Mode for Aerosol Species ASVOO3J
Added an Accumulation Mode for Aerosol Species AAVB1J
Added an Accumulation Mode for Aerosol Species AAVB2J
Added an Accumulation Mode for Aerosol Species AAVB3J
Added an Accumulation Mode for Aerosol Species AAVB4J
Added an Accumulation Mode for Aerosol Species AMT1J
Added an Accumulation Mode for Aerosol Species AMT2J
Added an Accumulation Mode for Aerosol Species AMT3J
Added an Accumulation Mode for Aerosol Species AMT4J
Added an Accumulation Mode for Aerosol Species AMT5J
Added an Accumulation Mode for Aerosol Species AMT6J

 Total number of aerosol species:  84


 |> Map Aerosol Species:
 +========================


 Optional Species APHGJ Not found in AE namelist.


 ===========================================
 |>---   INITIALIZE PROCESS ANALYSIS   ---<|
 ===========================================
 PA_DATAGEN: Process Analysis is not activated


 ==================================
 |>---   CONFIGURE SCENARIO   ---<|
 ==================================

/public2/home/Wangyh/model/copy-CMAQ/CMAQ-5.4/CCTM/scripts/isam_control_cmaq54.
txt: Sucessfully Opened
Start counting the list…
Reading ISAM control file
TAGCLASSES = PM_IONS
for 32 tags
CMAQ Processing of Day 20200101 Began at 2023年 03月 26日 星期日 11:34:06 CST

    CTM_APPL  |  v54_ISAM_intel_D01_20200101

#================================================================================#
#| |#
#| The Community Multiscale Air Quality (CMAQ) Model |#
#| Version 5.4 |#
#| |#
#| Built and Maintained by the |#
#| Office of Research and Development |#
#| United States Environmental Protection Agency |#
#| |#
#| https://www.epa.gov/cmaq |#
#| |#
#| Source Code: https://www.github.com/USEPA/cmaq/tree/master |#
#| Documentation: CMAQ/DOCS at main · USEPA/CMAQ · GitHub |#
#| |#
#| The CMAQ Model is tested and released with cooperation from |#
#| the Community Modeling and Analysis System (CMAS) Center via |#
#| contract support. CMAS is managed by the Institute for the |#
#| Environment, University of North Carolina at Chapel Hill. |#
#| CMAS URL: (https://www.cmascenter.org) |#
#| |#
#================================================================================#

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).                                 
 Copyright (C) 1992-2002 MCNC,                                   
 (C) 1992-2018 Carlie J. Coats, Jr.,                             
 (C) 2003-2012 Baron Advanced Meteorological Systems, LLC, and   
 (C) 2014-2021 UNC Institute for the Environment.                
 Released under the GNU LGPL  License, version 2.1.  See URL     
                                                                 
     https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html     
                                                                 
 for conditions of use.                                          
                                                                 
 ioapi-3.2: $Id: init3.F90 200 2021-05-10 14:06:20Z coats $
 netCDF version 4.1.3 of Jul 27 2022 15:47:04 $
  

 ===========================================
 |>---   ENVIRONMENT VARIABLE REPORT   ---<|
 ===========================================

 |> Grid and High-Level Model Parameters:
 +=========================================
  --Env Variable-- | --Value--
  --------------------------------------------------------------------------------
         BLD  |             (default)
      OUTDIR  |  /public2/home/Wangyh/model/data/dataout/cctm/2020_01/54ISAT
   NEW_START  |           T

IGNORE_SOILINP | F (default)
EXECUTION_ID | CMAQ_CCTMv54_Wangyh_20230326_033406_555717372
GRID_NAME | 2020_01_D01
CTM_TSTEP | 10000
CTM_RUNLEN | 10000
CTM_PROGNAME | DRIVER (default)
CTM_STDATE | 2020001
CTM_STTIME | 0
NPCOL_NPROW | 4 6
CTM_MAXSYNC | 300
CTM_MINSYNC | 60

 |> Multiprocess control, output and error checking:
 +====================================================
  --Env Variable-- | --Value--
  --------------------------------------------------------------------------------

PRINT_PROC_TIME | T
FL_ERR_STOP | F
CTM_CKSUM | T
AVG_FILE_ENDTIME | F
AVG_CONC_SPCS | ALL
CONC_SPCS | O3 NO ANO3I ANO3J NO2 FORM ISOP NH3 ANH4I ANH4J ASO4I ASO4J
ACONC_BLEV_ELEV | 1 1
CONC_BLEV_ELEV | 1 1
IOAPI_LOG_WRITE | F
VERTEXT | F
VERTEXT_COORD_PA | /public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/lonlat.csv
gc_matrix_nml | /public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/GC_cb6r5_ae7_aq.nml
ae_matrix_nml | /public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/AE_cb6r5_ae7_aq.nml
nr_matrix_nml | /public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/NR_cb6r5_ae7_aq.nml
tr_matrix_nml | /public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/Species_Table_TR_0.nml

 |> ISAM setting and output options:
 +====================================
  --Env Variable-- | --Value--
  --------------------------------------------------------------------------------

ISAM_NEW_START | Y
AISAM_BLEV_ELEV | 1 1
ISAM_BLEV_ELEV | 1 1
ISAM_O3_WEIGHTS | 5
VOC_NOX_TRANS | 0.350E+00
ISAM_NOX_CASE | 2
ISAM_VOC_CASE | 4

 |> Chemistry and Photolysis:
 +=============================
  --Env Variable-- | --Value--
  --------------------------------------------------------------------------------
CTM_PHOTDIAG  |           F

CORE_SHELL_OPTIC | F (default)
OPTICS_MIE_CALC | F (default)
GEAR_ATOL | 0.100E-08 (default)
GEAR_RTOL | 0.100E-02 (default)
RB_RTOL | 0.100E-02 (default)
RB_ATOL | 0.100E-06 (default)

 |> Aerosols:
 +=============
  --Env Variable-- | --Value--
  --------------------------------------------------------------------------------

IC_AERO_M2WET | F
BC_AERO_M2WET | F
IC_AERO_M2USE | F
BC_AERO_M2USE | F
STM_SO4TRACK | F

 |> Cloud Processes:
 +====================
  --Env Variable-- | --Value--
  --------------------------------------------------------------------------------
    CLD_DIAG  |           F

 |> Air-Surface Exchange Processes:
 +===================================
  --Env Variable-- | --Value--
  --------------------------------------------------------------------------------
  PX_VERSION  |           T
  CTM_ABFLUX  |           F
  CTM_MOSAIC  |           F
CTM_SFC_HONO  |           T
 CLM_VERSION  |           F
NOAH_VERSION  |           F

CTM_DEPV_FILE | F
CTM_HGBIDI | F
CTM_BIDI_FERT_NH | F
CTM_WBDUST_BELD | UNKNOWN (default)
STAGECTRL_NML | /public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CMAQ_Control_STAGE.nml
CTM_STAGE_P22 | F
CTM_STAGE_E20 | T
CTM_STAGE_S22 | F

 |> Transport Processes:
 +========================
  --Env Variable-- | --Value--
  --------------------------------------------------------------------------------

CTM_VDIFF_DIAG_F | F (default)
SIGMA_SYNC_TOP | 0.700E+00
ADV_HDIV_LIM | 0.900E+00 (default)
CTM_ADV_CFL | 0.950E+00
KZMIN | T
CTM_WVEL | T
CTM_GRAV_SETL | T

 |> Emissions Parameters:
 +=========================
  --Env Variable-- | --Value--
  --------------------------------------------------------------------------------
  CTM_EMLAYS  |           0 (default)

MISC_CTRL_NML | /public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CMAQ_Control_Misc.nml
DESID_CTRL_NML | /public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CMAQ_Control_DESID.nml
DESID_CHEM_CTRL_ | /public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CMAQ_Control_DESID_cb6r5_ae7_aq.nml
N_EMIS_GR | 5
N_EMIS_TR | 0 (default)
CTM_EMISCHK | F
CTM_BIOGEMIS_BE | F
CTM_BIOGEMIS_MG | F
USE_MEGAN_LAI | F (default)
MGN_ONLN_DEP | F (default)
BDSNP_MEGAN | F
BIOG_SPRO | DEFAULT (default)
B3GTS_DIAG | F
CTM_MGEMDIAG | F (default)
CTM_OCEAN_CHEM | F
CTM_WB_DUST | F
CTM_DUSTEM_DIAG | F
CTM_SSEMDIAG | F
CTM_LTNG_NO | F
LTNG_ASSIM_DT | 0 (default)
LTNGNO | InLine (default)
USE_NLDN | F (default)
LTNGDIAG | F
MOLSNCG | 0.350E+03 (default)
MOLSNIC | 0.350E+03 (default)
N_EMIS_PT | 0
IPVERT | 0 (default)
EMIS_SYM_DATE | F (default)

 |> Process Analysis Parameters:
 +================================
  --Env Variable-- | --Value--
  --------------------------------------------------------------------------------
  CTM_PROCAN  |           F
PA_BCOL_ECOL  |             (default)
PA_BROW_EROW  |             (default)
PA_BLEV_ELEV  |             (default)
   MET_TSTEP  |       10000 (default)

 MET data determined based on WRF ARW version 4.1.2


      -=-  MPP Processor-to-Subdomain Map  -=-
             Number of Processors =  24
____________________________________________________
|                                                  |
|  PE    #Cols    Col_Range     #Rows    Row_Range |
|__________________________________________________|
|                                                  |
|  0       68      1:  68         36      1:  36   |
|  1       68     69: 136         36      1:  36   |
|  2       67    137: 203         36      1:  36   |
|  3       67    204: 270         36      1:  36   |
|  4       68      1:  68         36     37:  72   |
|  5       68     69: 136         36     37:  72   |
|  6       67    137: 203         36     37:  72   |
|  7       67    204: 270         36     37:  72   |
|  8       68      1:  68         36     73: 108   |
|  9       68     69: 136         36     73: 108   |
| 10       67    137: 203         36     73: 108   |
| 11       67    204: 270         36     73: 108   |
| 12       68      1:  68         36    109: 144   |
| 13       68     69: 136         36    109: 144   |
| 14       67    137: 203         36    109: 144   |
| 15       67    204: 270         36    109: 144   |
| 16       68      1:  68         36    145: 180   |
| 17       68     69: 136         36    145: 180   |
| 18       67    137: 203         36    145: 180   |
| 19       67    204: 270         36    145: 180   |
| 20       68      1:  68         36    181: 216   |
| 21       68     69: 136         36    181: 216   |
| 22       67    137: 203         36    181: 216   |
| 23       67    204: 270         36    181: 216   |
|__________________________________________________|

*** Error in `/public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CCTM_v54_ISAM.exe’: free(): invalid next size (fast): 0x0000000034c376f0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81499)[0x2b806d9d6499]
/public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CCTM_v54_ISAM.exe(for_deallocate+0xb9)[0xaca7c9]
/public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CCTM_v54_ISAM.exe[0x5b10b0]
/public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CCTM_v54_ISAM.exe[0x93a55b]
/public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CCTM_v54_ISAM.exe[0x712d4b]
/public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CCTM_v54_ISAM.exe[0x712652]
/public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CCTM_v54_ISAM.exe[0x40a5ec]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x2b806d977445]
/public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CCTM_v54_ISAM.exe[0x40a4e9]
======= Memory map: ========
00400000-00c3c000 r-xp 00000000 00:29 32243126555 /public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CCTM_v54_ISAM.exe
00e3c000-00e3f000 r–p 0083c000 00:29 32243126555 /public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CCTM_v54_ISAM.exe
00e3f000-0ef8d000 rw-p 0083f000 00:29 32243126555 /public2/home/Wangyh/model/copy-CMAQ/5.4/CMAQ-5.4/CCTM/scripts/BLD_CCTM_v54_ISAM_intel/CCTM_v54_ISAM.exe
0ef8d000-33999000 rw-p 00000000 00:00 0
3490a000-34d24000 rw-p 00000000 00:00 0 [heap]
2b806c04b000-2b806c06d000 r-xp 00000000 fd:00 33592093 /usr/lib64/ld-2.17.so
2b806c06d000-2b806c071000 rw-p 00000000 00:00 0
2b806c088000-2b806c1f0000 rw-p 00000000 00:00 0
2b806c26c000-2b806c26d000 r–p 00021000 fd:00 33592093 /usr/lib64/ld-2.17.so
2b806c26d000-2b806c26e000 rw-p 00022000 fd:00 33592093 /usr/lib64/ld-2.17.so
2b806c26e000-2b806c26f000 rw-p 00000000 00:00 0
2b806c26f000-2b806c2fa000 r-xp 00000000 00:29 51581004771 /public2/home/Wangyh/model/grib2/lib/libnetcdff.so.5.1.0
2b806c2fa000-2b806c4f9000 —p 0008b000 00:29 51581004771 /public2/home/Wangyh/model/grib2/lib/libnetcdff.so.5.1.0
2b806c4f9000-2b806c4fa000 r–p 0008a000 00:29 51581004771 /public2/home/Wangyh/model/grib2/lib/libnetcdff.so.5.1.0
2b806c4fa000-2b806c4fb000 rw-p 0008b000 00:29 51581004771 /public2/home/Wangyh/model/grib2/lib/libnetcdff.so.5.1.0
2b806c4fb000-2b806c6b2000 rw-p 00000000 00:00 0
2b806c6b2000-2b806c75a000 r-xp 00000000 00:29 51581004760 /public2/home/Wangyh/model/grib2/lib/libnetcdf.so.7.1.1
2b806c75a000-2b806c959000 —p 000a8000 00:29 51581004760 /public2/home/Wangyh/model/grib2/lib/libnetcdf.so.7.1.1
2b806c959000-2b806c9a7000 r–p 000a7000 00:29 51581004760 /public2/home/Wangyh/model/grib2/lib/libnetcdf.so.7.1.1
2b806c9a7000-2b806c9a8000 rw-p 000f5000 00:29 51581004760 /public2/home/Wangyh/model/grib2/lib/libnetcdf.so.7.1.1
2b806c9a8000-2b806c9aa000 rw-p 00000000 00:00 0
2b806c9aa000-2b806cc9b000 r-xp 00000000 00:29 51587631584 /public2/home/Wangyh/model/grib2/lib/libmpi.so.12.1.0
2b806cc9b000-2b806ce9b000 —p 002f1000 00:29 51587631584 /public2/home/Wangyh/model/grib2/lib/libmpi.so.12.1.0
2b806ce9b000-2b806ceab000 r–p 002f1000 00:29 51587631584 /public2/home/Wangyh/model/grib2/lib/libmpi.so.12.1.0
2b806ceab000-2b806ceb5000 rw-p 00301000 00:29 51587631584 /public2/home/Wangyh/model/grib2/lib/libmpi.so.12.1.0
2b806ceb5000-2b806ceef000 rw-p 00000000 00:00 0
2b806ceef000-2b806cf28000 r-xp 00000000 00:29 51587631588 /public2/home/Wangyh/model/grib2/lib/libmpifort.so.12.1.0
2b806cf28000-2b806d127000 —p 00039000 00:29 51587631588 /public2/home/Wangyh/model/grib2/lib/libmpifort.so.12.1.0
2b806d127000-2b806d128000 r–p 00038000 00:29 51587631588 /public2/home/Wangyh/model/grib2/lib/libmpifort.so.12.1.0
2b806d128000-2b806d129000 rw-p 00039000 00:29 51587631588 /public2/home/Wangyh/model/grib2/lib/libmpifort.so.12.1.0
2b806d129000-2b806d22a000 r-xp 00000000 fd:00 33592108 /usr/lib64/libm-2.17.so
2b806d22a000-2b806d429000 —p 00101000 fd:00 33592108 /usr/lib64/libm-2.17.so
2b806d429000-2b806d42a000 r–p 00100000 fd:00 33592108 /usr/lib64/libm-2.17.so
2b806d42a000-2b806d42b000 rw-p 00101000 fd:00 33592108 /usr/lib64/libm-2.17.so
2b806d42b000-2b806d50c000 r-xp 00000000 00:29 60137139224 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libiomp5.so
2b806d50c000-2b806d70c000 —p 000e1000 00:29 60137139224 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libiomp5.so
2b806d70c000-2b806d716000 rw-p 000e1000 00:29 60137139224 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libiomp5.so
2b806d716000-2b806d739000 rw-p 00000000 00:00 0
2b806d739000-2b806d750000 r-xp 00000000 fd:00 33592126 /usr/lib64/libpthread-2.17.so
2b806d750000-2b806d94f000 —p 00017000 fd:00 33592126 /usr/lib64/libpthread-2.17.so
2b806d94f000-2b806d950000 r–p 00016000 fd:00 33592126 /usr/lib64/libpthread-2.17.so
2b806d950000-2b806d951000 rw-p 00017000 fd:00 33592126 /usr/lib64/libpthread-2.17.so
2b806d951000-2b806d955000 rw-p 00000000 00:00 0
2b806d955000-2b806db18000 r-xp 00000000 fd:00 33592100 /usr/lib64/libc-2.17.so
2b806db18000-2b806dd17000 —p 001c3000 fd:00 33592100 /usr/lib64/libc-2.17.so
2b806dd17000-2b806dd1b000 r–p 001c2000 fd:00 33592100 /usr/lib64/libc-2.17.so
2b806dd1b000-2b806dd1d000 rw-p 001c6000 fd:00 33592100 /usr/lib64/libc-2.17.so
2b806dd1d000-2b806dd22000 rw-p 00000000 00:00 0
2b806dd22000-2b806dd37000 r-xp 00000000 fd:00 33554508 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b806dd37000-2b806df36000 —p 00015000 fd:00 33554508 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b806df36000-2b806df37000 r–p 00014000 fd:00 33554508 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b806df37000-2b806df38000 rw-p 00015000 fd:00 33554508 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
2b806df38000-2b806df3a000 r-xp 00000000 fd:00 33592106 /usr/lib64/libdl-2.17.so
2b806df3a000-2b806e13a000 —p 00002000 fd:00 33592106 /usr/lib64/libdl-2.17.so
2b806e13a000-2b806e13b000 r–p 00002000 fd:00 33592106 /usr/lib64/libdl-2.17.so
2b806e13b000-2b806e13c000 rw-p 00003000 fd:00 33592106 /usr/lib64/libdl-2.17.so
2b806e13c000-2b806e163000 r-xp 00000000 00:29 12446494 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libifport.so.5
2b806e163000-2b806e362000 —p 00027000 00:29 12446494 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libifport.so.5
2b806e362000-2b806e365000 rw-p 00026000 00:29 12446494 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libifport.so.5
2b806e365000-2b806e36b000 rw-p 00000000 00:00 0
2b806e36b000-2b806e489000 r-xp 00000000 00:29 12446486 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libifcore.so.5
2b806e489000-2b806e688000 —p 0011e000 00:29 12446486 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libifcore.so.5
2b806e688000-2b806e69a000 rw-p 0011d000 00:29 12446486 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libifcore.so.5
2b806e69a000-2b806e6a1000 rw-p 00000000 00:00 0
2b806e6a1000-2b806e91b000 r-xp 00000000 00:29 60137139221 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libimf.so
2b806e91b000-2b806eb1a000 —p 0027a000 00:29 60137139221 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libimf.so
2b806eb1a000-2b806eb5d000 rw-p 00279000 00:29 60137139221 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libimf.so
2b806eb5d000-2b806eba8000 r-xp 00000000 00:29 60137139223 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libintlc.so.5
2b806eba8000-2b806eda7000 —p 0004b000 00:29 60137139223 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libintlc.so.5
2b806eda7000-2b806edaa000 rw-p 0004a000 00:29 60137139223 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libintlc.so.5
2b806edaa000-2b806edab000 rw-p 00000000 00:00 0
2b806edab000-2b806f551000 r-xp 00000000 00:29 60137165249 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libsvml.so
2b806f551000-2b806f751000 —p 007a6000 00:29 60137165249 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libsvml.so
2b806f751000-2b806f776000 rw-p 007a6000 00:29 60137165249 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libsvml.so
2b806f776000-2b806f77b000 r-xp 00000000 00:29 60137139227 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libirng.so
2b806f77b000-2b806f97b000 —p 00005000 00:29 60137139227 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libirng.so
2b806f97b000-2b806f97d000 rw-p 00005000 00:29 60137139227 /public2/home/Wangyh/intel/composer_xe_2013.2.146/compiler/lib/intel64/libirng.so
2b806f97d000-2b806f984000 r-xp 00000000 fd:00 33600098 /usr/lib64/librt-2.17.so
2b806f984000-2b806fb83000 —p 00007000 fd:00 33600098 /usr/lib64/librt-2.17.so
2b806fb83000-2b806fb84000 r–p 00006000 fd:00 33600098 /usr/lib64/librt-2.17.so
2b806fb84000-2b806fb85000 rw-p 00007000 fd:00 33600098 /usr/lib64/librt-2.17.so
2b806fb85000-2b8077e10000 rw-s 00000000 00:13 14714737 /dev/shm/mpich_shar_tmpcZaVYY (deleted)
2b8077e10000-2b8077e1c000 r-xp 00000000 fd:00 33592118 /usr/lib64/libnss_files-2.17.so
2b8077e1c000-2b807801b000 —p 0000c000 fd:00 33592118 /usr/lib64/libnss_files-2.17.so
2b807801b000-2b807801c000 r–p 0000b000 fd:00 33592118 /usr/lib64/libnss_files-2.17.so
2b807801c000-2b807801d000 rw-p 0000c000 fd:00 33592118 /usr/lib64/libnss_files-2.17.so
2b807801d000-2b807d265000 rw-p 00000000 00:00 0
2b8080000000-2b8080021000 rw-p 00000000 00:00 0
2b8080021000-2b8084000000 —p 00000000 00:00 0
7ffccd4ca000-7ffccd53b000 rw-p 00000000 00:00 0 [stack]
7ffccd5f4000-7ffccd5f6000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 57696 RUNNING AT node06
= EXIT CODE: 134
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
real 9.57
user 197.19
sys 21.11

** Runscript Detected an Error: CGRID file was not written. **
** This indicates that CMAQ was interrupted or an issue **
** exists with writing output. The runscript will now **
** abort rather than proceeding to subsequent days. **

==================================
***** CMAQ TIMING REPORT *****

Start Day: 2020-01-01
End Day: 2020-01-01
Number of Simulation Days: 1
Domain Name: 2020_01_D01
Number of Grid Cells: 2041200 (ROW x COL x LAY)
Number of Layers: 35
Number of Processes: 24
All times are in seconds.

Num Day Wall Time
01 2020-01-01 9.57
Total Time = 9.57
Avg. Time = 9.57

2

This might be a library or another type of compilation error. Are you able to compile and run the benchmark case?

Sergey

3

I can run the benchmark case ,if I do not set 32 regions, only set 9 regions ,it can normally run. it is so strange.can you help me to solve this problem?

4

you can see the question I posted earlier, I meet the same problem, but reporting errors is different.

5

How many source are you tracking with you 32-region simulation? How many with the 9-region simulation?

It is possible that you need to use the “large” version of the ioapi library or there is some kind of hardware limitation. Search the forum on how to download and use the “large” ioapi library.

Sergey

6

I am tracking one source,32 regions ,and 49species，this simulations uses ioapi-Large. this error has nothing to do with IOAPI_Large.

7

The error messages you are getting look like the type of messages that come up when your libraries are incorrectly compiled and/or linked or inconsistent is some other way (for example in compiler used).

So, are you able to run the benchmark case using the ioapi “large” library? Or, to ask in another way, are you running the benchmark and the 9-region cases with the standard library and the 32-region case with the “large” library?

Sergey

8

I used the ioapi-Large library to run all the cases, including the benchmark case, 9-regions case, and 32-regions case. and CMAQ-5.3.2 can normally run. In addition, I used CMAQ5.3.2 to run the 32-regions case normally. So I don’t think the error has anything to do with the ioapi-Large library. Is it something wrong with the model itself.

9

I’m not sure what is going on with your simulation. I was able to run the benchmark case, but with a new control file. I tracked EGU emissions from 32 separate states and used the ‘OZONE’ tagclass. I got good results for the states that are within the domain boundaries and zeros for the ones outside as expected. Maybe trying something like for the next test and see if works for you as well.

Another test would be to compile and run in debug mode to see if the traceback information could be useful.

Finally, if you don’t track ‘ALL’ tagclasses, but something simple like ‘EC’ or even just ‘OZONE’ there is probably not too many variables to require the ‘large’ ioapi library with 32 named tags. You could try that if you want with the standard ioapi.

Sergey

1 Like
10

thank you very much. I’ll try the advice you gave me.