Could not write ROOH to CTM_CONC_1

CMAQ Run Time Errors and Issues
1

I have been trying to run CMAQv5.3.1 (cb6r3_aq, LTNG off, NH3 bidirectional flow off, WBDUST off) 36km simulation for a few days (2014-01-02 to 2014-01-05), but in my three attempts, I have had CMAQ exit either during the very first timestamp (2014-01-02_00:00:00) or the second day’s beginning (2014-01-3_00:00:00) with this error:
image
It always stops with not being able to write ROOH concentration. Above error is only present in the CTM_LOG_* files of the first few cpus of the second node, third node, fourth node and so on (eg PE 032, 033, 064, 065, 096, 097) when I use 4 nodes with 32 cpus each. Longer part of the error in the CTM_LOG_* file is here:

 No IC found for species SULF_ICBC in INIT_CONC_1; set to  1.00E-30

 =================================================
 |>---   OPEN OR CREATE CONCENTRATION FILE   ---<|
 =================================================
 CTM_CONC_1      :/nas/longleaf/home/surendra/scratch/cmaq_runs/cmaq_run_36k_13jul2020/cmaq_out/CCTM_CONC_v531_intel_2016_CONUS_20140102.nc

 >>--->> WARNING in subroutine OPEN3
 File not available.
 Could not open CTM_CONC_1 for update - try to open new

 Conc File Header Description:
    Concentration file output
    From CMAQ model dyn alloc version CTM
    Set of variables (possibly) reduced from CGRID
    For next scenario continuation runs,
    use the "one-step" CGRID file
    Layer  1 to  1
    Layer  2 to  2
    Layer  3 to  3
    Layer  4 to  4
    Layer  5 to  5
    Layer  6 to  6
    Layer  7 to  7
    Layer  8 to  8
    Layer  9 to  9
    Layer 10 to 10
    Layer 11 to 11
    Layer 12 to 12
    Layer 13 to 13
    Layer 14 to 14
    Layer 15 to 15
    Layer 16 to 16
    Layer 17 to 17
    Layer 18 to 18
    Layer 19 to 19
    Layer 20 to 20
    Layer 21 to 21
    Layer 22 to 22
    Layer 23 to 23
    Layer 24 to 24
    Layer 25 to 25
    Layer 26 to 26
    Layer 27 to 27
    Layer 28 to 28
    Layer 29 to 29
    Layer 30 to 30
    Layer 31 to 31
    Layer 32 to 32
    Layer 33 to 33
    Layer 34 to 34


 Gas Chem species saved to CONC file:
    1 (  1) NO2
    2 (  2) NO
    3 (  3) O
    4 (  4) O3
    5 (  5) NO3
    6 (  6) O1D
    7 (  7) OH
    8 (  8) HO2
    9 (  9) H2O2
   10 ( 10) N2O5
   11 ( 11) HNO3
   12 ( 12) HONO
   13 ( 13) PNA
   14 ( 14) SO2
   15 ( 15) SULF
   16 ( 16) C2O3
   17 ( 17) MEO2
   18 ( 18) RO2
   19 ( 19) PAN
   20 ( 20) PACD
   21 ( 21) AACD
   22 ( 22) CXO3
   23 ( 23) ALD2
   24 ( 24) XO2H
   25 ( 25) PANX
   26 ( 26) FORM
   27 ( 27) MEPX
   28 ( 28) MEOH


 *** ERROR ABORT in subroutine OPCONC on PE 096
 Could not write ROOH to CTM_CONC_1
 PM3EXIT:  DTBUF 0:00:00   Jan. 2, 2014
     Date and time 0:00:00   Jan. 2, 2014   (2014002:000000)

CTM_LOG_052_v531_intel_2016_CONUS_20140103.txt (36.8 KB)

2

Did you try removing all existing output files, and/or allow the output files to be overwritten in your run script?
#> Keep or Delete Existing Output Files
set CLOBBER_DATA = TRUE