CCTM crashed when saving CGRID file

I was running CMAQ v5.2.1 (It was working well for a previous project, so the model should be ok). When running it for the EPA 12US2 domain, CCTM successfully generated all output files but crashed while saving the restart (CGRID) file. Here is the log message:

 State CGRID File Header Description:
 => Computational grid instantaneous concentrations
 => - for scenario continuation.

    State CGRID File Variable List:
 => VNAME3D(  1 ): NO2
 => VNAME3D(  2 ): NO
 => VNAME3D(  3 ): O
 => VNAME3D(  4 ): O3
 => VNAME3D(  5 ): NO3
 => VNAME3D(  6 ): O1D
 => VNAME3D(  7 ): OH
 => VNAME3D(  8 ): HO2
 => VNAME3D(  9 ): H2O2
 => VNAME3D( 10 ): N2O5
 => VNAME3D( 11 ): HNO3
 => VNAME3D( 12 ): HONO
 => VNAME3D( 13 ): PNA
 => VNAME3D( 14 ): SO2
 => VNAME3D( 15 ): SULF
 => VNAME3D( 16 ): SULRXN
 => VNAME3D( 17 ): C2O3
 => VNAME3D( 18 ): MEO2
 => VNAME3D( 19 ): RO2
=> VNAME3D( 20 ): PAN
 => VNAME3D( 21 ): PACD
 => VNAME3D( 22 ): AACD
 => VNAME3D( 23 ): CXO3
 => VNAME3D( 24 ): ALD2
 => VNAME3D( 25 ): XO2H
 => VNAME3D( 26 ): PANX
 => VNAME3D( 27 ): FORM
 => VNAME3D( 28 ): MEPX
 => VNAME3D( 29 ): MEOH
 => VNAME3D( 30 ): ROOH
 => VNAME3D( 31 ): XO2
 => VNAME3D( 32 ): XO2N
 => VNAME3D( 33 ): XPAR
 => VNAME3D( 34 ): XPRP
 => VNAME3D( 35 ): NTR1
 => VNAME3D( 36 ): NTR2
 => VNAME3D( 37 ): FACD
 => VNAME3D( 38 ): CO
 => VNAME3D( 39 ): HCO3
 => VNAME3D( 40 ): ALDX
 => VNAME3D( 41 ): GLYD
 => VNAME3D( 42 ): GLY              
 => VNAME3D( 43 ): MGLY
 => VNAME3D( 44 ): ETHA
 => VNAME3D( 45 ): ETOH
 => VNAME3D( 46 ): KET
 => VNAME3D( 47 ): PAR                  
 => VNAME3D( 48 ): ACET
 => VNAME3D( 49 ): PRPA         
 => VNAME3D( 50 ): ROR
 => VNAME3D( 51 ): ETHY            
 => VNAME3D( 52 ): ETH
 => VNAME3D( 53 ): OLE
 => VNAME3D( 54 ): IOLE
 => VNAME3D( 55 ): ISOP
 => VNAME3D( 56 ): ISO2
 => VNAME3D( 57 ): ISOPRXN
 => VNAME3D( 58 ): ISPD
 => VNAME3D( 59 ): INTR
 => VNAME3D( 60 ): ISPX
 => VNAME3D( 61 ): HPLD
 => VNAME3D( 62 ): OPO3
 => VNAME3D( 63 ): EPOX
 => VNAME3D( 64 ): EPX2
 => VNAME3D( 65 ): TERP
 => VNAME3D( 66 ): TRPRXN
 => VNAME3D( 67 ): BENZENE
 => VNAME3D( 68 ): CRES
 => VNAME3D( 69 ): BZO2
 => VNAME3D( 70 ): OPEN
=> VNAME3D( 71 ): BENZRO2
 => VNAME3D( 72 ): TOL 
 => VNAME3D( 73 ): TO2 
 => VNAME3D( 74 ): TOLRO2
 => VNAME3D( 75 ): XOPN
 => VNAME3D( 76 ): XYLMN
 => VNAME3D( 77 ): XLO2
 => VNAME3D( 78 ): XYLRO2
 => VNAME3D( 79 ): NAPH
 => VNAME3D( 80 ): PAHRO2
 => VNAME3D( 81 ): CRO 
 => VNAME3D( 82 ): CAT1
 => VNAME3D( 83 ): CRON
 => VNAME3D( 84 ): OPAN
 => VNAME3D( 85 ): ECH4
 => VNAME3D( 86 ): CL2 
 => VNAME3D( 87 ): CL  
 => VNAME3D( 88 ): HOCL
 => VNAME3D( 89 ): CLO
 => VNAME3D( 90 ): FMCL
 => VNAME3D( 91 ): HCL 
 => VNAME3D( 92 ): CLNO2
 => VNAME3D( 93 ): TOLNRXN          
 => VNAME3D( 94 ): TOLHRXN
 => VNAME3D( 95 ): XYLNRXN
 => VNAME3D( 96 ): XYLHRXN
 => VNAME3D( 97 ): BNZNRXN
 => VNAME3D( 98 ): BNZHRXN              
 => VNAME3D( 99 ): SESQ
 => VNAME3D(100 ): SESQRXN      
 => VNAME3D(101 ): PAHNRXN
 => VNAME3D(102 ): PAHHRXN         
 => VNAME3D(103 ): SOAALK
 => VNAME3D(104 ): ALKRXN
 => VNAME3D(105 ): H2NO3PIJ
 => VNAME3D(106 ): H2NO3PK
 => VNAME3D(107 ): VLVPO1
 => VNAME3D(108 ): VSVPO1 
 => VNAME3D(109 ): VSVPO2
 => VNAME3D(110 ): VSVPO3
 => VNAME3D(111 ): VIVPO1
 => VNAME3D(112 ): VLVOO1
 => VNAME3D(113 ): VLVOO2
 => VNAME3D(114 ): VSVOO1
 => VNAME3D(115 ): VSVOO2
 => VNAME3D(116 ): VSVOO3
 => VNAME3D(117 ): PCVOC 
 => VNAME3D(118 ): PCSOARXN
 => VNAME3D(119 ): FORM_PRIMARY
 => VNAME3D(120 ): ALD2_PRIMARY
 => VNAME3D(121 ): BUTADIENE13

=> VNAME3D(122 ): ACROLEIN
=> VNAME3D(123 ): ACRO_PRIMARY
=> VNAME3D(124 ): TOLU
=> VNAME3D(125 ): HG
=> VNAME3D(126 ): HGIIAER
=> VNAME3D(127 ): HGIIGAS
=> VNAME3D(128 ): RHOJ
=> VNAME3D(129 ): ASO4J
=> VNAME3D(130 ): ASO4I
=> VNAME3D(131 ): ANH4J
=> VNAME3D(132 ): ANH4I
=> VNAME3D(133 ): ANO3J
=> VNAME3D(134 ): ANO3I
=> VNAME3D(135 ): AALK1J
=> VNAME3D(136 ): AALK2J
=> VNAME3D(137 ): AXYL1J
=> VNAME3D(138 ): AXYL2J
=> VNAME3D(139 ): AXYL3J
=> VNAME3D(140 ): ATOL1J
=> VNAME3D(141 ): ATOL2J
=> VNAME3D(142 ): ATOL3J
=> VNAME3D(143 ): ABNZ1J
=> VNAME3D(144 ): ABNZ2J
=> VNAME3D(145 ): ABNZ3J
=> VNAME3D(146 ): APAH1J
=> VNAME3D(147 ): APAH2J
=> VNAME3D(148 ): APAH3J
=> VNAME3D(149 ): ATRP1J
=> VNAME3D(150 ): ATRP2J
=> VNAME3D(151 ): AISO1J
=> VNAME3D(152 ): AISO2J
=> VNAME3D(153 ): ASQTJ
=> VNAME3D(154 ): AORGCJ
=> VNAME3D(155 ): AECJ
=> VNAME3D(156 ): AECI
=> VNAME3D(157 ): AOTHRJ
=> VNAME3D(158 ): AOTHRI
=> VNAME3D(159 ): AFEJ
=> VNAME3D(160 ): AALJ
=> VNAME3D(161 ): ASIJ
=> VNAME3D(162 ): ATIJ
=> VNAME3D(163 ): ACAJ
=> VNAME3D(164 ): AMGJ
=> VNAME3D(165 ): AKJ
=> VNAME3D(166 ): AMNJ
=> VNAME3D(167 ): ACORS
=> VNAME3D(168 ): ASOIL
=> VNAME3D(169 ): NUMATKN
=> VNAME3D(170 ): NUMACC
=> VNAME3D(171 ): NUMCOR
=> VNAME3D(172 ): SRFATKN
=> VNAME3D(173 ): SRFACC
=> VNAME3D(174 ): SRFCOR
=> VNAME3D(175 ): AH2OJ
=> VNAME3D(176 ): AH2OI
=> VNAME3D(177 ): AH3OPJ
=> VNAME3D(178 ): AH3OPI
=> VNAME3D(179 ): ANAJ
=> VNAME3D(180 ): ANAI
=> VNAME3D(181 ): ACLJ
=> VNAME3D(182 ): ACLI
=> VNAME3D(183 ): ASEACAT
=> VNAME3D(184 ): ACLK
=> VNAME3D(185 ): ASO4K
=> VNAME3D(186 ): ANH4K
=> VNAME3D(187 ): ANO3K
=> VNAME3D(188 ): AH2OK
=> VNAME3D(189 ): AH3OPK
=> VNAME3D(190 ): AISO3J
=> VNAME3D(191 ): AOLGAJ
=> VNAME3D(192 ): AOLGBJ
=> VNAME3D(193 ): AGLYJ
=> VNAME3D(194 ): APCSOJ
=> VNAME3D(195 ): ALVPO1I
=> VNAME3D(196 ): ASVPO1I
=> VNAME3D(197 ): ASVPO2I
=> VNAME3D(198 ): ALVPO1J
=> VNAME3D(199 ): ASVPO1J
=> VNAME3D(200 ): ASVPO2J
=> VNAME3D(201 ): ASVPO3J
=> VNAME3D(202 ): AIVPO1J
=> VNAME3D(203 ): ALVOO1I
=> VNAME3D(204 ): ALVOO2I
=> VNAME3D(205 ): ASVOO1I
=> VNAME3D(206 ): ASVOO2I
=> VNAME3D(207 ): ALVOO1J
=> VNAME3D(208 ): ALVOO2J
=> VNAME3D(209 ): ASVOO1J
=> VNAME3D(210 ): ASVOO2J
=> VNAME3D(211 ): ASVOO3J
=> VNAME3D(212 ): NH3
=> VNAME3D(213 ): SV_ALK1
=> VNAME3D(214 ): SV_ALK2
=> VNAME3D(215 ): SV_XYL1
=> VNAME3D(216 ): SV_XYL2
=> VNAME3D(217 ): SV_TOL1
=> VNAME3D(218 ): SV_TOL2
=> VNAME3D(219 ): SV_BNZ1
=> VNAME3D(220 ): SV_BNZ2
=> VNAME3D(221 ): SV_PAH1
=> VNAME3D(222 ): SV_PAH2
=> VNAME3D(223 ): SV_TRP1
=> VNAME3D(224 ): SV_TRP2
=> VNAME3D(225 ): SV_ISO1
=> VNAME3D(226 ): SV_ISO2
=> VNAME3D(227 ): SV_SQT
=> VNAME3D(228 ): LV_PCSOG
ncendef: ncid 3080192: NetCDF: One or more variable sizes violate format constraints
Error putting netCDF file into data mode.
netCDF error number -62 processing file “S_CGRID”
NetCDF: One or more variable sizes violate format constraints

 *** ERROR ABORT in subroutine WR_CGRID on PE 000
 Could not open S_CGRID file
 Date and time 0:00:00   Jan. 2, 2016   (2016002:000000)

application called MPI_Abort(MPI_COMM_WORLD, 538976288) - process 0
0.112u 1.761s 2:50:34.39 0.0% 0+0k 3360+18592io 0pf+0w

I have never met this problem when using this model for a smaller eastern U.S. domain. I guess it may related to the input information, but can not figure out which species have the incorrect dimension. Any suggestion?

Hao

I am not certain, but this error could be because you are re-running the case, and IO/API is complaining that it can’t overwrite an output file.
Can you try running with the option:

#> Keep or Delete Existing Output Files
set CLOBBER_DATA = TRUE

-62 turns out to be NetCDF error NF_EVARSIZE, and is probably associated with one or more variable-dimensions being negative or zero. Don’t know how that happened, but…

I have recompiled IOAPI and it solved the problem, fyi.