Hello all,
I am running CMAQ for 12CONUS domain. From feb 1,2023 through May 1 2023. I am facing issues in cctm run named ‘EBI Euler convergence failure’. I am attaching the cctm run file that I am using along with a log file. Thanks for your help.
CTM_LOG_091.v532_gcc_12US1_459X299_20230201.txt (5.4 MB)
run_cctm_202302_12US1.csh (35.7 KB)
slurm-5107189.txt (20.4 KB)
Hasibul
The EBI convergence failure messages in your CTM_LOG file indicate some very high concentrations for some VOC species as well as CO.
Looking at the DESID emissions mapping report, this might be caused by some very high scaling factors being applied to your emissions, since the final scaling factors the code is trying to write to the log file appear to violate the Fortran format constraints so they are just being written as “******”:
CMAQ Species Phase/Mode Surrogate Region Op ScaleFac Basis FinalFac
------------ ---------- --------- ------ -- -------- ----- --------
NO2 GAS NO2 EVERYWHERE a 1.000 UNIT ******
NO GAS NO EVERYWHERE a 1.000 UNIT ******
HONO GAS HONO EVERYWHERE a 1.000 UNIT ******
SO2 GAS SO2 EVERYWHERE a 1.000 UNIT ******
SULF GAS SULF EVERYWHERE a 0.000 UNIT 0.000
AACD GAS AACD EVERYWHERE a 1.000 UNIT ******
ALD2 GAS ALD2 EVERYWHERE a 1.000 UNIT ******
FORM GAS FORM EVERYWHERE a 1.000 UNIT ******
MEOH GAS MEOH EVERYWHERE a 1.000 UNIT ******
FACD GAS FACD EVERYWHERE a 1.000 UNIT ******
when we would typically expect something like this
CMAQ Species Phase/Mode Surrogate Region Op ScaleFac Basis FinalFac
------------ ---------- --------- ------ -- -------- ----- --------
NO2 GAS NO2 EVERYWHERE a 1.000 UNIT 1.000
NO GAS NO EVERYWHERE a 1.000 UNIT 1.000
HONO GAS HONO EVERYWHERE a 1.000 UNIT 1.000
SO2 GAS SO2 EVERYWHERE a 1.000 UNIT 1.000
SULF GAS SULF EVERYWHERE a 0.000 UNIT 0.000
ALD2 GAS ALD2 EVERYWHERE a 1.000 UNIT 1.000
FORM GAS FORM EVERYWHERE a 1.000 UNIT 1.000
MEOH GAS MEOH EVERYWHERE a 1.000 UNIT 1.000
CO GAS CO EVERYWHERE a 1.000 UNIT 1.000
ALDX GAS ALDX EVERYWHERE a 1.000 UNIT 1.000
ETHA GAS ETHA EVERYWHERE a 1.000 UNIT 1.000
ETOH GAS ETOH EVERYWHERE a 1.000 UNIT 1.000
I admit that I don’t know what could be causing the final factor to be off the charts, but it might help if you could post your EmissCtrl file.
Please find attached my EmisCtrl file. Thats the file came with the CMAQ installation. Thanks!
Hasibul
EmissCtrl_cb6r3_ae7_aq.txt (31.0 KB)
Hasibul,
I see from your log file that you are using CMAQv5.3.2, which is now nearly 4 years old. There have been two releases since then, with another to come out shortly. Have you made any modifications to the code?
Try recompiling in debug mode and rerun the model.
I run the model in debug mode for a smaller domain to check. Similar issue occurring. Please find attached the new cctm file along with the log file. Thanks for your help.
Hasibul
CTM_LOG_027.v532_gcc_OOSL1330_105X135_20230201.txt (125.4 KB)
slurm-5108244.txt (52.5 KB)
run_cctm_202302_4FL.txt (35.7 KB)
If you are using the release version of the code and the DESID emission control file, then the log file should show scale factors of 1.000. The fact that the log file shows these are not printed correctly indicates an error. What modifications to the CMAQ source code have you made?
To clarify, when I say to compile in “debug mode”, this means to use flags appropriate for your compiler that will check for runtime error conditions like exceeding array bounds and floating point exceptions. For gcc, we recommend these (see config_cmaq.csh):
-Wall -O0 -g -fcheck=all -ffpe-trap=invalid,zero,overflow -fbacktrace"
Assuming you are using our standard build script, you can recompile invoking these debug flags with the command:
make clean; make DEBUG=TRUE
issued from your build directory.
A comment on your run script: START_DATE, END_DATE, and NSTEPS are not set correctly (our run script is rather confusing on this point). The way our run script is set up is to do a series of 24-hour executions, with NSTEPS set to 240000. You can think of START_DATE as the first execution date and END_DATE as the last execution date.
If instead you have prepared your inputs so they hold all the data for your 3-month simulation period, then you want to set your script so that it executes only once. One way to achieve this is to set END_DATE the same as START_DATE.
I did run in the debug mode but there are no new information. I tried with NSTEPS set to 240000 but still the same issue. When I did not use my gridded emission data (egts…), then the model worked. But when I used it, it did not work. So, there are some discrepancies between my GRIDDED_EMIS file and EmisCtrl file. My GRIDDED_EMIS file comes from SMOKE run with all the sources (point, mobile, area and biogenic) combined. That is why I do not have any inline emission or biogenic emission invoked in CMAQ. I decided to ncdump header of my gridded_emis and a standard test_case gridded_emis to have a look. Here is the ncdump output from both of them.
testcase_emis.txt (30.5 KB)
FL1330m_emis.txt (25.6 KB)
I see they used moles/s in test case and I used moles/hr. Is that the reason the scale factor is unreasonable for my case? Do I have to change EmisCtrl someway to resolve the issue?
Thank you so much for your help.
Hasibul
Or maybe I have issue with my initial and boundary concentration file or tolerence set up as the error says that,
Processing will continue with QS set to ZERO. <<–<<
After CLDPROC : Gas 1.287E+03 | Aer 3.928E+12 | Non 6.203E+00
Comments on Species in degradation routines
Euler Backward Iterative Parameters -
Chemistry Integration Time Interval (min): 2.0000
EBI maximum time step (min): 2.5000
Species convergence tolerances:
NO2 5.00E-04
NO 5.00E-04
O 1.00E+00
O3 5.00E-04
NO3 5.00E-04
O1D 1.00E+00
OH 5.00E-04
HO2 5.00E-04
H2O2 5.00E-04
N2O5 5.00E-04
HNO3 5.00E-04
HONO 5.00E-04
PNA 5.00E-04
SO2 5.00E-04
SULF 1.00E+00
SULRXN 1.00E+00
C2O3 5.00E-04
MEO2 5.00E-04
RO2 5.00E-04
PAN 5.00E-04
PACD 5.00E-04
AACD 5.00E-04
CXO3 5.00E-04
ALD2 5.00E-04
XO2H 5.00E-04
PANX 5.00E-04
FORM 5.00E-04
MEPX 5.00E-04
MEOH 5.00E-04
ROOH 5.00E-04
XO2 5.00E-04
XO2N 5.00E-04
NTR1 5.00E-04
NTR2 5.00E-04
FACD 5.00E-04
CO 5.00E-04
HCO3 1.00E+00
ALDX 5.00E-04
GLYD 5.00E-04
GLY 5.00E-04
MGLY 5.00E-04
ETHA 5.00E-04
ETOH 5.00E-04
KET 5.00E-04
PAR 5.00E-04
ACET 5.00E-04
PRPA 5.00E-04
XPRP 5.00E-04
XPAR 5.00E-04
ROR 5.00E-04
ETHY 5.00E-04
ETH 5.00E-04
OLE 5.00E-04
IOLE 5.00E-04
ISOP 5.00E-04
ISO2 5.00E-04
ISOPRXN 1.00E+00
ISPD 5.00E-04
INTR 5.00E-04
ISPX 5.00E-04
HPLD 5.00E-04
OPO3 5.00E-04
EPOX 5.00E-04
EPX2 5.00E-04
TERP 5.00E-04
TRPRXN 1.00E+00
TERPNRO2 5.00E-04
APIN 5.00E-04
BENZENE 5.00E-04
CRES 5.00E-04
BZO2 5.00E-04
OPEN 5.00E-04
BENZRO2 5.00E-04
TOL 5.00E-04
TO2 5.00E-04
TOLRO2 5.00E-04
XOPN 5.00E-04
XYLMN 5.00E-04
XLO2 5.00E-04
XYLRO2 5.00E-04
NAPH 5.00E-04
PAHRO2 1.00E+00
CRO 5.00E-04
CAT1 5.00E-04
CRON 5.00E-04
OPAN 5.00E-04
ECH4 5.00E-04
CL2 5.00E-04
CL 5.00E-04
HOCL 5.00E-04
CLO 5.00E-04
FMCL 5.00E-04
HCL 5.00E-04
CLNO2 5.00E-04
CLNO3 5.00E-04
SVAVB2 5.00E-04
SVAVB3 5.00E-04
SVAVB4 5.00E-04
SVAVB1 5.00E-04
SESQ 5.00E-04
SESQRXN 1.00E+00
SOAALK 5.00E-04
H2NO3PIJ 5.00E-04
H2NO3PK 5.00E-04
ACLI 5.00E-04
ACLJ 5.00E-04
ACLK 5.00E-04
IEPOXP 5.00E-04
ASO4J 5.00E-04
AISO3J 5.00E-04
AGLYJ 5.00E-04
MTNO3 5.00E-04
AMTNO3J 5.00E-04
AMTHYDJ 5.00E-04
AAVB2J 5.00E-04
AOLGAJ 5.00E-04
AAVB3J 5.00E-04
AAVB4J 5.00E-04
AISO1J 5.00E-04
AOLGBJ 5.00E-04
AISO2J 5.00E-04
ASQTJ 5.00E-04
APOCI 5.00E-04
APNCOMI 5.00E-04
APOCJ 5.00E-04
APNCOMJ 5.00E-04
PCVOC 5.00E-04
PCSOARXN 5.00E-04
VLVPO1 5.00E-04
VSVPO1 5.00E-04
VSVPO2 5.00E-04
VSVPO3 5.00E-04
VIVPO1 5.00E-04
VLVOO1 5.00E-04
VLVOO2 5.00E-04
VSVOO2 5.00E-04
VSVOO3 5.00E-04
VSVOO1 5.00E-04
FORM_PRIMARY 5.00E-04
ALD2_PRIMARY 5.00E-04
BUTADIENE13 5.00E-04
ACROLEIN 5.00E-04
ACRO_PRIMARY 5.00E-04
TOLU 5.00E-04
HG 5.00E-04
HGIIAER 5.00E-04
HGIIGAS 5.00E-04
WARNING: EBI solver in cell ( 1, 1, 1) Init.Conc. for CO = 1.7010E+03 ppmV
WARNING: EBI solver in cell ( 5, 3, 1) Init.Conc. for CO = 1.1389E+03 ppmV
WARNING: EBI solver in cell ( 5, 3, 1) Init.Conc. for ECH4 = 1.8168E+04 ppmV
WARNING: EBI solver in cell ( 6, 3, 1) Init.Conc. for CO = 2.2110E+03 ppmV
WARNING: EBI solver in cell ( 6, 3, 1) Init.Conc. for ECH4 = 7.0054E+04 ppmV
WARNING: EBI solver in cell ( 4, 4, 1) Init.Conc. for CO = 1.2668E+03 ppmV
WARNING: EBI solver in cell ( 4, 4, 1) Init.Conc. for ECH4 = 1.8166E+04 ppmV
WARNING: EBI solver in cell ( 5, 4, 1) Init.Conc. for NO = 1.6114E+03 ppmV
WARNING: EBI solver in cell ( 5, 4, 1) Init.Conc. for CO = 2.5065E+04 ppmV
WARNING: EBI solver in cell ( 5, 4, 1) Init.Conc. for ETHA = 6.6877E+03 ppmV
WARNING: EBI solver in cell ( 5, 4, 1) Init.Conc. for PAR = 3.2024E+03 ppmV
WARNING: EBI solver in cell ( 5, 4, 1) Init.Conc. for ECH4 = 1.3287E+06 ppmV
WARNING: EBI solver in cell ( 5, 4, 1) Init.Conc. for PCVOC = 1.8629E+03 ppmV
WARNING: EBI solver in cell ( 6, 4, 1) Init.Conc. for NO = 5.4779E+03 ppmV
WARNING: EBI solver in cell ( 6, 4, 1) Init.Conc. for SO2 = 1.4094E+03 ppmV
WARNING: EBI solver in cell ( 6, 4, 1) Init.Conc. for CO = 8.9320E+04 ppmV
WARNING: EBI solver in cell ( 6, 4, 1) Init.Conc. for ETHA = 2.4453E+04 ppmV
WARNING: EBI solver in cell ( 6, 4, 1) Init.Conc. for PAR = 9.7643E+03 ppmV
WARNING: EBI solver in cell ( 6, 4, 1) Init.Conc. for PRPA = 1.3725E+03 ppmV
WARNING: EBI solver in cell ( 6, 4, 1) Init.Conc. for OLE = 1.2606E+03 ppmV
WARNING: EBI solver in cell ( 6, 4, 1) Init.Conc. for TOL = 3.1300E+03 ppmV
WARNING: EBI solver in cell ( 6, 4, 1) Init.Conc. for ECH4 = 4.8729E+06 ppmV
WARNING: EBI solver in cell ( 6, 4, 1) Init.Conc. for PCVOC = 6.8216E+03 ppmV
WARNING: At solution attempt # 1
WARNING: EBI Euler convergence failure
Reducing EBI time step because of MAXPRED failure in
Cell ( 81, 38, 1) for the below species: : Init.Conc, Pred.Conc.
NO2 5.9723E+02 2.5739E+03 ppmV
Thanks for your help.
Hasibul
Could you please post a map of your layer 1 CO and NO emissions for one hour so we can look at their magnitude? Since SMOKE typically does not create emissions in moles/hour when generating files for CMAQ, it would also be helpful if you could provide more details on your emission inventories and how you processed them through SMOKE.
Hi Hasibul,
We recently found an error in the conversion of emissions from per hour units to per second units. Assuming you are running v5.3.2, please go to this line: CMAQ/CCTM/src/emis/emis/EMIS_DEFN.F at 5.3.2 · USEPA/CMAQ · GitHub
and change it to:
CONV = CONV / 3600.0
Hopefully this will alleviate your issue.
Please find attached the layer 1 emission for a particular hour for NO2 and CO.
Thanks for your help.
Hasibul
After changing to CONV = CONV / 3600.0, the model went through without stopping. Here is the log file (partial):
CTM_LOG_054.v532_gcc_OOSL1330_105X135_20230201.txt (115.1 KB)
Now the FinalFrac of my GRIDDED_EMIS file is zero.
|> EMISSIONS SCALING DIAGNOSTIC:
==================================
Note: these are populated using rules from the
Emission Control File Supplied by the User.
>--------------------------------------------------------------------------------
Stream Type: "Gridded Area Emissions File 1" | Sector Label: GRIDDED_EMIS (01)
Table of Aerosol Size Distributions Available for Use Sector-Wide.
Note that Mode 1 is reserved for gas-phase species and surrogates.
Number Surrogate Mode Reference Mode (see AERO_DATA.F)
------ -------------- --------------------------------
2 FINE FINE_REF
3 COARSE COARSE_REF
CMAQ Species Phase/Mode Surrogate Region Op ScaleFac Basis FinalFac
------------ ---------- --------- ------ -- -------- ----- --------
NO GAS NO EVERYWHERE a 1.000 UNIT 0.000
AACD GAS AACD EVERYWHERE a 1.000 UNIT 0.000
ALD2 GAS ALD2 EVERYWHERE a 1.000 UNIT 0.000
FORM GAS FORM EVERYWHERE a 1.000 UNIT 0.000
MEOH GAS MEOH EVERYWHERE a 1.000 UNIT 0.000
FACD GAS FACD EVERYWHERE a 1.000 UNIT 0.000
CO GAS CO EVERYWHERE a 1.000 UNIT 0.000
ALDX GAS ALDX EVERYWHERE a 1.000 UNIT 0.000
ETHA GAS ETHA EVERYWHERE a 1.000 UNIT 0.000
ETOH GAS ETOH EVERYWHERE a 1.000 UNIT 0.000
KET GAS KET EVERYWHERE a 1.000 UNIT 0.000
PAR GAS PAR EVERYWHERE a 1.000 UNIT 0.000
ACET GAS ACET EVERYWHERE a 1.000 UNIT 0.000
ETH GAS ETH EVERYWHERE a 1.000 UNIT 0.000
OLE GAS OLE EVERYWHERE a 1.000 UNIT 0.000
IOLE GAS IOLE EVERYWHERE a 1.000 UNIT 0.000
ISOP GAS ISOP EVERYWHERE a 1.000 UNIT 0.000
TERP GAS TERP EVERYWHERE a 1.000 UNIT 0.000
APIN GAS APIN EVERYWHERE a 1.000 UNIT 0.000
SESQ GAS SESQ EVERYWHERE a 1.000 UNIT 0.000
FORM_PRIMARY GAS FORM_PRIMARY EVERYWHERE a 1.000 UNIT 0.000
ALD2_PRIMARY GAS ALD2_PRIMARY EVERYWHERE a 1.000 UNIT 0.000
>--------------------------------------------------------------------------------
Stream Type: "Lightning NO Emissions" | Sector Label: LTNG (02)
Table of Aerosol Size Distributions Available for Use Sector-Wide.
Note that Mode 1 is reserved for gas-phase species and surrogates.
Number Surrogate Mode Reference Mode (see AERO_DATA.F)
------ -------------- --------------------------------
2 FINE FINE_REF
3 COARSE COARSE_REF
CMAQ Species Phase/Mode Surrogate Region Op ScaleFac Basis FinalFac
------------ ---------- --------- ------ -- -------- ----- --------
NO GAS NO EVERYWHERE a 1.000 UNIT 1.000
>--------------------------------------------------------------------------------
Stream Type: "Sea Spray Aerosol Emissions" | Sector Label: SEASPRAY (03)
Table of Aerosol Size Distributions Available for Use Sector-Wide.
Note that Mode 1 is reserved for gas-phase species and surrogates.
Number Surrogate Mode Reference Mode (see AERO_DATA.F)
------ -------------- --------------------------------
2 FINE FINE_SEASPRAY
3 COARSE COARSE_SEASPRAY
CMAQ Species Phase/Mode Surrogate Region Op ScaleFac Basis FinalFac
------------ ---------- --------- ------ -- -------- ----- --------
ASO4J FINE PMFINE_SO4 EVERYWHERE a 1.000 UNIT 1.000
ACAJ FINE PMFINE_CA EVERYWHERE a 1.000 UNIT 1.000
AMGJ FINE PMFINE_MG EVERYWHERE a 1.000 UNIT 1.000
AKJ FINE PMFINE_K EVERYWHERE a 1.000 UNIT 1.000
AH2OJ FINE PMFINE_H2O EVERYWHERE a 1.000 UNIT 1.000
ANAJ FINE PMFINE_NA EVERYWHERE a 1.000 UNIT 1.000
ACLJ FINE PMFINE_CL EVERYWHERE a 1.000 UNIT 1.000
ASEACAT COARSE PMCOARSE_SEACAT EVERYWHERE a 1.000 UNIT 1.000
ACLK COARSE PMCOARSE_CL EVERYWHERE a 1.000 UNIT 1.000
ASO4K COARSE PMCOARSE_SO4 EVERYWHERE a 1.000 UNIT 1.000
AH2OK COARSE PMCOARSE_H2O EVERYWHERE a 1.000 UNIT 1.000
>--------------------------------------------------------------------------------
Stream Type: "Wind-Blown Dust Emissions" | Sector Label: WBDUST (04)
Table of Aerosol Size Distributions Available for Use Sector-Wide.
Note that Mode 1 is reserved for gas-phase species and surrogates.
Number Surrogate Mode Reference Mode (see AERO_DATA.F)
------ -------------- --------------------------------
2 FINE FINE_WBDUST
3 COARSE COARSE_WBDUST
CMAQ Species Phase/Mode Surrogate Region Op ScaleFac Basis FinalFac
------------ ---------- --------- ------ -- -------- ----- --------
ASO4J FINE PMFINE_SO4 EVERYWHERE a 1.000 UNIT 1.000
ANH4J FINE PMFINE_NH4 EVERYWHERE a 1.000 UNIT 1.000
ANO3J FINE PMFINE_NO3 EVERYWHERE a 1.000 UNIT 1.000
AOTHRJ FINE PMFINE_OTHR EVERYWHERE a 1.000 UNIT 1.000
AFEJ FINE PMFINE_FE EVERYWHERE a 1.000 UNIT 1.000
AALJ FINE PMFINE_AL EVERYWHERE a 1.000 UNIT 1.000
ASIJ FINE PMFINE_SI EVERYWHERE a 1.000 UNIT 1.000
ATIJ FINE PMFINE_TI EVERYWHERE a 1.000 UNIT 1.000
ACAJ FINE PMFINE_CA EVERYWHERE a 1.000 UNIT 1.000
AMGJ FINE PMFINE_MG EVERYWHERE a 1.000 UNIT 1.000
AKJ FINE PMFINE_K EVERYWHERE a 1.000 UNIT 1.000
AMNJ FINE PMFINE_MN EVERYWHERE a 1.000 UNIT 1.000
ASOIL COARSE PMCOARSE_SOIL EVERYWHERE a 1.000 UNIT 1.000
AH2OJ FINE PMFINE_H2O EVERYWHERE a 1.000 UNIT 1.000
ANAJ FINE PMFINE_NA EVERYWHERE a 1.000 UNIT 1.000
ACLJ FINE PMFINE_CL EVERYWHERE a 1.000 UNIT 1.000
ACLK COARSE PMCOARSE_CL EVERYWHERE a 1.000 UNIT 1.000
ASO4K COARSE PMCOARSE_SO4 EVERYWHERE a 1.000 UNIT 1.000
ANO3K COARSE PMCOARSE_NO3 EVERYWHERE a 1.000 UNIT 1.000
AH2OK COARSE PMCOARSE_H2O EVERYWHERE a 1.000 UNIT 1.000
ALVPO1J FINE PMFINE_LVPO1 EVERYWHERE a 1.000 UNIT 1.000
ALVOO1J FINE PMFINE_LVOO1 EVERYWHERE a 1.000 UNIT 1.000
================================================================================
Please let me know the log file looks good now and if it is normal that the FinalFac to be 0.00. Thanks for your help.
Hasibul
Hello Hasibul,
thank you for posting the emission plots which confirm that the magnitude of your emissions is indeed consistent with units of moles/hr. And thank you for confirming that implementing the code fix posted by Ben resolves the convergence failure.
Here is my perspective on the issue you were encountering: