Hi all,
The CMAQ-DDM v5.4 was running fine the day before, but the ASENS file generated the next day contains all NaN values, while the CONC file is still normal. I saw another person’s question on the forum mentioning NaN issues, but their problem was due to emissions. I’m using EQUATES for emissions, which should have been evaluated. What could be the possible reasons for this?
Please, try downloading and compiling the 5.4+ branch instead of the base version. There were a lot of small bug fixes in that version that came out after the base 5.4 release and its possible one of these is causing your issue.
Also, please, delete the other duplicate post asking the same question.
Cheers,
Sergey
Thanks Sergey! Despite downloading version 5.4+, the issue persists.
Initially, I suspected it was related to emissions. Upon examining the ASENS file, I noticed NaN values appearing from 6:00 on December 18, 2023, onwards. Upon observing the input emissions, I found that NOx emissions from agricultural fires summed to zero between 4:00 to 12:00. However, even after disabling agricultural fire emissions, the problem persisted. Hence, emissions may not be the root cause.
Could you suggest other potential solutions? I attached the log file.
log.txt (367.2 KB)
Are you running the MEGAN biogenic emissions option? I think some users reported problems with that in the past. If that’s the case for you and you running the EQUATES case, it could be possible to track that down.
Sergey
I’m using BEIS to calculate in-line biogenic emissions (setenv CTM_BIOGEMIS_BE Y). Yes, I’m running the EQUATES case.
Hi Sergey,
Even after setting the biogenic emissions to “False,” I encountered the same issue.
I’ve noticed an interesting pattern: when I disable agricultural fires, the appearance of NaN values shifts from 5:00 to 6:00 on December 18, 2023. Similarly, when I turn off biogenic emissions, the first NaN value appears at 7:00. This leads me to wonder if there’s an overflow issue causing these problems.
@nna,
If there is an overflow issue, I suggest compiling in “debug” mode see: CMAQ/DOCS/Users_Guide/Tutorials/CMAQ_UG_tutorial_debug.md at main · USEPA/CMAQ · GitHub.
This will enable compilation with “debug” flags which will check and trap for floating point exceptions (divide by zeros, overflow, etc). Hopefully if a floating point calculation is going erroneous it will be flagged and will give you the line number.
Hi Fahim,
Thank you! I reran it in “debug” mode. However, it encountered an error immediately instead of running as before, stopping at the 8th hour. Here are the log files. I believe it’s primarily related to the nodes on our server. We’ll need to address the server issues first before determining if it’s an overflow problem.
log.txt (24.3 KB)
CTM_LOG_000.v54_DDM3D_intel_2023_12US1_20231218.txt (204.3 KB)
@nna,
While I can’t rule out sever issues, failing immediately is not uncommon in this case because compiling in debug mode enables floating point checks.
Looking at your log.txt, it’s preety clear that the code is encountering an overflow issue while doing tracer advection of the DDM “species”. I don’t think there is an issue with the advection routine itself, but instead the concentrations going into the routine.
Are you able to run the same scenario successfully without DDM turned on to verify that works and everything looks reasonable?
I have ruled out the sever issues. Here’s my summary:
DDM in debug mode: immediately fails as described in the log above.
DDM not in debug mode: runs but generates NaN values at the 8th hour.
Without DDM: works fine, and the NO2 concentration appears reasonable.
Could you help me in further diagnosing an overflow issue? Thanks!
Which specific EQUATES episode are you running and what sensitivities are you specifying in the sensinput.dat file? I am not sure how to diagnose this remotely, but I can try to reproduce the problem here.
Sergey
Hi Sergey,
I’m using the emissions from December 13, 2023 to December 18, 2023 for the simulation of the corresponding days in 2023. And the NAN issue in ASENS file occurs on December 18.
The details in the sensinput.dat file are as follows:
EMIS
TOTA
SPECIES
NO2, NO
@nna,
Since this problem occurs on the last day, this maybe an issue that has manifested itself over the course of your run. I suspect the sensitivity concentrations in your IC file are what’s probably driving the issue.
I suggest restarting the run on the first day Dec. 13, 2023 using the debug flags and see what it catches.
Thanks! I’ll try it.
Hi Fahim,
I understand that debug mode can be time-consuming, but does it really take around 700 seconds for a 5-minute simulation? At this rate, it would take half a month to run a simulation from December 13 to 18.
@nna,
Yes it can be quite time consuming and very slow because its checking each and every line of code.
Often it is useful to narrow down your search, but I’m not sure what your skill level with writing Fortran code is how comfortable you are with editing the CMAQ Fortran source code yourself.
If you are quite fluent in Fortran, you could adapt the CMAQ routine cksummer.F to check the DDM sensitivities for NaNs/Infs, then run debug mode on that specific day. If you do go that route note that unlike the CGRID species DDM sensitives can be negative and positive so be sure to use the absolute value when summing sensitives.
Thanks for your suggestion!