Forrtl: severe 65: floating invalid error in SSEMIS.F in CMAQv5.2.1


#6

I checked the variables for that specific day of crash. RN range is between min of 0.0 cm and max of 0.265 cm, but RC is -1.0 cm everywhere. Also, I checked other files and RC is -1.0 in every file! What do you think about this?

Also, I attached a screenshot of a warning in subroutine CONVCLD_ACM to my previous post. Do you think that might cause any issue related to my present situation?

Thanks.


#7

RC= -1.0 indicates that no convective parameterization was used in the WRF simulation, so there is no convective (subgrid) rain. This should not be a problem.

Your problem is in SSEMIS, even though your domain has no ocean. As I said previously, emissions processing has changed in v5.3, and it is much easier to turn off sea spray emissions. But if you’re committed to v5.2.1, try this:

In ASX_DATA_MOD.F, just before the RETURN statement at the end of the INIT_MET subroutine, set both the OCEAN and SZONE arrays to 0.

Grid_Data%OCEAN = 0.0
Grid_Data%SZONE = 0.0

Again, I have not tested this, but I think it should prevent SSEMIS from being called.


#8

Hello again,

I am trying CMAQv53 now. I added those lines to the source code in ASX_DATA_MOD file, and also set CTM_SS_AERO to N. Again CMAQv53 crashed with error “Maximum AQCHEM total iterations exceeded”. Then I compiled and run in dbg mode for that specific day and pull out the error message and stack:

“forrtl: severe (408): fort: (3): Subscript #1 of the array MW has value 0 which is less than the lower bound of 1”

I also set CTM_EMISCHK to T for this run.


#9

Ok, the problem isn’t in SSEMIS anymore, so we can declare victory on one thing. :slight_smile:

This is something else involving the new emissions processing code. There should be a much better error message and code abort rather than a crash. Something is not getting initialized. Can you tell from one of the log files (e.g., CTM_LOG_000*) what emissions stream was being processed? (Was it gridded emissions, or do you have inline point sources, or was it biogenics, or lightning NO, or … ?)


#10

This is version 5.3b2, right?


#11

Yes, I am using CMAQv5.3.b2.

I ran CMAQ in debug mode for the day of the crash (Sep13) with yesterday ICON, and CMAQ did not start at all and crashed with the error message in the screenshot I attached to my earlier post here.

Regarding emissions stream, I have inline fire emissions for each day. I checked the CTM_LOG_ files and CMAQ tried to open gridded emissions and then in runtime log file it shows the error in MW array.


#12

This is the screenshot of the settings of my run script for Sep13 using CCTM_CGRID from yesterday.


#13

Hello,

I managed to run CMAQv5.3 with debug mode for faulty day and looked at changes in species concentrations after each time step. CMAQ crashes at hr 8:00:00 with AQCHEM error at subdomain #77 and for one minute before the crash (7:59:00) there are NaN values for several modules in that subdomain. I have attached a screenshot of that to this message.

That might be the cause of the crash. So I was wondering what causes NaN values? If it comes from met files, which parameters should I check? If it comes from emissions, which file should I check? I have both inline (fires) and gridded emissions in this model run.

Thanks a lot for your advice.


#14

Why not run M3TOOL:S program “m3stat” (see https://cjcoats.github.io/ioapi/M3STAT.html or https://www.cmascenter.org/ioapi/documentation/all_versions/html/M3STAT.html)
all the input files? That would show up any NaNs that are in the inputs…


#15

The NaNs are almost certainly the cause of your model crash. They indicate your simulation is corrupted.
Carlie’s suggestion for checking the input files is good. It is strange to me that NaNs first occur at 7:59. If there was a bad value at 8:00 in one of the data files, then that bad value would have been used for interpolation at every time after 7:00, so I would expect the crash to occur sooner.
Try grepping for NaN in all your log files to see when the first one occurs–it might not be on the same node that crashes later. Then look at that log file around that time and see if there are any further clues.
On most compilers you can use flags to cause the model to crash if there are floating point violations, arrays out of bounds, uninitialized variables, or other exceptions. You might try that as well.


#16

Thank you dear Chris and Carlie,

You were right Chris. I checked my log files for the whole modeling period before model crash on Sep 13 and found out that NaNs occurred at almost every day of simulation at UTC 7:00 on two sub grids of #77 and #78. We are locally UTC-7hrs.

Then I ran CMAQv5.3 in debug mode for the first day of simulation with default ICON and BCON, CTM_SSEMDIAG = T, and CTM_SS_AERO = F to turn off sea salt. The run finished successfully, but the log files show there are NaN at 7:00 and 7:59 and several lines of following warning. I have attached two screenshots with more detail.

WARNING: EBI solver in cell ( 30, 15, 40) Init.Conc. for SULF = 1.0220E+03 ppmV

So I will check both MCIP and emission files for any bad value at UTC 7:00, but what should I look for as “bad value”?

Thanks a lot in advance.



#17

The CMAQ build script includes suggestions for debug flags for intel, gcc, and pgi compilers. If you are using one of those, then you can go into your build directory and type

make clean
make DEBUG=TRUE

Then rerun the model. It may crash with a useful stack trace that shows you where the problem is.


#18

Hello,

I grepped almost the following input files for Sep 01 for NaN values and could not find anything.

Should I look for NaN in input files? Or some weird values might represent as NaN in model run?

I see some very small values in fire emission files with values of 1E-33. Also, there are weird values for stack parameters for fires.

METCRO2D_20160901

GRIDCRO2D_160901

METCRO3D_160901

Ocean file

ICON original

BCON original

Inline emissions: no NaN, but very small values 1e-36

Gridded emissions



#19

Did you see this thread?


Since you are also getting errors at 0:00, maybe you have a similar problem with missing BCON data being set to 1e36.


#20

Yes I saw that, but in my case I am running in debug mode and I turned off sea spray emissions. I ran in debug mode for the the first day of my simulation and debug mode does not show anything and CMAQ finishes successfully, but there are NaNs in CTM_LOG files at hr=7:00.

I checked my ICON with m3stat and all variables are in the range of E±10, but only SULF is set to 1.0E-30.
I am using CMAQv5.3 and created ICON/BCON from CMAQv521 since ICON/BCON programs for v53 where not available. I tried to open BCON with m3stat, but m3stat does not produce output for BCON. I checked inside BCON and it has values.


#21

I was checking CMAQ log file to see if CMAQ has opened BCON file for the run and came across this.
No BC’s in file BNDY_AERO_1 for the following adv species: Set to 1.00E-30
I have this message for BNDY_GASC_1, BNDY_AERO_1, BNDY_NONR_1.


#22

Something has gone wrong here. If you are creating BCs from a simulation run on a parent domain, then you should have valid values for all model species. But your log file is indicating many species are not in your BCON file.

Taking a step back, how exactly did you create the BCs for this simulation? What you should do (ideally) is:

  1. Run CMAQ for the parent domain, saving all species (or at least all important species) to the CONC file, for all levels.
  2. Run the BCON program (using the CONC files generated in step 1) to generate BC files containing the value for each of your model species, for all vertical levels, for the boundary cells of your nested domain.
  3. Run CMAQ for the nested domain, pointing to the BC files created in step 2.

Is that the procedure you followed?


#23

I did not prepare BCs from running CMAQ on parent domain. I built BCON inside CMAQv521 and ran BCON with default profile data that comes with CMAQv521. Then I used this BCON file for CMAQv5.3.


#24

If you are using profile boundary conditions, then many species are not available, and those warnings that they are absent are totally normal.

Since the problem that was the original subject of this thread has been solved, I’m going to close this. If you’re still having problems, please post a new thread.


closed #25