NaN or Infinity detected in VDIFF in multi-day MPAS-CMAQ run

MPAS-CMAQ
1

Hello,

I am currently studying the MPAS-CMAQ model and would like to ask a question about an issue I encountered during execution.

First, I downloaded the one-year example dataset provided by the US EPA. I then attempted to run the model for the period from January 16 to January 22, 2017.

What I found interesting is the following:
(1) When I run the model one day at a time, the simulation completes successfully without any errors.
(2) However, when I try to run the entire period from January 16 to January 22 at once, an error occurs.

The error message is: “NaN or Infinity detected in VDIFF.”

This error occurs after the simulation for January 16 is completed, specifically when the model starts calculating 00:07:30 on January 17 (that is, during the first time step of the next day).

Could anyone explain why this error occurs?

Has anyone experienced a similar issue?

Thank you.

image
image852×377 11.1 KB

2

There are CTM_LOG_[xxx] files that are written for each day under the output directory. Please check to see what the CTM_LOG_018 contains:

example command:

cat 20170117/CTM_LOG_018

based on this message: Nan or Infinity detected on processor 018.

Perhaps the output files for the first day, aren’t fully written to the filesystem, prior to the start of the next day? You could try adding a sleep command after the line for mpirun -np … in the run script.

sleep 1m
3

Dear lizadams,

First of all, thank you very much for your reply.

I apologize for the delay in responding, as I was testing the MPAS-CMAQ model after adding the sleep command to the script as you suggested.

Unfortunately, even after applying your suggestion, the error “NaN or Infinity detected on processor” still occurred.

Additionally, I tested running the MPAS-CMAQ model with the CTM_WB_DUST and BDSNP_MEGAN options set to "N". When I applied this method, the continuous simulation of the MPAS-CMAQ model completed successfully. I suspect that this is because the model does not process feedback related to SOILINP/BDSNPINP, which allowed the continuous simulation to run without issue.

Although I have not yet evaluated the model performance, I believe that regardless of the results, I will ultimately need to run MPAS-CMAQ with these options set to "Y".

In the current situation, could you please advise me on what I should examine first in order to enable these options and run the MPAS-CMAQ model successfully?

Thank you again for your reply. I have been gaining very valuable experience through the CMAS Forum, and I truly appreciate it.

Sincerely,
Jeongbeom Lee

Note: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG IEEE_DIVIDE_BY_ZERO IEEE_OVERFLOW_FLAG IEEE_UNDERFLOW_FLAG IEEE_DENORMAL

     CSQY_DATA         |  /home/mpas-cmaq-v1.1/MPAS_v7.0/src/core_atmosphere/cmaq/CSQY_DATA_cracmm2

     OPTICS_DATA       |  /home/mpas-cmaq-v1.1/MPAS_v7.0/src/core_atmosphere/cmaq/PHOT_OPTICS.dat

     CSQY_DATA         |  /home/mpas-cmaq-v1.1/MPAS_v7.0/src/core_atmosphere/cmaq/CSQY_DATA_cracmm2

     OPTICS_DATA       |  /home/mpas-cmaq-v1.1/MPAS_v7.0/src/core_atmosphere/cmaq/PHOT_OPTICS.dat

     OMI               |  /raid/mpas-cmaq/2017_120km/omi_cmaq_2017.dat

     OMI               |  /raid/mpas-cmaq/2017_120km/omi_cmaq_2017.dat

     CSQY_DATA         |  /home/mpas-cmaq-v1.1/MPAS_v7.0/src/core_atmosphere/cmaq/CSQY_DATA_cracmm2

     OPTICS_DATA       |  /home/mpas-cmaq-v1.1/MPAS_v7.0/src/core_atmosphere/cmaq/PHOT_OPTICS.dat

       NaN or Infinity detected on processor 013


       During        VDIFF :  Gas        NaN  | Aer  1.031E+09  | Non  1.382E-04

Note: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG IEEE_DIVIDE_BY_ZERO IEEE_OVERFLOW_FLAG IEEE_UNDERFLOW_FLAG IEEE_DENORMAL

     OMI               |  /raid/mpas-cmaq/2017_120km/omi_cmaq_2017.dat

     CSQY_DATA         |  /home/mpas-cmaq-v1.1/MPAS_v7.0/src/core_atmosphere/cmaq/CSQY_DATA_cracmm2

     OPTICS_DATA       |  /home/mpas-cmaq-v1.1/MPAS_v7.0/src/core_atmosphere/cmaq/PHOT_OPTICS.dat

     CSQY_DATA         |  /home/mpas-cmaq-v1.1/MPAS_v7.0/src/core_atmosphere/cmaq/CSQY_DATA_cracmm2

     OPTICS_DATA       |  /home/mpas-cmaq-v1.1/MPAS_v7.0/src/core_atmosphere/cmaq/PHOT_OPTICS.dat

     OMI               |  /raid/mpas-cmaq/2017_120km/omi_cmaq_2017.dat

     CSQY_DATA         |  /home/mpas-cmaq-v1.1/MPAS_v7.0/src/core_atmosphere/cmaq/CSQY_DATA_cracmm2

     CSQY_DATA         |  /home/mpas-cmaq-v1.1/MPAS_v7.0/src/core_atmosphere/cmaq/CSQY_DATA_cracmm2

     OPTICS_DATA       |  /home/mpas-cmaq-v1.1/MPAS_v7.0/src/core_atmosphere/cmaq/PHOT_OPTICS.dat

     OPTICS_DATA       |  /home/mpas-cmaq-v1.1/MPAS_v7.0/src/core_atmosphere/cmaq/PHOT_OPTICS.dat

     OMI               |  /raid/mpas-cmaq/2017_120km/omi_cmaq_2017.dat

     OMI               |  /raid/mpas-cmaq/2017_120km/omi_cmaq_2017.dat