CMAQ-DMv.4.0.3 Sensitivity output during simulation becomes null value from a certain moment

Hello Anybody

I saw someone in the community reporting this issue, but it hasn’t been resolved(Issues with CMAQ-DDMv5.4 sensitivity output - #13 by bandaoshutiao). I believe this should be an occasional but real exist problem.

All concentration files are correct, and the logs have not reported any errors. However, one day after the simulation started, mine was on the 10th day, O3_ ANX starts from a certain grid point in the simulation domain to 0. As the simulation continued, after another 7 days, O3_ BCO also began to become zero. There are also issues with other variables. Given that there were no issues with the results at first few days of the simulation, I guess my model settings should not have been wrong.







Has anyone else encountered the same problem? Do you have any suggestions? Or can I send the file to you? @sergey

Ragrd
shutiao

I have not been able to replicate this behavior on our local system here. Is there any way for you to share the INPUTS so that I may try your exact case? The best would be the smallest number of simulation days that start from clean initial conditions and progress in the the first occurrence of the problem.

Sergey

Hi Sergey

Thank you for your attention and response to this question.
I shared my file on google drive, which includes scripts and input files.

The script runs as follows: csh xxx 20210530

I look forward to your reply very much. If you have any questions, please feel free to contact me at any time. @sergey

shutiao

Arfter I change exe to 5.3.2,the error will be disappear. So, I guess some change have happened between 5.4.0.3 and 5.3.2

Great! So your simulation proceeds normally with the newest code? Is that correct?

Sergey

@sergey, no, unfortunately I think @bandaoshutiao is encountering the problem with the newest code (CMAQv5.4.0.3), but can avoid this specific problem when downgrading to CMAQv5.3.2.

1 Like

Hi Sergey and Christian,

I have successfully executed CMAQ/DDM v5.2 and CMAQ/DDM v5.4.0.3 and obtained satisfactory results. In my practices, other versions of CMAQ, including CMAQ 5.3.2 with the DDM-3D, do not appear to be fully functional.

CMAQ/DDM v5.4 exhibits significantly elevated higher-order sensitivities. However, these issues have been addressed and resolved in CMAQ v5.4.0.3, as seen here. I do not have the problem posted here in utilizing CMAQ v5.4.0.3 with DDM-3D.
Thanks.

Feng

I am also seeing same issue with the CMAQv5.4.0.3 model and still waiting for a right solution.

@sergey Hy Sergey, were you able to replicate the issue using the data provided above? Any suggestion would be highly appreciated.

Regards,
Rasel

I have the data locally and I will post back here when I find some time to try it out.

Sergey

I ran this case locally and was able to trigger an instability in O3_ANX at hour 21. The rest of my results were without error. This is different from the other case reported here, but hopefully good enough to investigate. I will try to figure out what is causing it and will report back.

Sergey

@bandaoshutiao

I found that the restart SENGRID file for the case you sent me already has bad value for some VS* species. Here is one example from m3stat command on SENGRID:

 Variable:  SVAVB1_BVX      
      3-D grid statistics
     Max    1.30980E+22 @(c,r,l)=(172,61,20)
     Min   -2.38146E+22 @(c,r,l)=(171,60,17)
     Mean  -4.97627E+17
     Sigma     Infinity

Could you, please, go back further in time checking SENGRID restart files until you find a clean one?

Sergey

I am uploading file, pleasy wait a moment.

I’m sorry for replying so late, I was on vacation before.

This is the SENGRID file which do not have bad value

@sergey Hy Sergey, were you able to find anything on the issue?

Thanks
Rasel

I have not yet had the chance to try the new inputs, but it is my my list for this week.

Sergey

@bandaoshutiao

It is possible that the issue originates in the sensitivity treatment of the organic PM module. However, I will need your help to track it down. Could you, please, go through the sensitivity output and see when in your simulation time species relating to organic PM first begin to show unreasonable behavior. Examining only the “ANX” sensitivity is sufficient. These species are listed below.

Sergey

 Variable:  MTNO3_ANX       
 Variable:  VLVPO1_ANX      
 Variable:  VSVPO1_ANX      
 Variable:  VSVPO2_ANX      
 Variable:  VSVPO3_ANX      
 Variable:  VIVPO1_ANX      
 Variable:  VLVOO1_ANX      
 Variable:  VLVOO2_ANX      
 Variable:  VSVOO1_ANX      
 Variable:  VSVOO2_ANX      
 Variable:  VSVOO3_ANX      
 Variable:  SVAVB1_ANX      
 Variable:  SVAVB2_ANX      
 Variable:  SVAVB3_ANX      
 Variable:  SVAVB4_ANX      
 Variable:  AISO1J_ANX      
 Variable:  AISO2J_ANX      
 Variable:  ASQTJ_ANX       
 Variable:  AISO3J_ANX      
 Variable:  AOLGAJ_ANX      
 Variable:  AOLGBJ_ANX      
 Variable:  AGLYJ_ANX       
 Variable:  AMTNO3J_ANX     
 Variable:  AMTHYDJ_ANX     
 Variable:  APOCI_ANX       
 Variable:  APOCJ_ANX       
 Variable:  APNCOMI_ANX     
 Variable:  APNCOMJ_ANX     
 Variable:  APCSOJ_ANX      
 Variable:  ALVPO1I_ANX     
 Variable:  ALVPO1J_ANX     
 Variable:  ASVPO1I_ANX     
 Variable:  ASVPO1J_ANX     
 Variable:  ASVPO2I_ANX     
 Variable:  ASVPO2J_ANX     
 Variable:  ASVPO3J_ANX     
 Variable:  AIVPO1J_ANX     
 Variable:  ALVOO1I_ANX     
 Variable:  ALVOO1J_ANX     
 Variable:  ALVOO2I_ANX     
 Variable:  ALVOO2J_ANX     
 Variable:  ASVOO1I_ANX     
 Variable:  ASVOO1J_ANX     
 Variable:  ASVOO2I_ANX     
 Variable:  ASVOO2J_ANX     
 Variable:  ASVOO3J_ANX     
 Variable:  AAVB1J_ANX      
 Variable:  AAVB2J_ANX      
 Variable:  AAVB3J_ANX      
 Variable:  AAVB4J_ANX      
 Variable:  AMT1J_ANX       
 Variable:  AMT2J_ANX       
 Variable:  AMT3J_ANX       
 Variable:  AMT4J_ANX       
 Variable:  AMT5J_ANX       
 Variable:  AMT6J_ANX       
 Variable:  SVISO1_ANX      
 Variable:  SVISO2_ANX      
 Variable:  SVSQT_ANX       
 Variable:  LVPCSOG_ANX     
 Variable:  SVMT1_ANX       
 Variable:  SVMT2_ANX       
 Variable:  SVMT3_ANX       
 Variable:  SVMT4_ANX       
 Variable:  SVMT5_ANX       
 Variable:  SVMT6_ANX

Can it be accurately searched on a certain day, or should I view in all output files throughout the simulation period?

I would pick one of those variables and go back through the output files until you find a day that it breaks. Then, check the other variables to make sure they don’t break earlier.

Sergey

ok, please give me some time