Issues with CMAQ-DDMv5.4 sensitivity output

Hy,
I am using CMAQ-DDMv5.4 to calculate first order and second order sensitivity for NOx and SO2 emissions from ptegu source for the year of 2020. My log files showing the " PROGRAM COMPLETED SUCCESSFULLY" and I did not see any error in the log files. The issue is my ASENS output file for first order NOx sensitivity on May 12, 2020 has some concentration. However, after May 12 the NOx values for the rest of the year has 0 concentration. On the other hand, when I look at the SO2 sensitivity, it has some concentration for all the days of 2020. Any idea why I am seeing this and how to resolve this? Please see the attached image and my job script, log files.

sensinput.ptegu.dat.txt (2.4 KB)
CTM_LOG_001.v54_DDM3D_gcc_AQF5X_20200513.txt (688.1 KB)
CTM_LOG_001.v54_DDM3D_gcc_AQF5X_20200512.txt (708.8 KB)
run_cctm_hddm_egu.csh_sbatch.txt (41.2 KB)


Regards,
Rasel

Hi Rasel,

looking at your plots, I don’t think this statement is correct:

However, after May 12 the NOx values for the rest of the year has 0 concentration

VERDI might show a color bar with all zeros for NO2_ENX for May 13, but the values in the file aren’t actually zero (otherwise they’d be plotted in grey) but rather missing, infinity, or BADVAL3 - the min and max values are reported as infinity. Try looking at this variable in ncview and see what values are being shown, and do the same thing for May 12. From the May 12 plot, it appears some bad (i.e. white) NO2_ENX values are already showing up in the Northeastern portion of your domain, and these might be spreading throughout your domain as time goes on and are then passed on to the next day through the SENSGRID restart file.

I don’t know what might be causing these bad values, but finding out when and where they are first occurring and how they are spreading would be a first step in tracking down the underlying cause.

Hy @hogrefe.christian , you are right. The bad (ie white) NO2_ENX values started to generated from 24th timestep of 11th May, 2020 and spreading throughout domain as time goes on and are then passed on to the next day through the SENSGRID restart file.

Any suggestion why this is happening or how to resolve this issue?

Similar behavior is found on 2nd order NOx sensitivity starting from March 24th, 2020.

I don’t have DDM-specific advice, but my general approach in this situation would be to go back to the beginning of your simulation, use a tool like m3stat to scan the ASENS output files for each day, identify when unreasonable values first appeared, and then compile a debug executable and run the day in question with that executable to see whether this points to any errors in the calculations or input files.

It does look like you have some instability in your 2nd order output. A few causes for this have been identified and addressed in the “5.4+” version of the code. I recommend you try that version and see if they problem is solved. If not, report back, and I will try to see if can I dig something up.

Sergey

Hy @sergey ,
I have installed CMAQv5.4.0.3 and reran model starting from May 10, 2020. However, I see the issue still persist. The model starts to generate bad value starting from 24th time step of May 11, 2020 and spreading throughout domain as time goes on [see the above plots]. Any other suggestions?
Thanks
Rasel

Hi Rasel,

I am sorry you are still having issues with the output. I could try to help you figure out the issue, but we need to isolate some smaller and more manageable case since it is not feasible for me to run 6 months of simulation to get to the problem.

To start, did you restart the model on May 10 using clean initial conditions (and thus starting sensitivity and zero)? If so, that could be a fine case for me to work with since its only 1 day of simulation.

Sergey

@sergey
Hy Sergey,
I did not restart the model using clean initial conditions. I started model run on May 11th and used May 10 output as restart file.
Should I run the model using clean initial condition? In that case, should I start from May 1st for 10 day spin up period?

Thanks
Rasel

Yes - that sounds like a good plan.

Sergey

HY Sergey,

Still I am having the same issue. I ran the job with 10 days spinup period from 2020-06-20 to 2020-09-30, however on 2020/06/30 the model started to give the same bad value. Please see the attached image below. I have used CMAQv5.4.0.3. Any suggestion how to resolve this?

Thanks
Rasel

H Rasel,

I think the best course of action would be for me to try to replicate your issues on our system here. Is there a way you could share the 10 days of inputs?

Sergey

Hy Sergey,

I am not sure how to share the large size of data . For example, our MCIP output is alone more than 3TB 9 (for whole year) . Any suggestions?

Thanks
Rasel

Hi, I meet the same question. Do you have found any idea about this problem?

Hi Rasel,

Some users have shared data through 3rd party hosting services. Also, I don’t need the full year of inputs. Just the subset that starts from clean initial conditions and develops in the issue you describe. Can you run some tests to try to cut down the number of days required?

Sergey