Combine run issue

[hogrefe.christian]

Hi Rasel,

1. If you really do need PM species in the AMS cutoff range for your analysis, the relevant v54 AELMO cutoff variables replacing v533 APMDIAG variables AMSAT, AMSAC, and AMSCO are FAMSAIT, FAMSACC, and FAMSCOR, analogous to the FPM25 cutoff variables replacing the PM25 cutoff variables. See user guide appendix F for more ELMO documentation. Alternatively, you could consider commenting out these calculations if you’re not planning on analyzing sensitivities of PM in the AMS instrument measurement range.

2. To look at this further, please post your run script, SpecDef files, and log file.

3. I don’t have a lot of experience with running DDM, but at first glance your sensinput file looks fine to me. And given that you were able to run CCTM and generate DDM output files, any issues with your post-processing setup are unlikely to be caused by any problems with our sensinput file.

[munshimdrasel]

Thank you @hogrefe.christian.

Please see the attached combine runscript, SpecDef file, and slurm job output file for issues with NO2_2NS error.

Regards,
Rasel
SpecDef_sens_egus_2NS_dep_cmaqv54_ts_cb6r3_ae7_aq.txt (10.0 KB)
2NS_asens_combine_cmaqv54_dep_conc_cb6r3_ae7_aq_batch_588205.out.txt (1.6 MB)
run_combine_asens_egus_2NS.csh.txt (9.8 KB)

[hogrefe.christian]

Thanks for posting these files.

So, the problem for the DEP file processing is that the SENDDEP files apparently do not contain valid TFLAG data for variable NO2_2NS (and possibly other variables) for the very first time step, to the program just quits at that point:

 >>--->> WARNING in subroutine RDTFLAG
 Time step not available in file INFILE1          for variable NO2_2NS
 M3WARN:  DTBUF 1:00:00   Dec. 20, 2019 (2019354:010000)

**ERROR** Invalid syntax for field: NO2_2NS[1]

**ERROR** Cannot read NO2_2NS from INFILE1

What do you see when you type

ncdump -v TFLAG /scratch/mrasel/cmaq-5.4/data_output/cctm_out/CCTM_SENDDEP_v54_DDM3D_gcc_AQF5X_20191220.nc | more

valid time steps, or just _, _ ? Can you run this file through m3stat without error?

[munshimdrasel]

I see following image when I run ncdump -v TFLAG /scratch/mrasel/cmaq-5.4/data_output/cctm_out/CCTM_SENDDEP_v54_DDM3D_gcc_AQF5X_20191220.nc | more

And I get following error when I run m3stat on the file:

[hogrefe.christian]

I was curious what you see after ncdump -v TFLAG shows all the header information with the variable definitions and file attributes, not what the header looks like. To get there, keep hitting the space bar in the command suggested above until you get to the data / TFLAG section and see something like the following (or likely things like _, _) if there is no valid data for TFLAG:
" ;
:HISTORY = “” ;
data:

TFLAG =
 2010026, 0,
 2010026, 0,

But given that your m3stat test already failed, I’m now pretty sure that this isn’t a post-processing issue, but an issue of bad / missing data in your SENSDDEP file. After confirming that that’s the case, you probably want to open a separate thread discussing the problems with your SENSDDEP file and asking people with more DDM experience to weigh in.

[cjcoats]

This seems to me like a (deliberately?) truncated portion of the m3stat log; immediately before the

Value for PROMPTFLAG not defined;returning default: TRUE

there should have been a report about the properties of INFILE – something like

 `"INFILE" opened as OLD:READ-ONLY`   
 `File name "/work/LSM/WADLEY/RTE_OUT_2D.WADLEY_RTE.2010035180000.18.ncf"`
 `File type GRDDED3` 
 `Execution ID "Coats:  private testing"`
 `Grid name "WADLEY_RTE"`
 `Dimensions: 3320 rows, 2760 cols, 1 lays, 8 vbles`
 `NetCDF ID:     65536  opened as READONLY`            
 `Starting date and time  2010035:183000 (18:30:00  Feb. 4, 2010)`
 `Timestep                          003000 (0:30:00 hh:mm:ss)`
 `Maximum current record number        95`

It would have been interesting to see the time-sequence properties of your file. Moreover, your ncdump -v TFLAG output is seriously incomplete: I don’t see the “data” section part of the results (and that is what is truly significant); that command when applied to the file above finishes with:

data:

TFLAG =
2010035, 183000,
2010035, 183000,
2010035, 183000,
2010035, 183000,
2010035, 183000,
2010035, 183000,
2010035, 183000,
2010035, 183000,
2010035, 190000,
2010035, 190000,
2010035, 190000,
2010035, 190000,
2010035, 190000,
2010035, 190000,
2010035, 190000,
2010035, 190000,
2010035, 193000,
…

Actually, an interesting question is: what is the origin of this file? Does it come from a proper I/O API client program? If not: The I/O API is a programming interface, not a data format. (Analogously, MS-Word creates files; if you properly understood the corresponding data format, you should be able to create, read, and edit them with a binary hex-editor (instead of using either the WORD program or the wxlib programming library). Is that what you do ???

[munshimdrasel]

Hy,

Here’s the ncdump output of my sensitivity dry deposition file. I did not understand your question about origin of the file. I am using CMAQv5.4 DDM HDDM sensitivity analysis on ptegu emissions and from that I get ASENS, dry and wet deposition sensitivity files. When I use Combine post processing tools (SpecDef, job script and slurm job output files uploaded above), I get an error NO2_2NS file not found in dry deposition sensitivity file. When I look into the file, the variable is there.

Please see the attached ncdump output for TFLAGS data.

Regards,
Rasel
tflag.txt (412.8 KB)

[hogrefe.christian]

Thanks for posting the full ncdump -v TFLAG output. This now shows that the SENDDEP file indeed has a problem, i.e. that the CCTM HDDM run somehow only wrote out every fifth sensitivity variable successfully, i.e. it wrote out NO2_ENX, NO_ENX, O3_ENX, but not NO2_ES2, NO2_2NX, NO2_2S2, NO2_2NS, NO_ES2, NO_2NX, etc.

When you say “When I look into the file, the variable is there”, do you mean that you can see non-missing fields of NO2_ES2, NO2_2NX, etc. using visualization tools like ncview or VERDI that do not require valid TFLAG data? Or do you simply mean that the header information from ncdump shows that the variable is defined in the file, but not that you were able to confirm that the file actually contains valid data for that variable?

Since this problem clearly originated when running CMAQv5.4 HDDM (though your run didn’t crash), I’m repeating my advice above to open a new thread saying that your CMAQv5.4 HDDM simulation created a SENDDEP file with (partially) missing data. This is not a combine issue.

In the meantime, you might also want to carefully examine your CTM_LOG files to see if there were any warnings or errors when CMAQv5.4 was writing to the SENDDEP file at the top of each hour.

Given that in one of your earlier threads you had mentioned using bidi and having to t-shift the EPIC-generated input files required by the bidi module, I’m wondering if somehow the strange CMAQ HDDM behavior for these SENDDEP files is related to that, but there may not be any connection at all. Others with more DDM knowledge hopefully will have a better sense of what might be happening here.

[munshimdrasel]

Thank you for your suggestion. Now I see, Indeed, SENDDEP file has a problem. The ASENS TFLAG output seems fine to me. However, SENDDEP file’s TFLAG is missing for most of the sensitivity parameter (ES2, 2NX, 2S2, 2NS). It wrote out every fifth sensitivity variable successfully.

When I said, “When I look into the file, the variable is there”, I meant that the header information from ncdump shows that the variable is defined in the file, but not that I was able to confirm that the file actually contains valid data for that variable. When I use verdi to visualize dry deposition sentivity data, I see the plot was available for the variable for my ENX (first order sensitivity, the one for which model wrote data successfully), however, for others (such as ES2, 2NX, 2S2, 2NS) plot is showing infinity. Here’s a sample.

I have looked into my log file, I am not seeing any errors. When I use grep -i error command it’s giving following:
CTM_LOG_179.v54_DDM3D_gcc_AQF5X_20191230: Performing Basic Error Checks for Emission Scaling Rules
CTM_LOG_179.v54_DDM3D_gcc_AQF5X_20200101: Timestep 718773184 ( hh:mm:ss)
The timerror is coming from ammonia data. Please see the attached image.

I am also attaching my job script if you want to have a look for anything else.
run_cctm_hddm_egu.csh_sbatch.txt (41.2 KB)
CTM_LOG_179.v54_DDM3D_gcc_AQF5X_20200101.txt (746.1 KB)
SENDDEP_TFLAG.txt (412.8 KB)
ASENS_TFLAG.txt (743.4 KB)
sensinput.ptegu.dat.txt (2.4 KB)

I have another quick question. If you look into my combine slurm job output script (attached here), at some point it’s saying:
Problem evaluating: AOMIJ[0]/AOCIJ[0]
Denominator, AOCIJ[0], equals zero.

Is it going to be an issue for PM2.5 calculation? I am also attaching SpecDef script here,
SpecDef_asens_egus_2NS_cmaqv54_cb6r3_ae7_aq.txt (17.3 KB)

2NS_asens_combine_cmaqv54_dep_conc_cb6r3_ae7_aq_batch_590327.out.txt (813.5 KB)

[hogrefe.christian]

Yes, what you see in VERDI is consistent with the missing TFLAG variable for all DDEP sensitivity parameters except for the first one (_ENX). Apparently CMAQv5.4 HDDM failed to write out fields for the remaining four sensitivity parameters.

I still do not know why that might be, and I still suggest you open a new thread dedicated to that issue. That said, the fact that you are running bidi, that you had to manipulate the 2019 EPIC-generated required files to run for 2020, that you are seeing the error below in your log file, and that the problem is showing up in deposition might not be a coincidence, so I suggest that in your new post you provide the context of how you went about setting up your 2020 bidi DDM run without having EPIC-generated 2020 input files, maybe linking to your other thread where this was discussed.

This error in your log file might be relevant:

 "E2C_SOIL" opened as OLD:READ-ONLY   
 File name "/scratch/mrasel/cmaq-5.4/data_input/ammonia/2020r1_EPIC0509_12US1_soil.nc4"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 42 lays, 13 vbles
 NetCDF ID:   1703936  opened as READONLY            
 Starting date and time        0:000000 (0:00:00   Jan. 0, -000)
 Timestep                       718773184 (<TIMERROR> hh:mm:ss)
 Maximum current record number         1

Clearly CMAQ does not open this file quite as intended - how did you generate it? the 2019 EQUATES EPIC0509_12US1_soil.nc4 file you downloaded likely was a time-independent file, with SDATE, STIME, and TSTEP all set to 0 - did you run it through m3tshift along with all the daily time-dependent EPIC0509_12US1_time2019… files? That may have caused a problem.

Again, I don’t have further insights, it’s better to discuss this in a separate thread with people who know both the DDM and bidi code and related input file requirements.

Regarding the combine run for ASENS, the warning “Problem evaluating: AOMIJ[0]/AOCIJ[0] Denominator, AOCIJ[0], equals zero.” only occurred once in your log file, on the very first day. I’m guessing that ASENS parameters for the very first hour on the first day might be zero, so you probably would want to comment out this calculation of the OM/OC ratio since the calculated OC aggregate (representing a sensitivity, not an actual concentrations like when working off of ACONC files) would be zero. Based on the objectives of the study you described, it seems that calculating ratios of OM/OC sensitivities wouldn’t be a high priority anyway.

[hogrefe.christian]

I have flagged this issue with your SENDDEP files to a colleague. There may indeed be a bug in CMAQ with respect to writing DDM deposition outputs to this file, and we will look at this further. In the meantime, it would be good if you could start a new thread describing the issue you’re seeing with the SENDDEP files in your application, so any future discussion of this issue could go there.