Combine run issue

My mistake. Now I’ve changed the file in the CMAQ_REPO directory (REPO_HOME/POST/combine/scripts/spec_def_files/SpecDef_Dep_{MECH}.txt) and it resolved the issue. However, I guess I’m facing another similar problem. Now the error is saying:
ERROR Invalid syntax for field: CLNO2[2]

ERROR Cannot read CLNO2 from INFILE2

Here’s the attached log filecombine.aconc.log.txt (170.3 KB)

Thanks for confirming that this fixed the initial model crash due to the comment field that should not have been there.

The new error you encounter is caused by the fact that the definition for WDEP_TCL appears to reference a species (CLNO2) that is not part of the CCTM_WETDEP file. Unless you are really interested in WDEP_TCL and TDEP_CL, you could change these two lines:

WDEP_TCL ,kg/ha ,0.972*HCL[2]+0.435*CLNO2[2] + ACLJ[2] + ACLK[2]
TDEP_CL ,kg/ha ,DDEP_ACLJK[0] + WDEP_TCL[0]

to

!WDEP_TCL ,kg/ha ,0.972*HCL[2]+0.435*CLNO2[2] + ACLJ[2] + ACLK[2]
!TDEP_CL ,kg/ha ,DDEP_ACLJK[0] + WDEP_TCL[0]

or remove them altogether.

Alternatively, you could change

WDEP_TCL ,kg/ha ,0.972*HCL[2]+0.435*CLNO2[2] + ACLJ[2] + ACLK[2]

to

WDEP_TCL ,kg/ha ,0.972*HCL[2]+ ACLJ[2] + ACLK[2]

and then keep in mind that your definition of WDEP_TCL would not include any contribution from CLNO2 which isn’t available in your CCTM_WETDEP files

If you really wanted to include that species in your CCTM_WETDEP files and analysis, you would have to edit GC_cb6r3_ae6nvPOA_aq.nml as follows and then rerun CMAQ:

Change

'CLNO2:81.5:::::::::::Yes:::Yes',

to

'CLNO2:81.5:::::::::::Yes:Yes:Yes:Yes',

3 Likes

I’ve another question. I’ve also CMAQ DDM sensitivity files. Now if I want to calculate PM2.5 using my ASENS file, it will not work since variable names won’t match with SpecDef file names (see the attached picture). Is there any way I can resolve this issue? I want to calculate PM using sensitivity files. Also would like to see deposition using sensitivity files.

You’ll have to create new SpecDef files that use the variable names as they exist in your DDM files.

For example, if you’re interested in total I/J mode sulfate from the “ENX” tag and would like to call it “ASO4IJ_ENX”, you’d use a line like this (assuming your ASENS file is referenced as INFILE1):

ASO4IJ_ENX ,ug/m3 ,ASO4I_ENX[1]+ASO4J_ENX[1]

As a starting point, you can use the SpecDef and SpecDef_Dep files used to process regular concentration and deposition fields and then append the tag names and remove any aggregate species you may not be interested in and/or may not have available in your DDM output files.

1 Like

got it. Thank you for the help

Hi there,
Could anyone help with combine error I have? The problem seems known, but I don’t know how to resolve it. So any tip is appreciated. I use CMAQv5.2.
My CMAQ output has only 1-verical layer, but WRF model has 37-layers. Now, combine could not process the run, reporting the following error:
=====>
“INFILE1” opened as OLD:READ-ONLY
File name “/aqpmodels/CMAQ/output/YVPM25emiuptd_Oct2019_CCTM_v52_gcc_EMAQ1p3km/CCTM_ACONC_v52_gcc_EMAQ1p3km_20191027.nc”
File type GRDDED3
Execution ID “CCTM_v52.exe”
Grid name “EMAQ1p3km”
Dimensions: 100 rows, 88 cols, 1 lays, 233 vbles
NetCDF ID: 65536 opened as READONLY
Starting date and time 2019300:080000 (8:00:00 Oct. 27, 2019)
Timestep 010000 (1:00:00 hh:mm:ss)
Maximum current record number 24

 "INFILE2" opened as OLD:READ-ONLY
 File name "/home/tes/Documents/WRF/5.2/data/EMAQ1p3km/mcip/METCRO3D_191027.nc"
 File type GRDDED3
 Execution ID "mcip"
 Grid name "METCRO_EMAQ1p3_C"
 Dimensions: 100 rows, 88 cols, 37 lays, 17 vbles
 NetCDF ID:    131072  opened as READONLY
 Starting date and time  2019300:080000 (8:00:00   Oct. 27, 2019)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number        25

Error Inconsistence file domain for INFILE2
ERROR Cannot open all input files
<====

I’m guessing from the name of your directory path (“EMAQ1p3km”) that your grid resolution is 1.3333 km and that there may be very slight differences in the XCELL and/or YCELL attributes between the CCTM_ACONC file opened as INFILE1 and the METCRO3D files opened as INFILE2. You could verify this by doing an ncdump -h on both files and comparing the grid attributes.

There is a known shortcoming of the portion of combine that ensures consistency of grids across files before combining variables from the different files. This shortcoming affects domains with “odd” grid spacing and/or grid origin values. There are multiple examples of this issue in this thread, along with suggested code changes to solve this issue by relaxing the tolerance criteria or making these criteria relative rather than absolute (the more robust solution).

Thank you, @hogrefe.christian:
You diagnosed the problem correctly. And following your linked thread above, I was able to change the two lines from absolute to relative conditions, cleaned the make and re-built it. Combine is now running normally!
Thank you,
Tes

@hogrefe.christian

Hy,

I have two issues after calculating PM2.5 from DDM output files in CMAQv5.4

  1. I have modified specdef file based on my DDM sensitivity parameters. After running combine post processing tools, I looked into my slurm job output file. It’s saying:

When I looked into AELMO file (input file 3), I could not find AMSAT variable. Any suggestion, how to resolve this?

  1. I also found another error in my slurm job output, it’s saying:

When I looked into my INFILE1 (Sensitivity Dry deposition file), I see the variable NO2_2NS is available in the file. Any suggestion how to resolve this?

  1. Can you have a look into my sensitivity file that I am using for DDM run for any formating error? I am trying to do HDDM sensitivity analysis on ptegu emissions for NOx, and SO2 emissions. Any other suggestion would be highly appreciated.
    sensinput.ptegu.dat.txt (2.4 KB)

Regards,
Rasel

Hi Rasel,

  1. If you really do need PM species in the AMS cutoff range for your analysis, the relevant v54 AELMO cutoff variables replacing v533 APMDIAG variables AMSAT, AMSAC, and AMSCO are FAMSAIT, FAMSACC, and FAMSCOR, analogous to the FPM25 cutoff variables replacing the PM25 cutoff variables. See user guide appendix F for more ELMO documentation. Alternatively, you could consider commenting out these calculations if you’re not planning on analyzing sensitivities of PM in the AMS instrument measurement range.

  2. To look at this further, please post your run script, SpecDef files, and log file.

  3. I don’t have a lot of experience with running DDM, but at first glance your sensinput file looks fine to me. And given that you were able to run CCTM and generate DDM output files, any issues with your post-processing setup are unlikely to be caused by any problems with our sensinput file.

Thank you @hogrefe.christian.

Please see the attached combine runscript, SpecDef file, and slurm job output file for issues with NO2_2NS error.

Regards,
Rasel
SpecDef_sens_egus_2NS_dep_cmaqv54_ts_cb6r3_ae7_aq.txt (10.0 KB)
2NS_asens_combine_cmaqv54_dep_conc_cb6r3_ae7_aq_batch_588205.out.txt (1.6 MB)
run_combine_asens_egus_2NS.csh.txt (9.8 KB)

Thanks for posting these files.

So, the problem for the DEP file processing is that the SENDDEP files apparently do not contain valid TFLAG data for variable NO2_2NS (and possibly other variables) for the very first time step, to the program just quits at that point:

 >>--->> WARNING in subroutine RDTFLAG
 Time step not available in file INFILE1          for variable NO2_2NS
 M3WARN:  DTBUF 1:00:00   Dec. 20, 2019 (2019354:010000)

**ERROR** Invalid syntax for field: NO2_2NS[1]

**ERROR** Cannot read NO2_2NS from INFILE1

What do you see when you type

ncdump -v TFLAG /scratch/mrasel/cmaq-5.4/data_output/cctm_out/CCTM_SENDDEP_v54_DDM3D_gcc_AQF5X_20191220.nc | more

valid time steps, or just _, _ ? Can you run this file through m3stat without error?

I see following image when I run ncdump -v TFLAG /scratch/mrasel/cmaq-5.4/data_output/cctm_out/CCTM_SENDDEP_v54_DDM3D_gcc_AQF5X_20191220.nc | more


And I get following error when I run m3stat on the file:

I was curious what you see after ncdump -v TFLAG shows all the header information with the variable definitions and file attributes, not what the header looks like. To get there, keep hitting the space bar in the command suggested above until you get to the data / TFLAG section and see something like the following (or likely things like _, _) if there is no valid data for TFLAG:
" ;
:HISTORY = “” ;
data:

TFLAG =
 2010026, 0,
 2010026, 0,

But given that your m3stat test already failed, I’m now pretty sure that this isn’t a post-processing issue, but an issue of bad / missing data in your SENSDDEP file. After confirming that that’s the case, you probably want to open a separate thread discussing the problems with your SENSDDEP file and asking people with more DDM experience to weigh in.

This seems to me like a (deliberately?) truncated portion of the m3stat log; immediately before the

Value for PROMPTFLAG not defined;returning default: TRUE

there should have been a report about the properties of INFILE – something like

 `"INFILE" opened as OLD:READ-ONLY`   
 `File name "/work/LSM/WADLEY/RTE_OUT_2D.WADLEY_RTE.2010035180000.18.ncf"`
 `File type GRDDED3` 
 `Execution ID "Coats:  private testing"`
 `Grid name "WADLEY_RTE"`
 `Dimensions: 3320 rows, 2760 cols, 1 lays, 8 vbles`
 `NetCDF ID:     65536  opened as READONLY`            
 `Starting date and time  2010035:183000 (18:30:00  Feb. 4, 2010)`
 `Timestep                          003000 (0:30:00 hh:mm:ss)`
 `Maximum current record number        95`

It would have been interesting to see the time-sequence properties of your file. Moreover, your ncdump -v TFLAG output is seriously incomplete: I don’t see the “data” section part of the results (and that is what is truly significant); that command when applied to the file above finishes with:

data:

TFLAG =
2010035, 183000,
2010035, 183000,
2010035, 183000,
2010035, 183000,
2010035, 183000,
2010035, 183000,
2010035, 183000,
2010035, 183000,
2010035, 190000,
2010035, 190000,
2010035, 190000,
2010035, 190000,
2010035, 190000,
2010035, 190000,
2010035, 190000,
2010035, 190000,
2010035, 193000,

Actually, an interesting question is: what is the origin of this file? Does it come from a proper I/O API client program? If not: The I/O API is a programming interface, not a data format. (Analogously, MS-Word creates files; if you properly understood the corresponding data format, you should be able to create, read, and edit them with a binary hex-editor (instead of using either the WORD program or the wxlib programming library). Is that what you do ???

Hy,

Here’s the ncdump output of my sensitivity dry deposition file. I did not understand your question about origin of the file. I am using CMAQv5.4 DDM HDDM sensitivity analysis on ptegu emissions and from that I get ASENS, dry and wet deposition sensitivity files. When I use Combine post processing tools (SpecDef, job script and slurm job output files uploaded above), I get an error NO2_2NS file not found in dry deposition sensitivity file. When I look into the file, the variable is there.

Please see the attached ncdump output for TFLAGS data.

Regards,
Rasel
tflag.txt (412.8 KB)

Thanks for posting the full ncdump -v TFLAG output. This now shows that the SENDDEP file indeed has a problem, i.e. that the CCTM HDDM run somehow only wrote out every fifth sensitivity variable successfully, i.e. it wrote out NO2_ENX, NO_ENX, O3_ENX, but not NO2_ES2, NO2_2NX, NO2_2S2, NO2_2NS, NO_ES2, NO_2NX, etc.

When you say “When I look into the file, the variable is there”, do you mean that you can see non-missing fields of NO2_ES2, NO2_2NX, etc. using visualization tools like ncview or VERDI that do not require valid TFLAG data? Or do you simply mean that the header information from ncdump shows that the variable is defined in the file, but not that you were able to confirm that the file actually contains valid data for that variable?

Since this problem clearly originated when running CMAQv5.4 HDDM (though your run didn’t crash), I’m repeating my advice above to open a new thread saying that your CMAQv5.4 HDDM simulation created a SENDDEP file with (partially) missing data. This is not a combine issue.

In the meantime, you might also want to carefully examine your CTM_LOG files to see if there were any warnings or errors when CMAQv5.4 was writing to the SENDDEP file at the top of each hour.

Given that in one of your earlier threads you had mentioned using bidi and having to t-shift the EPIC-generated input files required by the bidi module, I’m wondering if somehow the strange CMAQ HDDM behavior for these SENDDEP files is related to that, but there may not be any connection at all. Others with more DDM knowledge hopefully will have a better sense of what might be happening here.

Thank you for your suggestion. Now I see, Indeed, SENDDEP file has a problem. The ASENS TFLAG output seems fine to me. However, SENDDEP file’s TFLAG is missing for most of the sensitivity parameter (ES2, 2NX, 2S2, 2NS). It wrote out every fifth sensitivity variable successfully.

When I said, “When I look into the file, the variable is there”, I meant that the header information from ncdump shows that the variable is defined in the file, but not that I was able to confirm that the file actually contains valid data for that variable. When I use verdi to visualize dry deposition sentivity data, I see the plot was available for the variable for my ENX (first order sensitivity, the one for which model wrote data successfully), however, for others (such as ES2, 2NX, 2S2, 2NS) plot is showing infinity. Here’s a sample.

I have looked into my log file, I am not seeing any errors. When I use grep -i error command it’s giving following:
CTM_LOG_179.v54_DDM3D_gcc_AQF5X_20191230: Performing Basic Error Checks for Emission Scaling Rules
CTM_LOG_179.v54_DDM3D_gcc_AQF5X_20200101: Timestep 718773184 ( hh:mm:ss)
The timerror is coming from ammonia data. Please see the attached image.

I am also attaching my job script if you want to have a look for anything else.
run_cctm_hddm_egu.csh_sbatch.txt (41.2 KB)
CTM_LOG_179.v54_DDM3D_gcc_AQF5X_20200101.txt (746.1 KB)
SENDDEP_TFLAG.txt (412.8 KB)
ASENS_TFLAG.txt (743.4 KB)
sensinput.ptegu.dat.txt (2.4 KB)

I have another quick question. If you look into my combine slurm job output script (attached here), at some point it’s saying:
Problem evaluating: AOMIJ[0]/AOCIJ[0]
Denominator, AOCIJ[0], equals zero.

Is it going to be an issue for PM2.5 calculation? I am also attaching SpecDef script here,
SpecDef_asens_egus_2NS_cmaqv54_cb6r3_ae7_aq.txt (17.3 KB)

2NS_asens_combine_cmaqv54_dep_conc_cb6r3_ae7_aq_batch_590327.out.txt (813.5 KB)

Yes, what you see in VERDI is consistent with the missing TFLAG variable for all DDEP sensitivity parameters except for the first one (_ENX). Apparently CMAQv5.4 HDDM failed to write out fields for the remaining four sensitivity parameters.

I still do not know why that might be, and I still suggest you open a new thread dedicated to that issue. That said, the fact that you are running bidi, that you had to manipulate the 2019 EPIC-generated required files to run for 2020, that you are seeing the error below in your log file, and that the problem is showing up in deposition might not be a coincidence, so I suggest that in your new post you provide the context of how you went about setting up your 2020 bidi DDM run without having EPIC-generated 2020 input files, maybe linking to your other thread where this was discussed.

This error in your log file might be relevant:

 "E2C_SOIL" opened as OLD:READ-ONLY   
 File name "/scratch/mrasel/cmaq-5.4/data_input/ammonia/2020r1_EPIC0509_12US1_soil.nc4"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 42 lays, 13 vbles
 NetCDF ID:   1703936  opened as READONLY            
 Starting date and time        0:000000 (0:00:00   Jan. 0, -000)
 Timestep                       718773184 (<TIMERROR> hh:mm:ss)
 Maximum current record number         1

Clearly CMAQ does not open this file quite as intended - how did you generate it? the 2019 EQUATES EPIC0509_12US1_soil.nc4 file you downloaded likely was a time-independent file, with SDATE, STIME, and TSTEP all set to 0 - did you run it through m3tshift along with all the daily time-dependent EPIC0509_12US1_time2019… files? That may have caused a problem.

Again, I don’t have further insights, it’s better to discuss this in a separate thread with people who know both the DDM and bidi code and related input file requirements.

Regarding the combine run for ASENS, the warning “Problem evaluating: AOMIJ[0]/AOCIJ[0] Denominator, AOCIJ[0], equals zero.” only occurred once in your log file, on the very first day. I’m guessing that ASENS parameters for the very first hour on the first day might be zero, so you probably would want to comment out this calculation of the OM/OC ratio since the calculated OC aggregate (representing a sensitivity, not an actual concentrations like when working off of ACONC files) would be zero. Based on the objectives of the study you described, it seems that calculating ratios of OM/OC sensitivities wouldn’t be a high priority anyway.