AMET could not read model output

AMET
#1

Hi,

I am using AMET to analyze CMAQ output. It is a four-day simulation from CMAQ (01/11/2015 - 01/14/2015). I visualized these output files in Verdi and they looked normal. I can run the AMET program without any error but when I checked the output files (AQS_Daily_aq_20150111.csv (2.5 MB) or IMPROVE_aq_20150111.csv (95.7 KB) ), there was no data read from the model (all values are -999). I also found some warnings in the log file:

image
image697Ă—195 7.08 KB

The script I am using is aqProject_post_only.csh (8.1 KB) and the CMAQ output files can be downloaded from https://drive.google.com/drive/folders/1Nhv1F8DlFCyxRVvzgq0VDPvspUJ_B8cl?usp=sharing.

I will so appreciate it if you can take a look at this issue.

Thanks,
Qian

#2

OK, there will have been a log-message when M3_FILE_1 was opened. That log-message will give the file’s starting date, starting time, and number of time-steps. What are they?

#3

Did you mean the log file generated by aqProject_post_only.csh? It is attached here (https://drive.google.com/file/d/1R-EzfNZ_3RjFHKluTP6aY2sGrXSezBS0/view?usp=sharing) Thanks for spending time on this!

Qian

#4

Immediately above the indicated warning message in the log, you will see that there is a complete list of the variables in M3_FILE_1 from which you will further see that the indicated variable ASO4IJ is not one of the variables in the file. This indicates an inconsistency between your CCTM setup and your AMET setup for this case.

#5

So, the error you posted in the image indicates that the model species “ASO4IJ” was not found in your combine file.

Based on your run script, it looks like your input M3_Files are the raw CMAQ ACONC output files and not combine files. You need to run the program combine first to create a single file for the 01/11/2015-01/14/2015 period.

In addition to putting all the days into a single file, combine also creates “new” species for AMET to use. For example, the combine program creates the new species “ASO4IJ” from the “ASO4I” and “ASO4J” species in the raw CMAQ ACONC output files. The error you’re seeing is because AMET is looking for the species “ASO4IJ” in your input CMAQ files but can’t find them. This error will occur repeatedly for a number of species since AMET looks for a lot species that are not in the raw CMAQ output files and are only available after you run combine.

There are instructions in the AMET documentation on how to run combine for AMET. Please take a look at those instructions. You could use the aqProject_pre_and_post.csh to accomplish this. Once setup properly, that script will run combine for your project in addition to all the things that the aqProject_post_only.csh script does. You just need to point it to the location of your raw CMAQ output files.

Also note that when you look at your files in VERDI, you’ll see there is no species called “ASO4IJ”. Again this is because that species (and many others) is only available after you run combine.

Let me know if you need further assistance getting combine files created.

Wyat

2 Likes
#6

Thanks for pointing this out, @cjcoats and @wyat.appel.

I tried to run aqProject_pre_and_post.csh (30.7 KB) but got some errors about input files and the mySQL. The log file is atteched here (log.pre.txt (69.2 KB) )

image
image885Ă—201 8.46 KB

I am not sure what the file number refers to or how to set the script to skip running mySQL.

Thanks,
Qian

#7

Hi Qian,
Please try editing your aqProject_pre_and_post.csh to use the following settings:

## ==================================================================
#> 1. Select which analysis steps you want to execute
# ==================================================================

#> Combine and sitecmp options
 setenv RUN_COMBINE       T     #> T/F; Run combine on CCTM output?
 setenv RUN_HR2DAY        F     #> T/F; Run hr2day program (needed for analyzing TOAR network data)
 setenv WRITE_SITEX       T     #> T/F; Write scripts for running site compare for each selected network?
 setenv RUN_SITEX         T     #> T/F; Run site compare scripts for each selected network?

#> AMET options
 setenv CREATE_PROJECT    T     #> T/F; Create AMET project?
 setenv LOAD_SITEX        F     #> T/F; Load site compare output for each selected network into AMET?
 setenv UPDATE_PROJECT    F     #> T/F; Update the project info for an existing project (all data are retained)
 setenv REMAKE_PROJECT    F     #> T/F; Remake an existing AMET project. Note that all existing data will be deleted
 setenv DELETE_PROJECT    F     #> T/F; Delete an existing AMET project. This will delete all data in the existing
                                #>      AMET table and remove the table from the database

#> Plotting options
 setenv AMET_DB           F     #> T/F; Set to T if the model/obs pairs are loaded in the AMET database (i.e. by setting LOAD_SITEX = T)
                                #>      When set to F, plotting scripts will read the sitecmp .csv files directly
#8

Note: This also requires that the combine.exe executable has been built.

AMET provides a Makefile under the directory:
AMET_v14b/POST/combine/scripts

There is also a way to build the combine executable using a bldit_combine.csh that is provided under the CMAQ directory. If you have difficulty in getting combine to compile using the Makefile for AMET, you can go to your CMAQ build environment and look for the bldit script available there. The advantage of this, is that the I/O API and NETCDF libraries are already set up, and ready to use under your CMAQ directory:

CMAQv5.3.2/POST/combine/scripts/bldit_combine.csh

The main issue with the Makefile in the AMET distribution, is that it doesn’t specify both the netcdf fortran and netcdf c libraries, and of course, you must specify the correct locations for these libraries on your system.

NETCDF = -L$(LIB)/netcdff/lib  -L$(LIB)/netcdf/lib -lnetcdf

Change to:

NETCDF = -L$(LIB)/netcdff/lib -lnetcdff -L$(LIB)/netcdf/lib -lnetcdf
#9

Thanks for the reply Liz. In addition to setting those options, the script also needs to be updated where it calls the database and project generation in MySQL. Currently that call is made even if AMET_DB is set to F. So, changes need to be made similar to those I did to the aqProject_post_only.csh script. I’m working on an updated version of the aqProject_pre_and_post.csh script that I can provide. I should have that available sometime today or Monday.

Wyat

#10

Thank you so much, Liz and Wyat! I am able to combine the files using aqProject_pre_and_post.csh and then do the analysis with aqProject_post_only.csh. However, an updated version of the aqProject_pre_and_post.csh script without running mySQL will be super appreciated!

Thanks,
Qian

#11

Hi Qian,

I’ve attached an updated version of the aqExample_pre_and_post.csh script. This version is setup to ignore creating a MySQL database and project if AMET_DB is set to F. You should be able to use this script to run combine on your CMAQ output. From there, you could either continue to use this script or just use the aqExample_post_only.csh script, pointing to the combine files instead of the raw CMAQ output files.

Also, it just dawned on me that you could have changed the flags in your existing aqExample_pre_and_post.csh script to only run combine (i.e set the RUN_COMBINE flag to T and all other flags to F). They should actually work if all your paths are set correctly. They you could use your aqExample_post_only.csh script with the combine files you just created. I think that approach should work also. But ultimately having a single script that does everything is ideal in my option. But it’s up to you.

Let me know how it goes.aqProject_pre_and_post.csh (30.9 KB)

Wyat

#12

Hi Wyat,

Thanks for updating this script and the suggestions on running the combine program. There are some minor issues with this script. For example, RUNID and OBS_DATA_DIR were not defined in this version. I added these environment variables and ran this script. However, when using the updated script you just provided to do the analysis, I got some other errors:

image

My input files do not have this species (AXYL1J) and here is the log file after running the updated scripts (log.pre2.txt (139.7 KB) ).

Thanks,
Qian

#13

Thanks. I did miss those two variables.

For your CMAQ files, did you write out all the ACONC variables or just a subset of variables? This option is specified in the CMAQ run script.

There are two reasons why a species would be missing. One is that you wrote out just a subset of the available CMAQ variables, and one of the requested variables in combine is missing in your raw CMAQ ACONC files. The other reason is you’re using a Species Definition File file (spec_def file) for a different CMAQ chemical mechanism. For example, specifying the chemical mechanism as cb6r3_ae7_aq in your AMET run script when your CMAQ simulation used the cb6r3_ae6_aq chemical mechanism. This could lead to a mismatch in the requested species in combine.

If you’ve simply written out a subset of the ACONC species then you will need to create your own spec_def file and point to that file. Start with an existing spec_def file for the chemical mechanism you used and then remove any species in the spec_def file that are not in the ACONC file. If are writing out the full ACONC species, then check to make sure your chemical mechanism specification is consistent.

Wyat

#14

Hi Wyat,

Thanks for your suggestions. I did use the wrong mechanism in the script. After correcting the mechanism, I did not get the previous error. However, I met another problem:

image
image721Ă—66 3.78 KB

Is this still associated with mySQL? I set AMET_DB to F.

Thanks,
Qian

#15

No, my guess is there is an error somewhere else. Did this screenshot come from the AMET log file? Can you attach the entire AMET log file. There’s probably a clue in there. Also, did you get any site compare scripts or .csv files? I’m guessing no .csv files were generated yet.

wyat

#16

I reran the program and succeeded without the error I mentioned before for some unknown reason. So the script works now. Thank you so much, Wyat!

Qian