Hello,
I have been trying to get the “aqProject_pre_and_post.csh” to run for my own CMAQ run. Everything seems to run fine except no plots are produced. I dont fully understand the “Run_Misc_Plots_Batch.Rout” but it seems to have weird errors in it. I have attached the Rout file.
Run_Misc_Plots_Batch.Rout.txt (30.8 KB)
Hi. You mention only the Misc_Plots_Batch script. Did all the batch scripts fail? Meaning, you go no plots at all (which is what you said, I just want to confirm)?
The errors you’re seeing in the Misc_Plots scripts is my fault. In the header for the Soccergoal and Bugle plots scripts, I added some parenthesis. Those cause the script to fail. The simplest solution is to modify the Soccergoal and Bugle plots scripts in the R_analysis_scripts directory by removing the parenthesis in the header. You’ll see them added around “soccergoal” and “bugle”. Just remove any parenthesis you see and save the script. That will fix the error you’re seeing in the Misc_Plots script.
However, if you’re not getting any plots at all, then there is likely a larger issue at play. If that is the case, can you send any other log files you have? That way I can see what other errors may be occurring. Thanks.
Wyat
Thanks for the reply! I will modify those headers.
I dont get any plots at all. Ive attached the script and log files for the plots.
aqProject_pre_and_post.csh.txt (30.8 KB)
Run_Misc_Plots_Batch.txt (30.8 KB) Run_Scatter_Plots_All_Batch.txt (134.6 KB) Run_Spatial_Plots_All_Batch.txt (138.9 KB) Run_Stacked_Barplot_All_Batch.txt (27.8 KB) Run_Time_Plots_All_Batch.txt (415.1 KB)
It looks like the start and end dates of the analysis time period are not being set. Do you have AMET_SDATE and AMET_EDATE set in your run_AMET_batch_script.csh script? Those are read in as environment variables in the AMET_batch.input file. That’s the first thing to check.
If you are setting those dates, then something else is going on.
So I set those dates in the run_AMET_batch_script.csh which I found in the scripts_analysis folder. I am a little confused since I am running the pre and post script in the scripts_db folder. Do I need to run it from the scripts_analysis folder?
I did run the pre and post script again but still no plots.
Ah, thanks. That actually is an error. I’m using different environment names in the pre_and_post script and the run_AMET_batch_scripts.csh script.
OK, the simple fix is to edit your batch script config file. You can find it here.
$AMETBASE/script_analysis/aqExample/input_files/AMET_batch.input
Open that file and make the change below.
start_date <- gsub(’-’,’’,Sys.getenv(“AMET_SDATE”))
end_date <- gsub(’-’,’’,Sys.getenv(“AMET_EDATE”))
to
start_date <- gsub(’-’,’’,Sys.getenv(“START_DATE_H”))
end_date <- gsub(’-’,’’,Sys.getenv("END_DATE_H))
Then try running your script again. Hopefully that fixes the error. Note that if you make this change, you should change the environment names in the run_AMET_batch_scripts.csh from AMET_SDATE and AMET_EDATE to START_DATE_H and END_DATE_H as well in case you end up running that script at some point.
I’ll tag this as something I need to update in the next AMET release. Let me know if this fixes your problem or not.
Wyat
Ive updated the variables in both scripts but there is still no output.
Ive reattached the Rout files.
Run_Misc_Plots_Batch.txt (30.8 KB) Run_Scatter_Plots_All_Batch.txt (132.5 KB) Run_Spatial_Plots_All_Batch.txt (137.3 KB) Run_Stacked_Barplot_All_Batch.txt (27.0 KB) Run_Time_Plots_All_Batch.txt (413.0 KB)
Based on the error, it looks like no data are available for hourly O3, and likely other species as well. In fact, the database does not even have a column for the O3 observations. Have you tried testing an individual script to see it that works? The batch script is really meant to be used with the full scope of all the AMET networks. So, if you’ve limited your analysis to just one of a few networks, the batch script will not work very well.
I suggest trying one of the simple scatter plot scripts for a single species and network. If we can confirm that works, we can start to diagnose what’s going on with the batch script. My initial thought is that the data loading did not work correctly. If you have the log file from aqProject_pre_and_post_script.csh, we should examine that to make sure everything worked properly.
I am only running dust emissions in CMAQ so there wouldnt by any O3 data. I am mostly interested in PM. I ran the run_scatterplot.csh in scripts_analysis folder and get the following error and log file.
The AMET R script did not produce any output, please check the LOGFILE scatterplot.log for more details on the error.
Often, this indicates no data matched the specified criteria (e.g., wrong dates for project). Please check and re-run!
scatterplot.txt (2.2 KB)
Here is the run script as well
run_scatterplot.csh (5.3 KB)
You can leave the start and end date in the run script as AMET_SDATE and AMET_EDATE. Those environment names only needed to be changed for the batch script and batch input file. Hopefully the script will work then.
Once we’ve established that works, I can show you how to change the batch scripts to focus on PM.
Wyat
So the scatterplot script still doesnt work when I changed the two variables back.
run_scatterplot.csh (5.3 KB) scatterplot.txt (2.1 KB)
Alright, it looks like you’ve specified the species to plot as PM_25. Unless you’ve changed the variable names in AMET yourself, PM_25 is not a default variable in AMET. By default, I call PM2.5 “PM_TOT.” Try setting the species to PM_TOT (unless you have changed the default name of PM2.5) and see if the plot works. If not, we’ll need to check to see if you’re data have loaded properly into the database.
Wyat
Also, I realize that it’s a bit confusing to have variables called PM_TOT and PM25_TOT, and it’s a consequence of early choices in the development of AMET. PM_TOT is the combination of the i and j modes from CMAQ, and is intended to represent “mostly” PM mass that is 2.5 microns or smaller. PM25_TOT uses the actually particle diameter information in CMAQ to determine the mass that’s smaller than 2.5 microns. So, PM25_TOT is theoretically a more accurate representation of PM2.5 mass. However, it assumes that CMAQ is correctly estimating the particle size distribution, which studies have shown is not always the case.
So, assuming you have both PM_TOT and PM25_TOT, you can use either to compare to PM2.5 observations. There will be small differences however. We have plans to do a study using the latest version of CMAQ to quantify the differences between PM_TOT and PM25_TOT. However, it is not available yet.
Wyat
Thank you for the explanation! I started using all these programs a year ago so keeping all these variables straight is a bit difficult haha.
I had the same result as before.
scatterplot.txt (2.1 KB)
Alright, it would seem that there was a problem either setting up or loading the data into the database. Do you have the log file from the aqProject_pre_and_post.csh script available? That will tell us if there was an error loading the data.
Also, did you get output from site compare? Meaning, do you have .csv files with paired model and observation values?
Wyat
Here are the log files. I had to split them up.
log_partaa.txt (2.5 MB) log_partab.txt (2.5 MB) log_partacc.txt (2.6 MB) log_partad.txt (2.1 MB)
I dont think I get any of those. I dont see anything like that in the output folder
It looks like the model/ob matching never took place. Perhaps you didn’t have that flag set to T, because it looks like it never even tried to do the matching. Can you attach your aqProject_pre_and_post.csh script? Thanks.
Yes here is the script.
aqProject_pre_and_post.csh.txt (30.8 KB)
Alright, everything seems to be moving along correctly in that script.
You should have a directory called /home/pkasper/AMET_v14/output/v531_gcc_2010_30_PXMLandUse/sitex_output/YYYYMM. In there should be another log file called amet_extract_all_v531_gcc_2010_30_PXMLandUse.log. It’s likely that the error is in that log file.
Let me know if you can find that log file, and if so, send it along. Hopefully that will tell us what’s going on. Thanks.