I’ve downloaded the mesonet data for July of 2017, and the file 20170706_0200 is missing. The output log says:
MADIS FILE NOT Found: /uufs/chpc.utah.edu/common/home/daq-group2/AMET_v14/obs/MET/LDAD/mesonet/netCDF/20170706_0200 for time: 20170706 02:00:00
Skipping to next time
But the script just stops running. I ended up copying the previous file and renaming it to the 0200 hour which works, but I’m wondering if there is a better way to handle missing data.
Thanks for your time!
I can find this file on ftp madis-data.ncep.noaa.gov. If missing data, can request help from MADIS. I did that before, they replied soon and upload data soon, hope this help.
Thank you! I’m actually working with MADIS to update the data, but I’m surprised matching_surface.csh can’t skip a missing file without breaking.
Interesting error. I cannot explain without testing this exact date with the mesonet option on our end.
The scripting was done to skip over dates where MADIS files are missing. But apparently that is not air tight for the code you have installed. I did have an issue some time back. Maybe 2 years ago where empty MADIS files were being downloaded and that passed through the file.exist() check function. Not sure that applies here because you hit the final check of the file where it skips if not present.
I did make an update though to check on whether or not the file is empty. Just not sure what version of the code you have. Which of the below fits what you have installed?
The old version would look something like this (R_db_code/MET_observation.read.R):
New version looks like this:
You can see a check on file size is done and if it is empty, the downloaded file is deleted and that triggers the script to push NA back to the main script where it skips that time.
At any rate, I did look at MADIS and this is the listing for Jul 6, 2017 mesonet. You can see the files for 0200 and 0300 UTC are odd size compared to the rest. This has me thinking that the files may have some issues. I’ll have to just test on our system.
With all of this said, there is a way to skip this time. With this option in your matching_surface.csh script. So for your case you’d run until the error at 0200 UTC. You can then set the first element in this SKIPIND array below to UTC hour+ 1. SO in the cast of your 0200 UTC fail, you can change 1 to 3 and the script will start off on time index 3 of the WRF output. If you leave the second element of the array as 1, all WRF outputs after the first one will reset back to time index 1.
setenv SKIPIND “1 1”
to
setenv SKIPIND “2 1”
I could not replicate this error with the latest version of AMET.
Suggest that if you do not want to grab the most up-to-date version that you use the SKIPIND setting to jump past this date/time that is causing a problem.
But suggest that you do grab the latest AMET in the develop branch on Git-Hub. This will not change existing results. It mostly fixes various issues and adds some functionality like MCIP compatibility, forecast considerations, SurfRAD obs option, etc. Should be backward compatible.
Regards,
Rob
