I downloaded the Observation data for 2021. AQS_hourly_data_2021.csv contains almost does not have data for CO, SO2, NO2, NOX, NOY and for all other variables.
Hi,
I assume your comment is saying that the file contains ALMOST no data for O3, CO, SO2, NO2, NOX, and NOY and no data for the other variables. Is that correct?
How did you come to that conclusion? I looked at the file and there appears to be data for every record. Note there are over 24 million records in that file, so it’s not possible to check every record.
One thing to keep in mind is that there are a lot of missing values in the file, so it can appear that there are very little data or a large amount of sites without complete data. However, all species are not reported for every AQS site. In fact, most sites typically report several species. So, a record for a single site will have a lot of missing values but still contain data for one or more species. There should be no records without at least some non-missing data. If you have identified records in the file that contain no valid data, please let me know. That should not happen.
If you simply open the file and look at it, it may appear to be mostly missing data, but that is not the case. But, I do want to know how you determined that there were missing data in the file. Thanks.
Wyat
Upon further investigation of the AQS hourly data file for 2021 I noticed the version of the file in the tar/zip file on the CMAS website contains an erroneous line at the end of the file. If you’re using the AQS hourly file with AMET, that erroneous line will cause site compare to crash and no hourly data will be processed.
Please remove the erroneous line (see below) from the file before using it with AMET. I’m not sure if this is the root cause of the issue you’re having, but the line should be removed regardless. I also intend to update the tar/zip files for 2017-2023 soon. EPA recently updated some of the historical PM2.5 values in AQS and I need to update the tar/zip data files with those updates.
Erroneous line at the end of the AQS hourly data file:
State CodeCounty CodeSite Num,POC,Date Local Ti:00:00,Date Local Ti:59:00,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999,-999
Wyat
1 Like
Thanks Wyat that was the root cause of the issue.
Manish