While viewing the MET example on the official AMET github website, I found a strange discrepancy in the range of wind directions from the output of the “run_summary.csh” script.
In particular, according to the “Example csv” file of diurnal and overall statistics (https://github.com/USEPA/AMET/blob/master/docs/images/stats.metExample_wrf.JULY2016.csv), the range of wind directions is [2.887e-11, 9.999e-4] for the METAR observational data and [-179.999, 179.994] for the WRF model data. This in turn results in a very high RMSE value of 49.7.
Does anyone know why this is the case? Is there a typical / acceptable range for the RMSE (or the other model performance summary statistics) of wind direction? In a separate analysis using different WRF data, I get a wind direction RMSE value of about 90.
You bring up a good issue. And it needs to be made much clearer in our next iteration. I will put it on the list of AMET updates.
RMSE of wind direction on the order of 50 deg is very common. In fact, that is an expect value for most modeling cases. I would advise to use Mean Absolute Error (MAE) with wind direction. At least that is my preference. The reason… it is a unique variable unlike a scalar such as temperature. An example: when the wind is light (couple m/2), it is almost always highly variable. And the wind is light probably more often than strong at least over land. So if the wind is 1-2 m/s it is very difficult for a model to capture the variable nature of the wind direction, so you may have differences with the model of 50, 60, 90, 180 deg even. The inherent math of the RMSE where errors get squared actually inflates the RMSE because there will be frequent large differences between model and obs. MAE at least removes the squared difference part of the error calculation.
We see MAE in our best modeling for a US domain on the order of 30 deg. High 30 deg range is larger than typical. When it drops into the 20 deg range that is very good from our experience.
In short, ignore the min and max obs. Again, the nature of wind direction makes these values irrelevant. But, to get in the weeds, the reason these min and max values are different is how I calculate in AMET. In order to use the standard RMSE, MAE, bias functions in AMET that the other variables use, I calculate the angle difference between the model and observations and set that equal to the modeled wind direction and set the observed wind direction to 0. When this is done I can again use the same statistics functions as other variables.
With that said, these issues should definitely be made more well known. And these statistics like min and max should be set to NA or missing because they are irrelevant. The correlation is also irrelevant in this regard.
