AMET 1.4 raob error

[kgurer]

Hello Robert,

I cannot get “raob” data extracted from MPAS outputs via “matching_raob.csh” script while extracting “surface” works via “matching_surface.csh” script in AMET_v14 version. I downloaded the latest 1.4 version and repeated this process several times and it is still the same result. I used MPAS global model outputs just to confine the model to those files, rather than running into the possibility of reading limited area domain problem that we had faced earlier (although that is resolved now).

The problem is somewhere in MET_model.read.R script during the reading “timestamp”. What I don’t understand is that MET_model.read.R works when MET_matching_surface.R accesses to it, but MET_model.read.R doesn’t work when MET_matching_raob.R accesses to it.

Here is a cut-paste portion of the output obtained from unmodified matching_raob.csh script:

---

Matching MPAS output file with observations: /scratch/kgurer/MPAS/MPAS-Model-7.0.2013.1/history.2013-01-09_00.00.00.nc4
Creating new database if it does not exist: CREATE DATABASE IF NOT EXISTS test ;
Database Action: An AMET ** stations ** table exists in database test . Will use this existing table.
Database Action: An AMET ** project_log ** table exists in database test . Will not re-create this existing table.
Database Action: mpas_2013_48km_global_raob AMET table exists. Will add new model-obs pairs to this existing table.
Reading MPAS Model Output File: /scratch/kgurer/MPAS/MPAS-Model-7.0.2013.1/history.2013-01-09_00.00.00.nc4
timestamp:
Error in strsplit(tmp1, “-”) : non-character argument
Calls: model_time_format -> unlist -> strsplit
Execution halted
Date/Time END
Thu May 7 11:43:48 PDT 2020

That error suggested that it somehow (!) cannot properly assign the timestamp when it reads model files. Therefore, I modified MET_model.read.R to assign timestamp variable (below, 1) to what it should read and in fact it successfully read and assign the value to “timestamp” but this time I come up with some other error (below, 2):

------------------------ 1 — (my modified section in MET_model.read.R ---------------------------

tmp1 <- unlist(strsplit(timestamp,".nc"))
writeLines(paste(“tmp1:”,tmp1))

timestamp <- unlist(strsplit(tmp1[2],"_"))[1]
writeLines(paste(“timestamp:”,timestamp))

yc <- unlist(strsplit(tmp1[2],"-"))[1]
writeLines(paste(“yc:”,yc))

mc <- unlist(strsplit(tmp1[2],"-"))[2]
writeLines(paste(“mc:”,mc))

dc <- unlist(strsplit(tmp1[2],"-"))[3]
dc <- unlist(strsplit(dc,"_"))[1]
writeLines(paste(“dc:”,dc))

tmp2 <- unlist(strsplit(tmp1[2],".00.00"))
hc <- unlist(strsplit(tmp2[2],"\."))[2]
writeLines(paste(“hc:”,hc))

minc <- “00”
writeLines(paste(“minc:”,minc))

secc <- “00”
writeLines(paste(“secc:”,secc))

------------- original section ------------------

tmp1 <- unlist(strsplit(file,"_"))

writeLines(paste(“tmp1:”,tmp1))

yc <- unlist(strsplit(tmp1,"-"))[1]

writeLines(paste(“yc:”,yc))

mc <- unlist(strsplit(tmp1,"-"))[2]

writeLines(paste(“mc:”,mc))

dc <- unlist(strsplit(tmp1,"-"))[3]

writeLines(paste(“dc:”,dc))

hc <- unlist(strsplit(tmp1[2],":"))[1]

writeLines(paste(“hc:”,hc))

minc <- unlist(strsplit(tmp1[2],":"))[2]

writeLines(paste(“minc:”,minc))

secc <- unlist(strsplit(tmp1[2],":"))[3]

writeLines(paste(“secc:”,secc))

------------------------ end of 1 --------------------------------

------------------------ 2 ---------------------------
Reading MPAS Model Output File: /scratch/kgurer/MPAS/MPAS-Model-7.0.2013.1/history.2013-01-09_00.00.00.nc4
— 00 —
— 01 —
timestamp: /scratch/kgurer/MPAS/MPAS-Model-7.0.2013.1/
timestamp: 2013-01-09_00.00.00.nc4
tmp1: /scratch/kgurer/MPAS/MPAS-Model-7.0.2013.1/
tmp1: 2013-01-09_00.00.00
tmp1: 4
timestamp: 2013-01-09
yc: 2013
mc: 01
dc: 09
hc: 00
minc: 00
secc: 00
modeldate: 20130109
modeltime: 00:00:00
Opening MADIS raob for time: 20130109 00:00:00
/home/kgurer/test.AMET-master/obs/MET/point/raob/netcdf/20130109_0000
Reading MPAS Model Output File: /scratch/kgurer/MPAS/MPAS-Model-7.0.2013.1/history.2013-01-09_00.00.00.nc4
— 00 —
Error in Rsx_nc4_get_vara_double: NetCDF: Index exceeds dimension bound
Var: pressure Ndims: 3 Start: 4294967295,0,0 Count: 1,835586,55
Error in ncvar_get_inner(ncid2use, varid2use, nc$var[[li]]$missval, addOffset, :
C function R_nc4_get_vara_double returned error
Calls: mpas_raob -> ncvar_get -> ncvar_get_inner
Execution halted
Date/Time END
Thu May 7 11:55:32 PDT 2020
---------------------- end of 2 -----------------------------

I have to mention that both unmodified and modified MET_model.read.R script works to extract matching_surface data, but not for matching_raob.

Could you please check the functionality of raob extract script in your AMET version and let me know if you are running into a problem?

Thank you Robert. With regards,

Kemal.

[rgilliam]

Kemal,

If it failed in model.read, it is likely something required is not in the MPAS output. These are all the reads in the mpas_raob() function. Let me know if any of these are missing. hpbl and surface pressure may not be needed. The others certainly are required. I did not make exceptions where if for example, wind was missing, but temp and moisture present. It will just not work unless there is wind, temp, moisture, pressure and height is available.

[kgurer]

Hi Robert,

I check the model output. All model variables are there. If you look at the the error that it gives, it specifically has a problem in creating the “timestamp” variable.

However, I was suspecting if this has anything to do with skipping the first time stamp, and sure enough, when I set SKIPIND to “1 1” instead of to “0 0”, then the script works. What is this index for? As far as I understand, first 1 skips the first model timestamp when it reads all timestamps available in the first file and the second 1 is for the other model output files. If I use WRF model output, then I might want to skip certain timestamps at the beginning of a file since I prefer to dump model results to a single output file for WRF results. But, for MPAS, I don’t write all model results into a single output file since the output file size gets very large. Therefore, I want to read all hours that I generate and I don’t want to skip any timestamp and set it to “0 0”. To reiterate the issue that I am having, “matching_surface.csh” that uses MET_model.read.R works fine with SKINPIND set to “0 0”, but not “matching_raob.R” that reads exactly the same MET_model.read.R when SKIPIND set to “0 0”. I checked how MET_matching_surface.R and MET_matching_raob.R uses SKIPIND but couldn’t see any difference and understand how one works but not the other.

In short, the matching_raob.csh script works if I set to SKIPIND “1 1”, but not “0 0”. I would appreciate if you could kindly tell me if the way that I understand and explained above is the right way that that variable works. Thank you Robert.

Kemal.

[rgilliam]

You got it.

These are time indices in R. I guess the variable name makes it a bit confusing, but this is the time index to skip to, not the number of time indices to skip. So 0 would cause R to look in spot 0 of the time dimension of the array, which does not exist since it starts at 1.

And yes, the second number is where the script starts after the first model output file. I do not see any instances where one would use anything other than “2 1” or “1 1”. Where “2 1” is used when your first model output file has the initialization time where many variables have not been computed yet. And “1 1” if there is no initialization period. I guess one could do something like “12 1” if they want to allow the model to spin up for a few hours before any evaluation.

Good deal buddy.
Let me know if you have any other issues.

Rob

[kgurer]

Thank you for the explanation Robert. Have a great day.

Kemal.