How do we go about converting a gridded netCDF file (in lat-lon coordinates with several variables in it) to the IO/API format?
This is a duplicate posting. Please do not duplicate the ticket unless it is necessary.
Be aware that this is not necessarily trivial, since netCDF does not mandate all the needed metadata, and when available it is frequently not of adequate precision.
Be aware also that netCDF is much more tedious to write robustly, since netCDF is a much-lower-level data interface. Basically, you’ll need to write your own netCDF reader-code, and glue it to I/O API writer-code, probably in a custom program.
For examples of codes where this kind of thing is done, see I/O API M3TOOLS programs “camxtom3” and “wrftom3”,
which are documented at:
https://cjcoats.github.io/ioapi/CAMXTOM3.html
https://cjcoats.github.io/ioapi/WRFTOM3.html
Just wanted to draw your attention to my I/O API v3.2 installation problem:
I recently used python package PseudoNetCDF to copy over some gridded 3D data from a netCDF file into an I/O API format file, but I cannot plot the new I/O API file in VERDI, so I am assuming something went missing/wrong. I am showing here the result of ncdump -v TFLAG on the new I/O API file:
dimensions:
TSTEP = UNLIMITED ; // (1460 currently)
DATE-TIME = 2 ;
LAY = 34 ;
VAR = 1 ;
ROW = 45 ;
COL = 72 ;
variables:
int TFLAG(TSTEP, VAR, DATE-TIME) ;
TFLAG:units = "<YYYYJJJ,HHMMSS>" ;
TFLAG:long_name = "TFLAG " ;
TFLAG:var_desc = "TFLAG " ;
float SO2(TSTEP, LAY, ROW, COL) ;
SO2:long_name = "so2" ;
SO2:var_desc = "SO2 " ;
SO2:units = "vmr" ;
SO2:fill_value = -1.e+10f ;
SO2:missing_value = -1.e+10f ;
SO2:cell_methods = "time: mean" ;
SO2:time_avg_info = "average_T1,average_T2,average_DT" ;
SO2:interp_method = "conserve_order2" ;
// global attributes:
:IOAPI_VERSION = "ioapi-3.2: $Id: init3.F90 120 2019-06-21 14:18:20Z coats $ " ;
:EXEC_ID = "PYTH36 24MAY2020" ;
:FTYPE = 1 ;
:CDATE = 2020148 ;
:CTIME = 112727 ;
:WDATE = 2020148 ;
:WTIME = 112727 ;
:NTHIK = 1 ;
:GDTYP = 1 ;
:P_ALP = 0. ;
:P_BET = 0. ;
:P_GAM = 0. ;
:XCENT = 0. ;
:YCENT = 0. ;
:VGTYP = 7 ;
:VGTOP = 5000. ;
:GDNAM = "my_GFDL_latlon_grid" ;
:UPNAM = "MAKEIOAPI " ;
:FILEDESC = "CREATION ON PSEUDONETCDF" ;
:HISTORY = "" ;
:CDI = "Climate Data Interface version 1.9.3 (http://mpimet.mpg.de/cdi)" ;
:Conventions = "CF-1.6" ;
:history = "Sun May 24 11:45:09 2020: cdo sellonlatbox,-180,0,0,90 step4final_so2_ml2pl_34layers_drenamed_24may24may2020.nc2 step4final_so2_ml2pl_34laye
rs_drenamed_sellonlatbox_24may24may2020.nc2" ;
:filename = "atmos_level.2014010100-2014123123.SO2.nc" ;
:title = "CM3Z_H1_2006-2100_2014_2015" ;
:grid_type = "regular" ;
:CDO = "Climate Data Operators version 1.9.3 (http://mpimet.mpg.de/cdo)" ;
:VAR-LIST = "SO2 " ;
:NVARS = 1 ;
:NCOLS = 72 ;
:NROWS = 45 ;
:NLAYS = 34 ;
:TSTEP = 60000 ;
:SDATE = 2014001 ;
:STIME = 30000 ;
:VGLVLS = 1., 0.995, 0.989, 0.983, 0.976, 0.968, 0.959, 0.95, 0.94, 0.929, 0.916, 0.902, 0.887, 0.871, 0.853, 0.833, 0.811, 0.787, 0.76, 0.731, 0.699, 0
.663, 0.624, 0.581, 0.534, 0.482, 0.429, 0.375, 0.322, 0.268, 0.214, 0.161, 0.107, 0.054, 0. ;
:XORIG = -180. ;
:YORIG = 0. ;
:XCELL = 2.5 ;
:YCELL = 2. ;
I do see some extra header info like CDI, filename that are usually not in I/O API files, so I was wondering if they ‘confused’ VERDI when trying to open the file?
From the I/O API Home Page, https://cjcoats.github.io/ioapi/index.html:
The Models-3 I/O API is A programming interface, not a data format !!
I/O API files are not synonymous with “netCDF files” !!
There are data structures that the I/O API embeds into the files and that it then uses that are needed. So far, no one who has attempted to treat it as “just another data format” has succeded in understanding all these data structures and getting it completely right.
MS-Word .docx is likewise “just a data format” but no one writes .docx with a binary-editor! The same is true here.
Carlie J. Coats, Jr., Ph.D. cjcoats@email.unc.edu
Senior Software Engineer I/O API Author/Maintainer
Center for Environmental Modeling for Policy Development,
UNC Institute for the Environment www.ie.unc.edu
What does M3Tools program “m3stat” say about this file?
BTW, VERDI is another of those programs that attempts to treat I/O API
as “just a data format” and doesn’t get it all correct either. So this
may be a VERDI problem – and +is a VERDI problem if “m3stat” worked
correctly.
– Carlie
The REPORTFILE of m3stats on the file shows this:
File: INFILE
Date and time: 2014001:030000 (3:00:00 Jan. 1, 2014 )
Variable: SO2
3-D grid statistics
Max 1.06599E-02 @(c,r,l)=(37,19,1)
Min -1.00000E+10 @(c,r,l)=(41,1,1)
Mean -5.92956E+08
Sigma 2.36177E+09
File: INFILE
Date and time: 2014001:090000 (9:00:00 Jan. 1, 2014 )
Variable: SO2
3-D grid statistics
Max 1.28460E-02 @(c,r,l)=(38,17,1)
Min -1.00000E+10 @(c,r,l)=(41,1,1)
Mean -5.95951E+08
Sigma 2.36735E+09
File: INFILE
Date and time: 2014001:150000 (15:00:00 Jan. 1, 2014 )
Variable: SO2
3-D grid statistics
Max 1.29322E-02 @(c,r,l)=(38,19,2)
Min -1.00000E+10 @(c,r,l)=(41,1,1)
Mean -6.03758E+08
Sigma 2.38182E+09
File: INFILE
Date and time: 2014001:210000 (21:00:00 Jan. 1, 2014 )
Variable: SO2
3-D grid statistics
Max 9.96882E-03 @(c,r,l)=(37,19,1)
Min -1.00000E+10 @(c,r,l)=(41,1,1)
Mean -6.09749E+08
Sigma 2.39284E+09
The Min values are all ‘missing values’ inherited from the original data - minimum should have been 0 not negative but m3stat probably has to be told somehow to not consider ‘missing values’ in its calculations.
[snip…]
So this says authoritatatively that it’s not an I/O API problem; it’s a
VERDI problem. ;-( …and the VERDI authors treat it as a data-format
problem without consulting the I/O API author, even though he’s only
half-a-dozen doors down the hall…
– Carlie