IOAPI to netcdf4 compress mode


#1

Hi there,

I have a netcdf library compiled with hdf5 and zlib, and subsequently an ioapi 3.2 has been built on it.

Does this ioapi enable writing to netcdf4 in compressed mode? If not, will there be any plan in the near future?

Thanks!


#2

No.
One of the requirements for the I/O API is direct access, and compression prevents this, since it requires sequential decompression of the data, or else potentially arbitrarily-long index structures.
Consider the (actual, I’ve-done-this) situation of having a decade-long 3000x2000 gridded hourly surface meteorology file, for dates 19910101 - 20000101, and you want to use it to drive a land-surface simulation from 19990701 through 19910801. Or suppose you want to look at that simulation-start date&time 1990701:000000. It is unacceptable to force you to wait for hours while the program is decompressing/reading nine-years-plus of data before you can actually start simulating or visualizing anything.


#3

By the way, one of the reasons netCDF was chosen instead of HDF as the lower layer for the I/O API was the experience that HDF had very long access times for situations like the above, because of the time required to access those (randomly-stored) index structures that it maintains before it ever started reading the data.


#4

Thank you Carlie for providing the insight I never perceived before.

Roger