Why disabling netcdf4?

I/O API
1

Hi,

I was wondering why is it encouraged to disable netcdf4, and what happens if you compile with netcdf4 enabled?

Thanks,

1 Like
2

See https://www.cmascenter.org/ioapi/documentation/all_versions/html/AVAIL.html#ncf4:

…
NetCDF Version 4.x have lots of additional build options, that will require a complex set of additional libraries in your Makefiles and all other model-building systems.
…
If you build netCDF-4 without the --disable-netcdf4, you will also need to add -DIOAPI_NCF4=1 either to the ARCHFLAGS in your Makeinclude.${BIN} or to the DEFINEFLAGS in your Makefile, since defining netCDF-4 causes netCDF to change parts of the netCDF-3 interface.

3

Thanks for your reply.

Dose compiling CMAQ with netcdf4 enabled cause ram issues? I have installed with and without netcdf4 enabled (since some input data were in nc4). Both pass the benchmark test easily and successfully, yet when it comes to 12US1 domain, it freezes for nc4 and there I have tried to increase the ram, keep generating for 2 hours of sim, and then freezing. The without nc4 seems there is no such issue. Could you please expand on this more?

Thanks,

4

Hi,

Depending on what you are running there maybe an issue. Could you expand on a few things:

  • What dataset are you using, you alluded to a 12US1 domain. Could you please direct me to where you got the dataset from?
  • Could you please expand on what you mean by ‘freezing’? Does the model crash at that time, or does it continue to run but is stuck at a certain line infinitely?
5

Hi,

Thank you for your comment,
I am using the EQUATES (EPA) dataset, the met dataset comes from co-labs.
I figured out what the crashes were:
I was testing two HPCs at the same time, Cheyenne (NCAR) and Discovery (Northeastern University).
The job would get dropped or crashed just after the script dispatched the jobs into the cores/cpus.
Although this was happening on the same spot, each had its own problem, for Cheyenne I needed to use lots of more memory (the nc4 seems to need much more of that), and for Discovery, there was a problem with /scratch file system that the files could not be read fast and kept the job alive but without any further progress.

6

Thanks for providing additional information. Through testing at the EPA, we’ve noticed that certain domains encounter memory leaks reading in large amounts of compressed netCDF data, such as the EQUATES dataset you mentioned. We have had no issues when reading in uncompressed datasets.

This is because of a memory leak in HDF5 routines used in the netCDF library to read in compressed data, eventually leading the system to kill jobs due to using more memory than available to the job. Please see: memory leak reading compressed files - Hdf-forum archives - HDF Forum. The latest version of HDF5 we’ve test is version 1.10.18, so anything after that may contain a fix.

A work around for those cases, is uncompressing that data manually via nccopy or m3cple (compiled with HDF5) before running the simulation. Usually, for us, that takes care of the issue regardless of the library we use to compile our model netCDF4 or netCDF3. I recommend you try this (uncompressing your files manually via nccopy or m3cple) and running CMAQ compiled with netCDF4 to see if that addresses your problem.

If it does fix your issues, please report back so other community members can benefit from your experience.

Thanks!

4 Likes
7

An easier way might be to use an HDF-enabled version of M3Tools program m3cple, the defaults for which amount to file-copy: its I/O API output always produces (64-bit-enabled)_ “netCDF Classic” format…

3 Likes
8

Thanks, I’ve updated my original post to include m3cple as an option.

9

Thanks for your feedback!

That is exactly what I did. I used nccopy to conver my netCDF4 files to netCDF3, and ran with a netCDF3 compiled CMAQ.
Actually from this comparison I first asked for the difference between nc3 and nc4 and also asking that does nc4 lead to ram/memory leakage and issues.

Again thanks for your explanation and solution to bypass it.
Best,

1 Like