Multiple offline 3-D emission streams support in CMAQ v5.2

Hello, CMAS folks. Does anyone know how to use multiple offline 3D emission files in CMAQ v5.2? I know the latest 5.3 versions support reading in multiple offline 3D emissions but my simulations will use CMAQ v5.2 due to species mapping issues. I intend to use CB05e51 AE6, which is not supported by CMAQ v5.3. My emissions have a very fine resolution (999 columns * 1069 rows * 20 layers * 44 variables) and I have to split a single-day emission file into two (43 GB each file). Thank you.

Ken

Hi Ken,

Unfortunately, it’s not possible in v5.2 to read multiple 3D emissions files. Your easiest solution would be to copy the folder for CB05e51 AE6 into the CMAQv5.3 repo as an additional mechanism and then rebuild the RXNS_DATA and RXNS_FUNC modules with the chemmech tool. CMAQ/UTIL/chemmech at 5.3.3+ · USEPA/CMAQ · GitHub

You may also want to use the create_ebi tool if you’re going to invoke the ebi solver. CMAQ/UTIL/create_ebi at 5.3.3+ · USEPA/CMAQ · GitHub

Finally, the inline photolysis preprocessor is also potentially necessary if you are going to use inline photolysis. CMAQ/UTIL/inline_phot_preproc at 5.3.3+ · USEPA/CMAQ · GitHub

Note that CMAQv5.4 (to be released in Fall 2022) will have automated scripts for preprocessing mechanisms when mech.def and chemical species namelist files are provided.

Best wishes,
Ben

Hi Ken,

is there a system or data format limit that prevents you from using mrggrid or another comparable tool to create a single 86GB 3D emissions file? As long as IOAPI_OFFSET_64 is set to T for any program (including CMAQ) reading or writing such a file, the netcdf classic data format restriction of 2GB per time step should not be a problem.

Christian

Hi Christian,

Thanks for your information. I understand CMAQ supports reading files larger than 2Gb. My issue was that I tried to use the M3FAKE tool from IOAPI3.2 to get the correct format readable by CMAQ. But this process crashed due to “out of bounds” error. So I had to split my emission data into two files, then I could assign the real emission values into this fake file. I made a few tries and found the maximum variable number allowed per single file was 25 vars * 24 hr * 1069 rows * 999 columns * 20 layers. I suspect it’s limited by IOAPI but I am not sure what compilation flags to change in IOAPI. So I look for solutions in reading multiple emission files by CMAQ.

Ken

Hi Ben,

Really appreciate your information. I will read the guides you point to and hope it will work in CMAQ5.3. Looking forward to the next CMAQ version!

Best regards,
Ken

This is most probably a Linux2_x86_64 memory-model issue: see https://cjcoats.github.io/ioapi/AVAIL.html#medium for more about this:

Basically, the default for ifort and gfortran can’t handle things over 2 GB; you need to compile with an extra -mcmodel=medium compile-flag for large problems – that’s what I/O API compile-types BIN = Linux2_x86_64gfort_medium (or Linux2_x86_64ifort_medium, Linux2_x86_64pg_medium) are for.

Out of curiosity, when you used M3FAKE to try to create the large file and encountered the ‘out of bounds’ error, did you use ‘setenv IOAPI_OFFSET_64 T’ in the shell or script where you were running that program?

As @cjcoats wrote, your problem might require I/O API recompilation with the medium memory model compiler flags, but if the error was caused solely by violating the constraints of the underlying netCDF classic data model (rather than also exceeding the array limits of the small memory model), setting that environment variable to change the underlying netCDF data model to ‘netCDF 64-bit format’ might help.

The “out of bounds” error suggests that this is rather a memory-model problem. Note 4 bytes * 25 vars * 1069 rows * 999 columns * 20 layers is 2135862000 bytes is almost 2GB = 2147483648 bytes.

BTW, IOAPI_OFFSET_64 has defaulted to TRUE since the Dec. 25, 2017 I/O API version.

FWIW

Yes, the 4 bytes * 25 vars * 1069 rows * 999 columns * 20 layers certainly approaches the 2GB-per-timestep limit imposed by the netCDF classic data model, while 4 bytes * 44 vars * 1069 rows * 999 columns * 20 layers exceeded it by almost a factor of 2. So using the netCDF 64-bit data model via the IOAPI_OFFSET_64 (thanks for clarifying that it now defaults to T) is certainly necessary but (probably) not sufficient for this to work.

What I wasn’t sure is whether M3FAKE creates an array that’s dimensioned to accommodate all the desired 44 variables at once in which case the memory-model problem would also come into play, or whether it loops over variables in which case a real array just holding 1069 rows * 999 columns * 20 layers should still work even with the small memory model.

Hi Christian and Carlie,

Thanks a lot for the information. Like Carlie mentioned, my IOAPI_OFFSET_64 is set to TRUE in IOAPI v3.2. I just successfully recompiled all M3TOOLS with the -mcmodel=medium flag and the new fake emission file can hold all my 44 variables. Thanks!!

Best regards,
Ken