# CMAQ memory problem

**URL:** <https://forum.cmascenter.org/t/cmaq-memory-problem/4486>\
**Category:** CMAQ\
**Created:** [October 16, 2023, 6:37pm UTC](https://forum.cmascenter.org/t/cmaq-memory-problem/4486 "2023-10-16T18:37:47Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![haohe.umcp](https://avatars.discourse-cdn.com/v4/letter/h/b19c9b/32.png) [@haohe.umcp](https://forum.cmascenter.org/u/haohe.umcp)\
**Post date:** [October 16, 2023, 6:37pm UTC](https://forum.cmascenter.org/t/cmaq-memory-problem/4486/1 "2023-10-16T18:37:47Z")

</div>

I am running CMAQ v5.3.3 with the EQUATES 2013 case. The model run around 6-7 hours and then was killed. Attached please find the slurm log and computing node log files.

I contacted our supercomputer administrator for help and got the following information about the job.

`login-1.zaratan.umd.edu{haohe}1403: sacct -j 3847222 -o NodeList%20,Start,Elapsed,State%16,ExitCode,AveRSS%16,MaxRSS,AllocTRES%40`  
NodeList Start Elapsed State ExitCode AveRSS MaxRSS AllocTRES

* * *

compute-a8-[25-26,3+ 2023-10-14T21:55:39 00:38:04 COMPLETED 0:0 billing=108,cpu=108,mem=432000M,node=4  
compute-a8-25 2023-10-14T21:55:39 00:38:04 COMPLETED 0:0 31683384K 31683384K cpu=9,mem=36000M,node=1  
compute-a8-[25-26,3+ 2023-10-14T21:55:39 00:38:04 COMPLETED 0:0 1798K 1816K billing=108,cpu=108,mem=432000M,node=4  
compute-a8-[26,31-3+ 2023-10-14T21:55:40 00:38:03 OUT\_OF\_MEMORY 0:125 129761143466 192096384K cpu=99,mem=396000M,node=3

My run with 108 CPUs allocated 432 Gb memory, sufficient for the 12US1 domain. He suggested that there could be a memory leak somewhere which consumed more and more memory until it crashed. I have used v533 for a lot of modeling before and did not meet this problem, so I believe the code should be fine.

The only new thing is that for this EQUATES study, I used the input files from the EPA shared Google Drive. Is it possible the input files cause this problem? If yes, how can I debug the problem? Suggestions and comments are highly appreciated.

Hao

[CTM\_LOG\_083.v533\_cb6r3\_ae7\_aq\_WR413\_MYR\_STAGE\_EQUATES\_20130102.txt](https://forum.cmascenter.org/uploads/short-url/rPGDm6sFcctuD0PWv5deCZ2s0uv.txt) (300.2 KB)  
[slurm-3847222.out.txt](https://forum.cmascenter.org/uploads/short-url/tuCjb2srfgQsHMLbKngIyLSLqQg.txt) (112.4 KB)

---

<div class="post-metadata">

**Author:** ![lizadams](https://avatars.discourse-cdn.com/v4/letter/l/49beb7/32.png) [@lizadams](https://forum.cmascenter.org/u/lizadams)\
**Post date:** [October 16, 2023, 8:17pm UTC](https://forum.cmascenter.org/t/cmaq-memory-problem/4486/2 "2023-10-16T20:17:29Z")

</div>

This is likely the [netcdf4 memory leak](https://forum.cmascenter.org/t/why-disabling-netcdf4/3866) issue. The solution is to convert EQUATES netcdf4 files to netcdf3-classic files using [nccopy](https://www.unidata.ucar.edu/software/netcdf/workshops/2011/utilities/Nccopy.html).

Please see:  
[Example instruction to convert EQUATES input data from nc4 to nc3](https://forum.cmascenter.org/t/equates-memory-leak-issue/4250)  
This ncopy conversion requires using the netCDF version built to support netCDF-4 compression.

Then [use the I/O API and netCDF libraries that were built to disable netCDF-4 compression](./configure --prefix=$cwd/…/netcdf --disable-netcdf-4 --disable-dap) to run CMAQ using the netcdf3-classic input files.

To manage multiple versions of netCDF builds, please consider building and using custom [Environment Modules](https://researchcomputing.princeton.edu/support/knowledge-base/custom-modules).

---

<div class="post-metadata">

**Author:** ![haohe.umcp](https://avatars.discourse-cdn.com/v4/letter/h/b19c9b/32.png) [@haohe.umcp](https://forum.cmascenter.org/u/haohe.umcp)\
**Post date:** [October 18, 2023, 3:22am UTC](https://forum.cmascenter.org/t/cmaq-memory-problem/4486/3 "2023-10-18T03:22:38Z")

</div>

This methods works, thanks!