Is 68GB memory enough to run CMAQv5.3.1 with 12US1 platform?

CMAQ
1

I am trying to run CMAQv5.3.1 using the 2016 12US1 emissions platform (continental US 459X299 grid cells). I am using 8 processors with 68 GB of memory.The run is aborting during “TIME INTEGRATION” process. I am not getting any specific error,however, I noticed the run stops when the used memory reaches 68GB. Is it possible that I don’t have enough memory for this specific set up? I have attached my log file and run script.run_cctm_2016_12US1_v53.june_july_csh.txt (33.7 KB) run_cctm_2016_12US1_v53.june_july_log.txt (52.5 KB)

2

I am not certain, but I believe 68 GB should be enough memory.

Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 7 with PID 0 on node ip-10-21-154-88 exited on 
signal 9 (Killed).

Look at the “ancillary” log file from processor 7, which should have a name beginning with CTM_LOG_007. Is there an error message?

Please see this post (and the Debug Tutorial) for ideas on how to debug this issue.

3

I should have done this before posting, but I just tested by running on 8 CPUs on our machine. The total memory used (as shown by top) is about 89g. So, that suggests that 68 GB is insufficient without taking steps to reduce the memory footprint.

4

@cgnolte What are some suggestions to reducing memory usage? would using more processors help? I have access to 36 processors but my memory is only 70GB.

5

I am not an expert in this area, but my guess is that if your 8 processors have access to the same total memory as your 36 processors, then using more processors would not resolve the issue. Try 16 or 24 processors and see?

You can reduce your memory requirements by reducing the size of the problem. The simplest way to do so is by running on a smaller modeling domain. It would also be possible to use fewer chemical species, though developing your own mechanism is not recommended.

Perhaps others can offer some further suggestions.

6

For now, the best way to move forward is to reduce the grid domain. My area of focus is the southwest part of the US. Do you know any tool I can use to extract the southwest grid cells from the Continental US gird domain? I know there is one for CAMx binary files but it does not work with netcdf files.

7

You can use the m3wndw program for this purpose.
https://www.cmascenter.org/ioapi/documentation/all_versions/html/M3WNDW.html

8

…and bcwndw for the boundary-files.

9

@cjcoats I am getting the following error when I run bcwndw:

*** ERROR ABORT in subroutine BCWNDW
Input file “infile” has type 2 (type GRDDED3==1 required)

What 's wrong with the boundary condition input file ? I am using files I downloaded from EPA’s google drive https://drive.google.com/drive/folders/1oLgDp-jVzVv4Ec3ewzCU29Jv036fGZMy

10

You need to provide BCWNDW with a 3D gridded concentration file (such as CCTM_CONC) to create boundary conditions for your target domain. You cannot create boundary conditions for a new domain from a boundary conditions file for a different domain since it doesn’t contain the required information.

11

@cgnolte @cjcoats @hogrefe.christian Thanks for your help on this issue. My new grid is 154 col by 110 row and only requires 15 GB of RAM