Hello
I am running CMAQv5.5 built using gcc on a cray machine. I’m testing the optimal configuration (e.g., domain decomposition for my HPC), for CONUS 12US1 domain using 2022v1 EMP data. In the process I set up CMAQ to run with more than 1000 processors:
@ NPCOL = 46; @ NPROW = 30
This fails almost immediately and I see that no log files are created for processor ID exceeding 999 (last two shown below):
CTM_LOG_999.v55_gcc_IE_12US1_cb6r5_ae7_aq_m3dry_20220101
'CTM_LOG_***.v55_gcc_IE_12US1_cb6r5_ae7_aq_m3dry_20220101'
Is that an expected behavior and CMAQ is set to run with less than 1000 processors total (which I’m guessing based on the log file name structure) or am I doing something wrong here? I’ve another ongoing run which is progressing fine with 900 processors (for testing).
Thank you for your insights.
PS: I recall we had a similar issue with MCIP but with the domain specified such that number of rows (or columns) exceeded 999 resulted in MCIP failing.
PS2: In my testing, a 7x increase in the number of processors resulted in a 50% reduction in run time.
PS3: my simulation run time has essentially flattened beyond 750 processors.