# Installing CMAQ on HPC

**URL:** <https://forum.cmascenter.org/t/installing-cmaq-on-hpc/3101>\
**Category:** CMAQ\
**Created:** [March 15, 2022, 12:10pm UTC](https://forum.cmascenter.org/t/installing-cmaq-on-hpc/3101 "2022-03-15T12:10:11Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![farzaneh](https://avatars.discourse-cdn.com/v4/letter/f/67e7ee/32.png) [@farzaneh](https://forum.cmascenter.org/u/farzaneh)\
**Post date:** [March 15, 2022, 12:10pm UTC](https://forum.cmascenter.org/t/installing-cmaq-on-hpc/3101/1 "2022-03-15T12:10:11Z")

</div>

Hi,  
I am trying to install CMAQ on HPC cluster. I shift to tcsh shell script with srun --pty /bin/tcsh and then install Libraries and CMAQ.  
I am so new with HPC and CMAQ .  
as I try to run "./run\_cctm\_Bench\_2016.csh in cctm script, It keeps giving me an error "srun: step creation still disabled, retrying(requested nodes are busy)  
I am not sure what is this error about and is it caused by CMAQ or my HPC. Can you please help me

---

<div class="post-metadata">

**Author:** ![Ryan](https://avatars.discourse-cdn.com/v4/letter/r/a8b319/32.png) [@Ryan](https://forum.cmascenter.org/u/Ryan)\
**Post date:** [March 15, 2022, 11:43pm UTC](https://forum.cmascenter.org/t/installing-cmaq-on-hpc/3101/2 "2022-03-15T23:43:10Z")

</div>

If you are using HPC, you will need to get to know the job submission commands for writing a submit.sub script for requesting the nodes. For details, you can jump to your institution’s HPC website for more information because various HPC has different set ups (scratch, srun, bsub, qsub, etc).

---

<div class="post-metadata">

**Author:** ![Gopikrishnan](https://avatars.discourse-cdn.com/v4/letter/g/41988e/32.png) [@Gopikrishnan](https://forum.cmascenter.org/u/Gopikrishnan)\
**Post date:** [March 12, 2025, 3:01am UTC](https://forum.cmascenter.org/t/installing-cmaq-on-hpc/3101/3 "2025-03-12T03:01:10Z")

</div>

HI @Ryan

I am also getting a similar error.

Attached is the sbatch script:

#!/bin/bash  
#SBATCH -A fiore # Account  
#SBATCH --job-name=test\_run # The job name  
#SBATCH -c 32 # Number of cores  
#SBATCH -N 4 # Ensure that all cores are on one machine  
#SBATCH -t 0-12:00 # Runtime in D-HH:MM  
#SBATCH --exclusive # Memory pool for all cores  
#SBATCH -C mem768  
#SBATCH --mem=700GB

source /burg/ocp/users/gg2995/CMAQ/cmaq\_spack.env  
source /burg/ocp/users/gg2995/CMAQ/.profile

export OMP\_NUM\_THREADS=32  
export F\_UFMTENDIAN=big  
export OMP\_STACKSIZE=500m

srun -c $OMP\_NUM\_THREADS time -p /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/CCTM/scripts/run\_cctm\_Bench\_2018\_12NE3.csh \> output.log 2\>&1

Could you help me out in this case.  
I used the following domain decomposition:

#\> Horizontal domain decomposition  
if ( $PROC == serial ) then  
setenv NPCOL\_NPROW “1 1”; set NPROCS = 1 # single processor setting  
else  
@ NPCOL = 8; @ NPROW = 4  
@ NPROCS = $NPCOL \* $NPROW  
setenv NPCOL\_NPROW “$NPCOL $NPROW”;  
endif

---

<div class="post-metadata">

**Author:** ![Gopikrishnan](https://avatars.discourse-cdn.com/v4/letter/g/41988e/32.png) [@Gopikrishnan](https://forum.cmascenter.org/u/Gopikrishnan)\
**Post date:** [March 14, 2025, 3:05pm UTC](https://forum.cmascenter.org/t/installing-cmaq-on-hpc/3101/4 "2025-03-14T15:05:59Z")

</div>

Hi,  
@jvukovich @hogrefe.christian Can you guys help me in this case.

Thanks in advance.!

A detailed description is here.!

> <https://github.com/USEPA/CMAQ/issues/232>
>
> Hi,
> 
> I am using CMAQ v54. I tried running model in the login node of my cluser u…sing NPCOL 4 and NPROW 1.
> 
> Model iterated well. But when i use the slurm script, i get the following error:
> 
> \`\`\`
> 
> Start Model Run At Tue Mar 11 21:12:08 EDT 2025
> Compiler is set to gcc
> 
> Working Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/CCTM/scripts
> Build Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/CCTM/scripts/BLD\_CCTM\_v54\_gcc
> Output Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/data/output\_CCTM\_v54\_gcc\_Bench\_2018\_12NE3
> Log Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/data/output\_CCTM\_v54\_gcc\_Bench\_2018\_12NE3/LOGS
> Executable Name is CCTM\_v54.exe
> 
> \---CMAQ EXECUTION ID: CMAQ\_CCTMv54\_sha=79a966615c\_gg2995\_20250312\_011208\_675630754 ---
> 
> Start Model Run At Tue Mar 11 21:12:08 EDT 2025
> Start Model Run At Tue Mar 11 21:12:08 EDT 2025
> Start Model Run At Tue Mar 11 21:12:08 EDT 2025
> Set up input and output files for Day 2018-07-01.
> Compiler is set to gcc
> Compiler is set to gcc
> ln: failed to create symbolic link '/burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/lib/x86\_64/gcc/mpi/lib': File exists
> mkdir: cannot create directory ‘/burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/lib/x86\_64/gcc/mpi’: File exists
> ln: failed to create symbolic link '/burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/lib/x86\_64/gcc/mpi/include': File exists
> 
> 
> Working Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/CCTM/scripts
> Build Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/CCTM/scripts/BLD\_CCTM\_v54\_gcc
> Output Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/data/output\_CCTM\_v54\_gcc\_Bench\_2018\_12NE3
> Working Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/CCTM/scripts
> Log Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/data/output\_CCTM\_v54\_gcc\_Bench\_2018\_12NE3/LOGS
> Build Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/CCTM/scripts/BLD\_CCTM\_v54\_gcc
> Executable Name is CCTM\_v54.exe
> Output Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/data/output\_CCTM\_v54\_gcc\_Bench\_2018\_12NE3
> Log Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/data/output\_CCTM\_v54\_gcc\_Bench\_2018\_12NE3/LOGS
> Executable Name is CCTM\_v54.exe
> Compiler is set to gcc
> mkdir: cannot create directory ‘/burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/lib/x86\_64/gcc/mpi’: File exists
> 
> Working Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/CCTM/scripts
> Build Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/CCTM/scripts/BLD\_CCTM\_v54\_gcc
> Output Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/data/output\_CCTM\_v54\_gcc\_Bench\_2018\_12NE3
> Log Directory is /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/data/output\_CCTM\_v54\_gcc\_Bench\_2018\_12NE3/LOGS
> Executable Name is CCTM\_v54.exe
> 
> 
> \---CMAQ EXECUTION ID: CMAQ\_CCTMv54\_sha=79a966615c\_gg2995\_20250312\_011208\_788809303 ---
> \---CMAQ EXECUTION ID: CMAQ\_CCTMv54\_sha=79a966615c\_gg2995\_20250312\_011208\_788844484 ---
> 
> \---CMAQ EXECUTION ID: CMAQ\_CCTMv54\_sha=79a966615c\_gg2995\_20250312\_011208\_791996632 ---
> 
> Existing Logs and Output Files for Day 2018-07-01 Will Be Deleted
> 
> Set up input and output files for Day 2018-07-01.
> 
> Set up input and output files for Day 2018-07-01.
> 
> Set up input and output files for Day 2018-07-01.
> /bin/rm: No match.
> 
> CMAQ Processing of Day 20180701 Began at Tue Mar 11 21:12:08 EDT 2025
> 
> cat: buff.txt: No such file or directory
> 
> Existing Logs and Output Files for Day 2018-07-01 Will Be Deleted
> 
> Existing Logs and Output Files for Day 2018-07-01 Will Be Deleted
> /bin/rm: No match.
> 
> Existing Logs and Output Files for Day 2018-07-01 Will Be Deleted
> /bin/rm: No match.
> /bin/rm: No match.
> 
> 
> 
> CMAQ Processing of Day 20180701 Began at Tue Mar 11 21:12:08 EDT 2025
> 
> CMAQ Processing of Day 20180701 Began at Tue Mar 11 21:12:08 EDT 2025
> 
> CMAQ Processing of Day 20180701 Began at Tue Mar 11 21:12:08 EDT 2025
> 
> srun: Job 1316301 step creation temporarily disabled, retrying (Requested nodes are busy)
> srun: Job 1316301 step creation temporarily disabled, retrying (Requested nodes are busy)
> srun: Job 1316301 step creation temporarily disabled, retrying (Requested nodes are busy)
> srun: Job 1316301 step creation temporarily disabled, retrying (Requested nodes are busy)
> srun: Job 1316301 step creation still disabled, retrying (Requested nodes are busy)
> srun: Job 1316301 step creation still disabled, retrying (Requested nodes are busy)
> srun: Job 1316301 step creation still disabled, retrying (Requested nodes are busy)
> srun: Job 1316301 step creation still disabled, retrying (Requested nodes are busy)
> srun: Job 1316301 step creation still disabled, retrying (Requested nodes are busy)
> \`\`\`
> 
> 
> This is the slurm script:
> 
> \`\`\`
> \#!/bin/bash
> \#SBATCH -A fiore # Account
> \#SBATCH --job-name=test\_run # The job name
> \#SBATCH -c 32 # Number of cores
> \#SBATCH -N 4 # Ensure that all cores are on one machine
> \#SBATCH -t 0-12:00 # Runtime in D-HH:MM
> \#SBATCH --exclusive # Memory pool for all cores
> \#SBATCH -C mem768
> \#SBATCH --mem=700GB
> 
> source /burg/ocp/users/gg2995/CMAQ/cmaq\_spack.env
> source /burg/ocp/users/gg2995/CMAQ/.profile
> 
> export OMP\_NUM\_THREADS=32
> export F\_UFMTENDIAN=big
> export OMP\_STACKSIZE=500m
> 
> srun -c $OMP\_NUM\_THREADS time -p /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/CCTM/scripts/run\_cctm\_Bench\_2018\_12NE3.csh \> output.log 2\>&1
> \`\`\`
> 
> The domain decomposition is :
> 
> \`\`\`
> \#\> Horizontal domain decomposition
> if ( $PROC == serial ) then
> setenv NPCOL\_NPROW "1 1"; set NPROCS = 1 # single processor setting
> else
> @ NPCOL = 8; @ NPROW = 4
> @ NPROCS = $NPCOL \* $NPROW
> setenv NPCOL\_NPROW "$NPCOL $NPROW";
> endif
> \`\`\`
> 
> I guess the model is giving error due to my incorrect input in the scheduler. Please help me refine this script. 
> 
> Thanks in advance for the help,
> Gopikrishnan

---

<div class="post-metadata">

**Author:** ![hogrefe.christian](https://avatars.discourse-cdn.com/v4/letter/h/74df32/32.png) [@hogrefe.christian](https://forum.cmascenter.org/u/hogrefe.christian)\
**Post date:** [March 14, 2025, 5:33pm UTC](https://forum.cmascenter.org/t/installing-cmaq-on-hpc/3101/5 "2025-03-14T17:33:55Z")

</div>

Hello @Gopikrishnan ,

as @Ryan had mentioned in his response to the original post in this thread, this type of slurm error is not specifically related to CMAQ and is something your institution’s IT department likely is in a better position to help you with than the users on this forum (though of course there may be some folks here who have specific insights - I do not).

Your run script contains several non-standard elements that caught my attention, though I do not know if they are related to your issue with slurm. CMAQ does not use OMP so I am unsure about the intent of those instructions.

```auto
source /burg/ocp/users/gg2995/CMAQ/cmaq_spack.env
source /burg/ocp/users/gg2995/CMAQ/.profile

export OMP_NUM_THREADS=32
export F_UFMTENDIAN=big
export OMP_STACKSIZE=500m

```

Please also note that opening issues on the CMAQ github repository is not the preferred way to ask for technical help on CMAQ, [while posting on this forum is](https://github.com/USEPA/CMAQ/wiki/CMAQv5.5-Series-FAQ#technical-support-for-cmaq). We will go ahead and close the issue you have opened on the CMAQ github repository shortly.

---

<div class="post-metadata">

**Author:** ![Gopikrishnan](https://avatars.discourse-cdn.com/v4/letter/g/41988e/32.png) [@Gopikrishnan](https://forum.cmascenter.org/u/Gopikrishnan)\
**Post date:** [March 14, 2025, 7:38pm UTC](https://forum.cmascenter.org/t/installing-cmaq-on-hpc/3101/6 "2025-03-14T19:38:52Z")

</div>

HI @hogrefe.christian,

Happy to hear from you,

I am sorry for including the OPENMPI there, and removed this particular section from the script.

> export OMP\_NUM\_THREADS=32  
> export F\_UFMTENDIAN=big  
> export OMP\_STACKSIZE=500m

The source file contents are:  
**.profile**

> export PATH=/burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/LIBRARIES/mpich/bin:$PATH  
> export PATH=/burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/LIBRARIES/netcdf/bin:$PATH  
> export NETCDF=/burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/LIBRARIES/netcdf  
> export LD\_LIBRARY\_PATH=/burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/LIBRARIES/netcdf/lib:$LD\_LIBRARY\_PATH

**cmaq\_spack (for the compiler)**

> source /burg/home/gg2995/spack/share/spack/setup-env.sh  
> spack load gcc@9.1.0  
> export CC=gcc  
> export CXX=g++  
> export FC=gfortran  
> export F77=gfortran

**The srun command is also modified as:**

> srun /burg/ocp/users/gg2995/CMAQ/Build\_CMAQ/CMAQ\_Project/CCTM/scripts/run\_cctm\_Bench\_2018\_12NE3.csh \> output.log 2\>&1

However, the same error still exists.!

---

<div class="post-metadata">

**Author:** ![hogrefe.christian](https://avatars.discourse-cdn.com/v4/letter/h/74df32/32.png) [@hogrefe.christian](https://forum.cmascenter.org/u/hogrefe.christian)\
**Post date:** [March 17, 2025, 11:26am UTC](https://forum.cmascenter.org/t/installing-cmaq-on-hpc/3101/7 "2025-03-17T11:26:57Z")

</div>

I have no in-depth familiarity with configuring srun with SBATCH options.

On our system, we use the following to run the benchmark case with a 8x4 domain composition on 32 cores:

> #SBATCH -n 32

which just specifies the number of tasks to run but does not specify a specific number of nodes or CPUs per task. Your setup instead uses

> #SBATCH -c 32 # Number of cores  
> #SBATCH -N 4 # Ensure that all cores are on one machine

which (if I read the srun man pages correctly) would request 32 CPUs per task and also request 4 nodes. You also specify

> #SBATCH --exclusive # Memory pool for all cores  
> #SBATCH -C mem768  
> #SBATCH --mem=700GB

which is not something we typically do in our CCTM scripts.

You could try using the simpler instructions we use (just specify the number of tasks as 32 for a 8x4 domain decomposition), but there might be things specific to your system that require the more specific instructions you are providing. Your organization’s IT department might be a good resource for that.

---

<div class="post-metadata">

**Author:** ![wong.david-c](https://avatars.discourse-cdn.com/v4/letter/w/bcef8e/32.png) [@wong.david-c](https://forum.cmascenter.org/u/wong.david-c)\
**Post date:** [March 19, 2025, 3:39pm UTC](https://forum.cmascenter.org/t/installing-cmaq-on-hpc/3101/8 "2025-03-19T15:39:31Z")

</div>

Hi Gopikrishnan,

Based on the detailed description page you have provided, it seems to me that the problem is not related to CMAQ but the way how to submit a job on your system proper. Please try to run this simple program on your system using the same method. If you can’t run this (that confirms my believe), please consult (as Christian has suggested) with your system administrator.

Cheers,  
David