# I can't find any error in log when i running CCTM script

**URL:** <https://forum.cmascenter.org/t/i-cant-find-any-error-in-log-when-i-running-cctm-script/4075>\
**Category:** Run Time Errors and Issues\
**Tags:** cmaq\
**Created:** [April 23, 2023, 3:44pm UTC](https://forum.cmascenter.org/t/i-cant-find-any-error-in-log-when-i-running-cctm-script/4075 "2023-04-23T15:44:17Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Doha](https://avatars.discourse-cdn.com/v4/letter/d/b5e925/32.png) [@Doha](https://forum.cmascenter.org/u/Doha)\
**Post date:** [April 23, 2023, 3:44pm UTC](https://forum.cmascenter.org/t/i-cant-find-any-error-in-log-when-i-running-cctm-script/4075/1 "2023-04-23T15:44:18Z")

</div>

I had a very strange problem about running CCTM and you can only see the cctm.log which I use the command ‘./nm\_27.csh \>& cctm.log’ . I haven’t seen any error in cctm.log. But I didn’t sucessfully run it yet.  
I can’t upload files because I’m a new user. Here are the last few lines in the log file.

_setenv CTM\_STDATE 2020276_  
_setenv CTM\_STTIME 010000_  
_setenv CTM\_RUNLEN 240000_  
_setenv CTM\_TSTEP 010000_  
_setenv EMIS\_1 /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/data/neimeng/emis/emis.mrg.neimeng.NM27.cmaq\_cb05ae6.25hrs.2020271.ncf_  
_setenv INIT\_GASC\_1 /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/data/output\_CCTM\_v52\_intel\_201001/CCTM\_CGRID\_v52\_intel\_201001\_20201001.nc_  
_setenv INIT\_AERO\_1 /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/data/output\_CCTM\_v52\_intel\_201001/CCTM\_CGRID\_v52\_intel\_201001\_20201001.nc_  
_setenv INIT\_NONR\_1 /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/data/output\_CCTM\_v52\_intel\_201001/CCTM\_CGRID\_v52\_intel\_201001\_20201001.nc_  
_setenv INIT\_TRAC\_1 /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/data/output\_CCTM\_v52\_intel\_201001/CCTM\_CGRID\_v52\_intel\_201001\_20201001.nc_  
_setenv BNDY\_GASC\_1 /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/data/neimeng/icbc/BCON\_v52\_neimeng\_profile_  
_setenv BNDY\_AERO\_1 /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/data/neimeng/icbc/BCON\_v52\_neimeng\_profile_  
_setenv BNDY\_NONR\_1 /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/data/neimeng/icbc/BCON\_v52\_neimeng\_profile_  
_setenv BNDY\_TRAC\_1 /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/data/neimeng/icbc/BCON\_v52\_neimeng\_profile_  
_setenv OMI /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/OMI\_1979\_to\_2015.dat_  
_setenv OPTICS\_DATA /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/PHOT\_OPTICS.dat_  
_set TR\_DVpath = /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/data/neimeng/mcip/NM\_27_  
_set TR\_DVfile = /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/data/neimeng/mcip/NM\_27/METCRO2D\_201001.nc_  
_setenv gc\_matrix\_nml /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/GC\_cb05e51\_ae6\_aq.nml_  
_setenv ae\_matrix\_nml /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/AE\_cb05e51\_ae6\_aq.nml_  
_setenv nr\_matrix\_nml /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/NR\_cb05e51\_ae6\_aq.nml_  
_setenv tr\_matrix\_nml /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/Species\_Table\_TR\_0.nml_  
_setenv CSQY\_DATA /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/CSQY\_DATA\_cb05e51\_ae6\_aq_  
_if ( ! ( -e /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/CSQY\_DATA\_cb05e51\_ae6\_aq ) ) then_  
_if ( ! ( -e /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/PHOT\_OPTICS.dat ) ) then_  
_ls -l /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/CCTM\_v52.exe_  
_-rwxrwxr-x 1 lzh lzh 22841356 Apr 21 00:51 /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/CCTM\_v52.exe_  
_size /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/CCTM\_v52.exe_

- text data bss dec hex filename\*  
_6275173 15665504 102050376 123991053 763f40d /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/CCTM\_v52.exe_  
_unlimit_  
_limit_  
_cputime unlimited_  
_filesize unlimited_  
_datasize unlimited_  
_stacksize unlimited_  
_coredumpsize unlimited_  
_memoryuse unlimited_  
_vmemoryuse unlimited_  
\*descriptors 32768 \*  
_memorylocked unlimited_  
\*maxproc 257548 \*  
_date_  
_Sun Apr 23 23:25:27 CST 2023_  
_mpirun -r ssh -np 36 /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/CCTM\_v52.exe_  
_--------------------------------------------------------------------------_  
_mpirun was unable to launch the specified application as it could not find an executable:_

_Executable: -r_  
_Node: compute-0-10.local_

_while attempting to start process rank 0._  
_--------------------------------------------------------------------------_  
_96 total processes failed to start_  
_0.016u 0.033s 0:00.15 26.6% 0+0k 0+32io 0pf+0w_  
_date_  
_Sun Apr 23 23:25:27 CST 2023_  
_if ( ! -e /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/data/output\_CCTM\_v52\_intel\_201001/LOGS ) then_  
_mv: No match._  
_setenv NEW\_START false_  
_set TODAYG = `date -ud "${TODAYG}+1days" +%Y-%m-%d`_  
_date -ud 2020-10-02+1days +%Y-%m-%d_  
_set TODAYJ = `date -ud "${TODAYG}" +%Y%j`_  
_date -ud 2020-10-03 +%Y%j_  
_end_  
_while ( 2020277 \< = 2020276 )_  
_exit_

_My CMAQ version is 5.2. Is there anyone who can help me? THANKS_

---

<div class="post-metadata">

**Author:** ![lizadams](https://avatars.discourse-cdn.com/v4/letter/l/49beb7/32.png) [@lizadams](https://forum.cmascenter.org/u/lizadams)\
**Post date:** [April 24, 2023, 3:12pm UTC](https://forum.cmascenter.org/t/i-cant-find-any-error-in-log-when-i-running-cctm-script/4075/2 "2023-04-24T15:12:12Z")

</div>

> [@Doha](#):
>
> /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/CCTM\_v52.exe

Please verify that the executable is located in the specified path.

ls -lrt _/public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/CCTM\_v52.exe_

I think that the issue is the -r ssh after the mpirun command.

Please try modifying the run script to use the following:

mpirun -np 36 /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/CCTM\_v52.exe\*

---

<div class="post-metadata">

**Author:** ![Doha](https://avatars.discourse-cdn.com/v4/letter/d/b5e925/32.png) [@Doha](https://forum.cmascenter.org/u/Doha)\
**Post date:** [April 24, 2023, 5:08pm UTC](https://forum.cmascenter.org/t/i-cant-find-any-error-in-log-when-i-running-cctm-script/4075/3 "2023-04-24T17:08:33Z")

</div>

Hi，lizadams  
thanks for your kindly reply  
My executable should exist exactly where it shows.  
An error occurred when I ran the run script following your advice.  
I also want to show the modified script, please help to have a look. Here are the scripts and errors：

```auto
# ===================================================================
#> Execution Portion
# ===================================================================

  #> Print attributes of the executable
  ls -l $BLD/$EXEC; size $BLD/$EXEC
  unlimit
  limit

  date

  #> Executable call for single PE, uncomment to invoke
  # /usr/bin/time $BLD/$EXEC

  #> Executable call for multi PE, configure for your system 
   set MPI = /public/software/mpi/mpich/intel/3.2/bin
   set MPIRUN = $MPI/mpirun

# time mpirun -np 36 $NPROCS $BLD/$EXEC
  mpirun -np 36 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CCTM_v52.exe*

  date

# ===================================================================
#> Finalize Run for This Day and Loop to Next Day
# ===================================================================

  #> Save Log Files and Move on to Next Simulation Day
  if (! -e $LOGDIR ) then
    mkdir $LOGDIR
  endif
  mv CTM_LOG_???.${CTM_APPL} $LOGDIR

  #> The next simulation day will, by definition, be a restart
  setenv NEW_START false

  #> Increment both Gregorian and Julian Days
  set TODAYG = `date -ud "${TODAYG}+1days" +%Y-%m-%d` #> Add a day for tomorrow
  set TODAYJ = `date -ud "${TODAYG}" +%Y%j` #> Convert YYYY-MM-DD to YYYYJJJ

end #Loop to the next Simulation Day

exit

```

```auto
unlimit
limit
cputime unlimited
filesize unlimited
datasize unlimited
stacksize unlimited
coredumpsize unlimited
memoryuse unlimited
vmemoryuse unlimited
descriptors 4096 
memorylocked unlimited
maxproc 257509 
date
Tue Apr 25 01:06:49 CST 2023
set MPI = /public/software/mpi/mpich/intel/3.2/bin
set MPIRUN = /public/software/mpi/mpich/intel/3.2/bin/mpirun
mpirun -np 36 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CCTM_v52.exe
[cluster.hpc.org:2012] *** An error occurred in MPI_comm_size
[cluster.hpc.org:2012] *** on communicator MPI_COMM_WORLD
[cluster.hpc.org:2012] *** MPI_ERR_COMM: invalid communicator
[cluster.hpc.org:2012] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort
--------------------------------------------------------------------------
mpirun has exited due to process rank 11 with PID 2022 on
node cluster.hpc.org exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
[cluster.hpc.org:02010] 35 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[cluster.hpc.org:02010] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
date
Tue Apr 25 01:06:49 CST 2023
if ( ! -e /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/data/output_CCTM_v52_intel_201001/LOGS ) then
mv: No match.
setenv NEW_START false
set TODAYG = `date -ud "${TODAYG}+1days" +%Y-%m-%d`
date -ud 2020-10-02+1days +%Y-%m-%d
set TODAYJ = `date -ud "${TODAYG}" +%Y%j`
date -ud 2020-10-03 +%Y%j
end
while ( 2020277 < = 2020276 )
exit

```

---

<div class="post-metadata">

**Author:** ![Doha](https://avatars.discourse-cdn.com/v4/letter/d/b5e925/32.png) [@Doha](https://forum.cmascenter.org/u/Doha)\
**Post date:** [April 25, 2023, 3:31am UTC](https://forum.cmascenter.org/t/i-cant-find-any-error-in-log-when-i-running-cctm-script/4075/4 "2023-04-25T03:31:16Z")

</div>

I can upload files！  
This my run script and run.log.  
[nm\_27.csh](https://forum.cmascenter.org/uploads/short-url/777z6f9TEbKZlijrT1HrNMfIknw.csh) (24.5 KB)  
[run.txt](https://forum.cmascenter.org/uploads/short-url/39oYFIc7tm8a0s0eWxmed8WDTZM.txt) (78.0 KB)

---

<div class="post-metadata">

**Author:** ![lizadams](https://avatars.discourse-cdn.com/v4/letter/l/49beb7/32.png) [@lizadams](https://forum.cmascenter.org/u/lizadams)\
**Post date:** [April 25, 2023, 12:18pm UTC](https://forum.cmascenter.org/t/i-cant-find-any-error-in-log-when-i-running-cctm-script/4075/5 "2023-04-25T12:18:54Z")

</div>

In the run.txt that you attached, you used the -r ssh command after mpirun. That gave you this error:

```auto
mpirun was unable to launch the specified application as it could not find an executable:
**Executable: -r**

```

Please **remove** the **-r ssh** from the command .

This line in the run script:  
mpirun -r ssh -np 8 /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/CCTM\_v52.exe

should be replaced by:

mpirun -np 8 /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/CCTM\_v52.exe

In the nm\_27.csh, you correctly removed the `-r ssh`, but you are using a different number of processors.

mpirun **-np 36** /public/home/lzh/HHJ/Build\_WRF/CMAQ-5.2/CCTM/scripts/BLD\_CCTM\_v52\_intel/CCTM\_v52.exe

You need to know how many cpus are available on your cluster. You can try to run the following command.

`sinfo -lN`

If you still get an error, you should try to run a hello world program on multiple processors to verify that your environment is set up correctly.  
[https://mpitutorial.com/tutorials/mpi-introduction/](https://mpitutorial.com/tutorials/mpi-introduction/)