I can't find any error in log when i running CCTM script

I had a very strange problem about running CCTM and you can only see the cctm.log which I use the command ‘./nm_27.csh >& cctm.log’ . I haven’t seen any error in cctm.log. But I didn’t sucessfully run it yet.
I can’t upload files because I’m a new user. Here are the last few lines in the log file.

setenv CTM_STDATE 2020276
setenv CTM_STTIME 010000
setenv CTM_RUNLEN 240000
setenv CTM_TSTEP 010000
setenv EMIS_1 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/data/neimeng/emis/emis.mrg.neimeng.NM27.cmaq_cb05ae6.25hrs.2020271.ncf
setenv INIT_GASC_1 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/data/output_CCTM_v52_intel_201001/CCTM_CGRID_v52_intel_201001_20201001.nc
setenv INIT_AERO_1 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/data/output_CCTM_v52_intel_201001/CCTM_CGRID_v52_intel_201001_20201001.nc
setenv INIT_NONR_1 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/data/output_CCTM_v52_intel_201001/CCTM_CGRID_v52_intel_201001_20201001.nc
setenv INIT_TRAC_1 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/data/output_CCTM_v52_intel_201001/CCTM_CGRID_v52_intel_201001_20201001.nc
setenv BNDY_GASC_1 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/data/neimeng/icbc/BCON_v52_neimeng_profile
setenv BNDY_AERO_1 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/data/neimeng/icbc/BCON_v52_neimeng_profile
setenv BNDY_NONR_1 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/data/neimeng/icbc/BCON_v52_neimeng_profile
setenv BNDY_TRAC_1 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/data/neimeng/icbc/BCON_v52_neimeng_profile
setenv OMI /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/OMI_1979_to_2015.dat
setenv OPTICS_DATA /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/PHOT_OPTICS.dat
set TR_DVpath = /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/data/neimeng/mcip/NM_27
set TR_DVfile = /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/data/neimeng/mcip/NM_27/METCRO2D_201001.nc
setenv gc_matrix_nml /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/GC_cb05e51_ae6_aq.nml
setenv ae_matrix_nml /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/AE_cb05e51_ae6_aq.nml
setenv nr_matrix_nml /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/NR_cb05e51_ae6_aq.nml
setenv tr_matrix_nml /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/Species_Table_TR_0.nml
setenv CSQY_DATA /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CSQY_DATA_cb05e51_ae6_aq
if ( ! ( -e /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CSQY_DATA_cb05e51_ae6_aq ) ) then
if ( ! ( -e /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/PHOT_OPTICS.dat ) ) then
ls -l /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CCTM_v52.exe
-rwxrwxr-x 1 lzh lzh 22841356 Apr 21 00:51 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CCTM_v52.exe
size /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CCTM_v52.exe

  • text data bss dec hex filename*
    6275173 15665504 102050376 123991053 763f40d /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CCTM_v52.exe
    unlimit
    limit
    cputime unlimited
    filesize unlimited
    datasize unlimited
    stacksize unlimited
    coredumpsize unlimited
    memoryuse unlimited
    vmemoryuse unlimited
    *descriptors 32768 *
    memorylocked unlimited
    *maxproc 257548 *
    date
    Sun Apr 23 23:25:27 CST 2023
    mpirun -r ssh -np 36 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CCTM_v52.exe
    --------------------------------------------------------------------------
    mpirun was unable to launch the specified application as it could not find an executable:

Executable: -r
Node: compute-0-10.local

while attempting to start process rank 0.
--------------------------------------------------------------------------
96 total processes failed to start
0.016u 0.033s 0:00.15 26.6% 0+0k 0+32io 0pf+0w
date
Sun Apr 23 23:25:27 CST 2023
if ( ! -e /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/data/output_CCTM_v52_intel_201001/LOGS ) then
mv: No match.
setenv NEW_START false
set TODAYG = date -ud "${TODAYG}+1days" +%Y-%m-%d
date -ud 2020-10-02+1days +%Y-%m-%d
set TODAYJ = date -ud "${TODAYG}" +%Y%j
date -ud 2020-10-03 +%Y%j
end
while ( 2020277 < = 2020276 )
exit

My CMAQ version is 5.2. Is there anyone who can help me? THANKS

Please verify that the executable is located in the specified path.

ls -lrt /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CCTM_v52.exe

I think that the issue is the -r ssh after the mpirun command.

Please try modifying the run script to use the following:

mpirun -np 36 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CCTM_v52.exe*

Hi,lizadams
thanks for your kindly reply
My executable should exist exactly where it shows.
An error occurred when I ran the run script following your advice.
I also want to show the modified script, please help to have a look. Here are the scripts and errors:

# ===================================================================
#> Execution Portion
# ===================================================================

  #> Print attributes of the executable
  ls -l $BLD/$EXEC; size $BLD/$EXEC
  unlimit
  limit

  date

  #> Executable call for single PE, uncomment to invoke
  # /usr/bin/time  $BLD/$EXEC

  #> Executable call for multi PE, configure for your system 
   set MPI = /public/software/mpi/mpich/intel/3.2/bin
   set MPIRUN = $MPI/mpirun

# time mpirun -np 36 $NPROCS $BLD/$EXEC
  mpirun -np 36 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CCTM_v52.exe*

  date

# ===================================================================
#> Finalize Run for This Day and Loop to Next Day
# ===================================================================

  #> Save Log Files and Move on to Next Simulation Day
  if (! -e $LOGDIR ) then
    mkdir $LOGDIR
  endif
  mv CTM_LOG_???.${CTM_APPL} $LOGDIR

  #> The next simulation day will, by definition, be a restart
  setenv NEW_START false

  #> Increment both Gregorian and Julian Days
  set TODAYG = `date -ud "${TODAYG}+1days" +%Y-%m-%d` #> Add a day for tomorrow
  set TODAYJ = `date -ud "${TODAYG}" +%Y%j` #> Convert YYYY-MM-DD to YYYYJJJ

end  #Loop to the next Simulation Day

exit
unlimit
limit
cputime      unlimited
filesize     unlimited
datasize     unlimited
stacksize    unlimited
coredumpsize unlimited
memoryuse    unlimited
vmemoryuse   unlimited
descriptors  4096 
memorylocked unlimited
maxproc      257509 
date
Tue Apr 25 01:06:49 CST 2023
set MPI = /public/software/mpi/mpich/intel/3.2/bin
set MPIRUN = /public/software/mpi/mpich/intel/3.2/bin/mpirun
mpirun -np 36 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CCTM_v52.exe
[cluster.hpc.org:2012] *** An error occurred in MPI_comm_size
[cluster.hpc.org:2012] *** on communicator MPI_COMM_WORLD
[cluster.hpc.org:2012] *** MPI_ERR_COMM: invalid communicator
[cluster.hpc.org:2012] *** MPI_ERRORS_ARE_FATAL: your MPI job will now abort
--------------------------------------------------------------------------
mpirun has exited due to process rank 11 with PID 2022 on
node cluster.hpc.org exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
[cluster.hpc.org:02010] 35 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[cluster.hpc.org:02010] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
date
Tue Apr 25 01:06:49 CST 2023
if ( ! -e /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/data/output_CCTM_v52_intel_201001/LOGS ) then
mv: No match.
setenv NEW_START false
set TODAYG = `date -ud "${TODAYG}+1days" +%Y-%m-%d`
date -ud 2020-10-02+1days +%Y-%m-%d
set TODAYJ = `date -ud "${TODAYG}" +%Y%j`
date -ud 2020-10-03 +%Y%j
end
while ( 2020277 < = 2020276 )
exit

I can upload files!
This my run script and run.log.
nm_27.csh (24.5 KB)
run.txt (78.0 KB)

In the run.txt that you attached, you used the -r ssh command after mpirun. That gave you this error:

mpirun was unable to launch the specified application as it could not find an executable:
**Executable: -r**

Please remove the -r ssh from the command .

This line in the run script:
mpirun -r ssh -np 8 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CCTM_v52.exe

should be replaced by:

mpirun -np 8 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CCTM_v52.exe

In the nm_27.csh, you correctly removed the -r ssh, but you are using a different number of processors.

mpirun -np 36 /public/home/lzh/HHJ/Build_WRF/CMAQ-5.2/CCTM/scripts/BLD_CCTM_v52_intel/CCTM_v52.exe

You need to know how many cpus are available on your cluster. You can try to run the following command.

sinfo -lN

If you still get an error, you should try to run a hello world program on multiple processors to verify that your environment is set up correctly.
https://mpitutorial.com/tutorials/mpi-introduction/