Hi all. I’ve compiled v5.4 succesfully and am now trying to run it on my university’s cluster. I tested it in an interactive session a week ago and it worked fine, so I stopped the script with ctrl-c. Now today, I’m running into errors. My module list has GCC/9.3.0 and OpenMPI/4.0.3, which I used to compile. Here is a paste of my run_cctm script. I had a previous issue where the program wasn’t finding my net-cdf installation due to an error in my LD_LIBRARY_PATH, but I edited my cshrc accordingly and now this is my LD_LIBRARY_PATH containing my netcdf installations which fixed it.
This pastebin link is my output with an error from running CMAQ on the cluster. I’m assuming it’s not getting past the first fortran file given this error:
At line 88 of file setup_logdev.F
Fortran runtime error: Illegal seek
It appears from your log file that the I/O API banner is repeated 16 times, once for each task, when typically this should be opened once and then the environment variable report is generated. It is almost like each task is trying to it’s own copy of CMAQ , rather than the work within CMAQ being divided evenly among the tasks.
Please try to use the default method, on your cluster, to see if it works.
You may need to load modules required to run CMAQ such as openmpi, in addition to using the LD_LIBRARY_PATH.
I am a bit worried that it is trying to run 16 different CMAQ runs on each processor, but not really sure how to fix it. I’ve loaded OpenMPI version 4.0.3.
I tried, it’s the same issue unfortunately. The part that confuses me the most is this Fortran error. LOGDEV seems to be an object created by IOAPI, so I wonder if I’ve built IOAPI wrong at all. I did not install netcdf-4 support per recommendation on this forum and the IOAPI website. Could that be why I can’t use the 2018 benchmark? I know it was mentioned somewhere that may cause problems.
Sorry, I just remembered that you are using the 12NE3 Domain, which should be already uncompressed. You can test to see if I/O API is compiled correctly by using one of the m3tools such as m3stat.
You can also use the command to get a file that uses netCDF-4 versus netCDF-3.
I discovered the path to my netcdf/bin directory was malformed in ~/.cshrc and fixed that (note: in the CMAQ install docs, it recommends $NCDIR\bin instead of $NCDIR/bin, which was my issue). I also added my IOAPI files to my path. I rebuilt CMAQ (in case the incorrect path was the cuplrit) but I still get the same “Illegal seek” error. M3stat ran fine on the file you sent me with netcdf-3 compression, so I don’t know if I should try to rebuild IOAPI.
Yes, I have loaded the same versions of GCC and OpenMPI I used for compiling. Here’s the output of module list:
Unfortunately, I don’t have much advice for you because on my system using srun to start parallel MPI jobs doesn’t work as expected, mostly because I don’t think my version of SLURM was installed with the proper linkages to MPI to allow srun to work. However, I am following up with my system administrator.
From my tests, OpenMPI “srun” doesn’t work at all and crashes immediately with an error saying that SLURM wasn’t configured with this option. Intel-MPI seems to try, but launches several copies of the executable instead of 1 copy with the work divided.
I advise you to do two things:
Work with you system administrator to understand the linkages with MPI and SLURM on your system.
Work with a smaller problem to try and successfully see if you can run a MPI job, for example, for my tests cases I don’t run CMAQ but a simple “Hello, World” MPI code. Once I verify this works, then I know how to submit MPI type jobs on my system. Something like the below:
program hello_mpi
use mpi
implicit none
integer ierr, mype, npes
call MPI_INIT( ierr )
call MPI_COMM_SIZE( MPI_COMM_WORLD, npes, ierr )
call MPI_COMM_RANK( MPI_COMM_WORLD, mype, ierr )
write(*,*) 'Hello, world from process ', mype, ’ of ', npes
call MPI_FINALIZE( ierr )
end program hello_mpi
I compiled this program using my compiler of choice then try to run it in parallel. A successful “run” should look something like this with 4 processors :
$shell: mpirun -np 4 hello_world_mpi.exe
Hello, world from process 3 of 4
Hello, world from process 1 of 4
Hello, world from process 2 of 4
Hello, world from process 0 of 4
Thanks for the detailed response. I was able to compile and run the Hello, world program with this output. I compiled with the flags OpenMPI gave from running mpifort --showme:link.
[gap6@nlogin1 gap6]$ srun --partition=interactive --ntasks=4 --mem=1G --time=00:01:00 hello_world
srun: job 7849588 queued and waiting for resources
srun: job 7849588 has been allocated resources
Hello, world from process 0 of 4
Hello, world from process 1 of 4
Hello, world from process 2 of 4
Hello, world from process 3 of 4
Looks like it worked. I will also try to get in touch with a previous lab member, as they were able to run CMAQ on Rice computing resources. Their code I’ve seen shows a similar approach to me so not sure what I’m doing wrong.
EDIT 1: One possible source of error: for compiling the dependencies (netcdf, netcdf-fortran, and IOAPI), I believe I used gfortran and gcc for my FC and CC respectively. Was I supposed to use mpifort and mpicc so the wrapper versions would have the right flags for my MPI configuration?
EDIT 2: I tried recompiling all dependencies, consistently using mpifort and mpicc for compilation. Still the same problem.
As you know, mpifort and mpicc are really “wrappers” for a pair of underlying Fortran and C compilers (gfortran and gcc, in your case, I think). Unless you’re using the distributed-I/O variant, neither the I/O API nor netCDF use MPI at all, so it is sufficient to use those underlying Fortran and C compilers to compile them.
If you are using distributed-I/O, then you of course do need to use the same mpifort and mpicc as for your CCTM.
