My first stop to find answers for SMOKE and CAMx questions is usually the CMAS forum, but this time I did not find any discussions about running CAMx in parallel mode using OpenMP. As a relatively new systems administrator for our Linux servers I made many mistakes and went down many dead ends before I finally got CAMx working properly with OpenMP - all a part of the learning process. To make the process smoother for other new CAMx users I would like to share my recent experience installing CAMx v7.32 to run the Ramboll test case in OpenMP parallel mode on a 44-core shared-memory Dell T640. For context I am working within the minimum install version of Rocky Linux 10.1, and needed these additions:
dnf groupinstall “Development Tools”
dnf install epel-release
dnf install netcdf-devel
to get the Intel Fortran compiler
download compiler using wget https://registrationcenter-download.intel.com/akdlm/IRC_NAS/ce0f9b00-4780-483f-bc09-96d6fb4467ca/intel-fortran-essentials-2025.3.1.26_offline.sh
run installation file using sh ./intel-fortran-essentials-2025.3.1.26_offline.sh -a --silent --cli --eula accept
create file called ‘intel_compilers.sh’ in /etc/profile.d/ containing this line: PATH=/opt/intel/oneapi/compiler/2025.3/bin:$PATH
change to CAMx build directory
cd /home/SHARED/CAMx
to get the CAMx source code
download using wget https://www.camx.com/getmedia/CAMx_v7.32.src.250801.tgz
uncompress directories using tar -xvzf CAMx_v7.32.src.250801.tgz
to get the Ramboll CAMx test case files
download using wget https://www.camx.com/getmedia/CAMx7-32-test_run-inputs_met-250801.tgz
uncompress directory using tar -xvzf CAMx7-32-test_run-inputs_met-250801.tgz
download using wget https://www.camx.com/getmedia/CAMx7-32-test_run-inputs_emiss-250801.tgz
uncompress directory using tar -xvzf CAMx7-32-test_run-inputs_emiss-250801.tgz
download using wget https://www.camx.com/getmedia/CAMx7-32-test_run-inputs_other-250801.tgz
uncompress directory using tar -xvzf CAMx7-32-test_run-inputs_other-250801.tgz
download using wget https://www.camx.com/getmedia/v7.32.specific.inputs.250801.tgz
uncompress directory using tar -xvzf v7.32.specific.inputs.250801.tgz
to install the CAMx prerequisite software for NetCDF
follow instructions in https://www.camx.com/download/netcdf
cd /home/SHARED/CAMx/CAMx_BLD/
The CAMx v7.32 manual does not discuss the preparation of the prerequisite software. I noted that the instructions on https://www.camx.com/download/netcdf/ used serial C and Fortran compilers as defaults for building NetCDF, and reasoned that for parallel mode CAMx, the prerequisite software should be compiled using parallel C and Fortran compilers. Google searches turned up these examples of parallel configure flags:
./configure --prefix=/root/CAMx_BLD/hdf5-1.12.0 --enable-fortran --enable-parallel --enable-parallel-tools --with-zlib --disable-shared
./configure --prefix=/root/CAMx_BLD/netcdf-c-4.6.1 --disable-dap --disable-libxml2 --enable-pnetcdf
This led into rabbit hole of incompatible builds and untraceable CAMx errors that cost me several weeks of work until I reexamined my assumptions about using parallel C and Fortran compilers then got in touch with a Ramboll scientist who had this to say:
“CAMx does not use parallel NetCDF. Do not do anything for parallel NetCDF.”
So after rebuilding zlib-1.2.11, hdf5-1.12.0, netcdf-c-4.6.1 and netcdf-fortran-4.4.5 with serial C (gcc) and Fortran (ifx) compilers I ran
source /opt/intel/oneapi/2025.3/oneapi-vars.sh
then to address the ‘libnetcdff.so.7 not found’ and ‘libimf.so not found’ errors I updated $LD_LIBRARY_PATH using
export LD_LIBRARY_PATH=“/opt/intel/oneapi/2025.3/lib:/home/SHARED/CAMx/CAMx_BLD/ifx-netcdf-c-4.6.1/lib:$LD_LIBRARY_PATH”
next I updated the MPI and NetCDF directories in the CAMx Makefile
cd /home/SHARED/CAMx/CAMx7v32/src.v7.32/
vi /home/SHARED/CAMx/CAMx7v32/src.v7.32/Makefile
MPI_INST = /opt/intel/oneapi/mpi/2021.17 #(placeholder since MPI won’t be used)
NCF_INST = /home/SHARED/CAMx/CAMx_BLD/ifx-netcdf-c-4.6.1
next I compiled CAMx without MPICH/OpenMPI or IEEE #(no support in OneAPI?)
make clean
make COMPILER=ifortomp NCF=NCF4_C
using the CAMx test case files, I updated the directory with the new executable file in CAMx_v7.32.36.12.20160610-11.noMPI_NCF4_ifortomp.job
cd /home/SHARED/CAMx/CAMx7v32/runfiles/
vi CAMx_v7.32.36.12.20160610-11.noMPI_NCF4_ifortomp.job
set EXEC = “../src.v7.32/CAMx.v7.32.noMPI.NCF4.ifortomp”
finally I ran the jobscript
./CAMx_v7.32.36.12.20160610-11.noMPI_NCF4_ifortomp.job
Not many other changes need to be made to the default jobscript, the default settings for
setenv NCPUS 12
setenv OMP_NUM_THREADS 12
yield the shortest execution time for the fewest CPUs and threads based on my tests.
The Ramboll scientist also recommends using top to monitor the CAMx job while it executes in parallel:
"there is [may be] only one instance of CAMx, but it is using over 600% of a single CPU (611.0). This is because that program sometimes spawns 12 CPUs while performing a given task. Other times it is just 1 processor. So, it averaged out to that number over this time sampling period. The %CPU number will bounce around as the simulation progresses.
If you are just running OMP you can check that it is actually using parallelization by checking the %CPU usage"