Could not write XXX to CTM_CONC_1 --- MPI_ABORT

CMAQ
1

Hi,
I have read other assistance on this similar topic. However, those suggestions did not resolve my error. I would appreciate any insights. Thank you.

I am trying to run the bench_2018_12NE3_cracmm2 in parallel mode using openmp. I was able to run the bench in SERIAL but cannot get it running in MPI. However, in MPI mode I receive the following message. I have deleted my build folder and recompiled. After delete the output folder and then running, the output folder contains a CONC file with the header information, but CCTM fails when trying to write data to the file.

‘’’
*** ERROR ABORT in subroutine OPCONC on PE 000
Could not write O3 to CTM_CONC_1

MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
Proc: [[9741,0],0]
Errorcode: 1
‘’’

CMAQ v5.5
GNU Fortran (GCC) 11.5.0 20240719 (Red Hat 11.5.0-5)
ioapi-3.2: Id: init3.F90 247 2023-03-22 15:59:19Z coats
netCDF version 4.9.3 of Jun 18 2025 14:55:43 $

Below are my build, run and log files.
bldit_cctm_cramm2.csh.txt (37.3 KB)
CTM_LOG_000.v55_gcc_Bench_2018_12NE3_cracmm2_stage_20180701.txt (47.5 KB)
run_cctm_Bench_2018_12NE3_CRACMM2_bwz.csh.txt (39.8 KB)
v55_gcc_Bench_2018_12NE3_cracmm2_stage_20180701.txt (47.5 KB)
CCTM_v55_gcc_Bench_2018_12NE3_cracmm2_stage_20180701.cfg.txt (3.9 KB)

2

This is the error message in your CTM_LOG_000 file:

       
        
     "CTM_CONC_1" opened as NEW(READ-WRITE )
     File name "/data/cmaq/data/output_CCTM_v55_gcc_Bench_2018_12NE3_cracmm2_stage/CCTM_CONC_v55_gcc_Bench_2018_12NE3_cracmm2_stage_20180701.nc"
     File type GRDDED3 
     Execution ID "CMAQ_CCTMv55_sha=a1c596eaf4_root_20250625_170645_110883345"
     Grid name "2018_12NE3"
     Dimensions: 105 rows, 100 cols, 1 lays, 13 vbles
     NetCDF ID:   2490368  opened as VOLATILE READWRITE  
     Starting date and time  2018182:000000 (0:00:00   July 1, 2018)
     Timestep                          010000 (1:00:00 hh:mm:ss)
     Maximum current record number         0
  
     Gas Chem species saved to CONC file:

     *** ERROR ABORT in subroutine OPCONC on PE 000          
     Could not write O3 to CTM_CONC_1
 PM3EXIT:  DTBUF 0:00:00   July 1, 2018  
     Date and time 0:00:00   July 1, 2018   (2018182:000000)

In your bldit_cctm.csh, you have uncommented both ParOpt and DistEnvr environment variables.

Unless you are working on distributed environment then you should comment out the DistEnvr variable.

set ParOpt                             #> uncomment to build a multiple processor (MPI) executable; 
                                       #>   comment out for a single processor (serial) executable
#set DistrEnv                          #> uncomment to distribute environmental variables to multiple machines
                                       #>   comment out for a single processor (serial) executable (MPI only)

Please try to comment out this environment variable and re-run bldit_cctm.csh, and then try re-running CMAQ.

If that does not solve the problem, then please verify that you use the same MPI libraries to build I/O API as you use to build CMAQ.

Please share your environment.

Using environment modules, you can see what modules are loaded using module list

Example:

module list

Output:


Currently Loaded Modules:
  1) ioapi-3.2/gcc-13.3-classic   2) netcdf-4.5.3-classic/gcc-13.3-classic   3) gcc/13.3.0   4) openmpi_5.0.6/gcc_13.3.0

Alternatively, you can use `echo $LD_LIBRARY_PATH to see what libraries are available when you run the bldit_cctm.csh script.

Example:

echo $LD_LIBRARY_PATH

Output:

/nas/sycamore/apps/openmpi/5.0.6_gcc-13.3.0/lib:/nas/sycamore/apps/gcc/13.3.0/lib:/nas/sycamore/apps/gcc/13.3.0/lib64:/work/users/l/i/lizadams/test_classic_libraries_sycamore/LIBRARIES_gcc_disable-dap/lib:/work/users/l/i/lizadams/test_classic_libraries_sycamore/LIBRARIES_gcc_disable-dap/ioapi-3.2/ioapi/fixed_src

Reviewing your run_cctm_Bench_2018_12NE3_cracmm2_stage_20180701.txt file, it appears that you have commented out the mpirun command, so you may be only running in serial mode? Or this is the serial version of your run script.

  #> Executable call for single PE, uncomment to invoke
  ( /usr/bin/time -p $BLD/$EXEC  --allow-run-as-root) |& tee buff_${EXECUTION_ID}.txt

  #> Executable call for multi PE, configure for your system 
  # set MPI = /usr/local/intel/impi/3.2.2.006/bin64
  # set MPIRUN = $MPI/mpirun
  #( /usr/bin/time -p mpirun -np $NPROCS $BLD/$EXEC ) |& tee buff_${EXECUTION_ID}.txt

Please also share the settings in your config_cmaq.csh.

2 Likes
3

Thank you, LizAdams. After chasing many wrong leads, I re-started the library installations again, following: Frequent CMAQ Questions | CMAQ: The Community Multiscale Air Quality Modeling System | US EPA
Specifically, the section what you need, and what you don’t need. Although, I had done that several times, previously.
Again, running CCTM in parallel created an error once it started trying to write to CONC out file. I finally figured out, that I was compiling CCTM in DEBUG. When I turned commented out the debug option in BLDIT.csh, then CCTM runs okay in parallel.
…Brian

1 Like
4

That said, there must be something else going on. On a different linux system (B), we compile BLDIT with “set Debug_CCTM” un-commented, and on that system (B) CCTM runs okay in parallel with debug. I noticed that on my system (A), when I run in Debug I get many debug warning messages for temporary array creation, which are not displayed on system (B):
“Fortran runtime warning: An array temporary was created
At line xxx of file ELMO_PROC.F”

System (A) uses: GNU Fortran (GCC) 11.5.0 20240719 (Red Hat 11.5.0-5)
System (B) uses: GNU Fortran (GCC) 8.5.0 20210514 (Red Hat 8.5.0-26)

So, the issue may be in a debug warning message outputs on system (A) which is affecting parallel output.
…Brian

1 Like