Error while running CCTM

Catalyst2682 · August 30, 2019, 12:22pm

Dear @lizadams , @bbaek , @tlspero, @cjcoats and CMAQ Users,

I encountered an error while running CMAQ. Bellow is the last section of the log file for the run. Attached is the entire log file. Kindly help me out.

—>> WARNING in subroutine INIT_MET:INTERPX
Variable “SURF” not in file OCEAN_1
M3WARN: DTBUF 1:00:00 Jan. 29, 2016 (2016029:010000)

 *** ERROR ABORT in subroutine INIT_MET on PE 000        
  Error interpolating variable SURF from OCEAN_1
 Date and time 1:00:00   Jan. 29, 2016  (2016029:010000)

MPI_ABORT was invoked on rank 5 in communicator MPI_COMM_WORLD
with errorcode 21913.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

[catalyst-Precision-Tower-3620:16639] 2 more processes have sent help message help-mpi-api.txt / mpi-abort
[catalyst-Precision-Tower-3620:16639] Set MCA parameter “orte_base_help_aggregate” to 0 to see all help / error messages
12.427u 16.210s 0:14.26 200.7% 0+0k 0+246608io 30pf+0w
set TASKMAP = /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/machines
cat /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/machines
n001:24
mkdir -p /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/logs/2016029
mv ./CTM_LOG_001.36km_2016029 ./CTM_LOG_002.36km_2016029 ./CTM_LOG_003.36km_2016029 ./CTM_LOG_004.36km_2016029 ./CTM_LOG_005.36km_2016029 ./CTM_LOG_006.36km_2016029 ./CTM_LOG_007.36km_2016029 /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/logs/2016029
@ i = 27 + 1
end
while ( 28 < 28 )
date
Fri Aug 30 00:52:45 CST 2019
exit

Thanks

Catalystrun_cctm_log.txt (1.6 MB)

lizadams · August 30, 2019, 2:02pm

The OCEAN file is created by the Spatial Allocator Tool, and contains a variable that CMAQ-5.2 requires called SURF.

Please use the following command to verify what variable names are in your file.

ncdump -h  /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/PREP/ocean/2018/meic/china.36km.oceanfile.new_ola.ncf |& tee china.36km.oceanfile.new_ola.ncf_header.txt

Examine the china.36km.oceanfile.new_ola.ncf_header.txt file to determine what variables it contains. I am not sure how you generated this input file.

I obtained the path for your OCEAN_1 file from the log, as it is specified when the environment variable was set:

setenv OCEAN_1 /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/PREP/ocean/2018/meic/china.36km.oceanfile.new_ola.ncf

Please use these instructions to create your OCEAN_1 file:

https://github.com/CMASCenter/Spatial-Allocator/blob/master/docs/User_Manual/SA_ch03_vector.md#computing-area-percentages/SA_ch03_vector.md#allocate-mode-variables

Documentation on how to install the Spatial Allocator:

Link to the script: alloc_surf_zone_to_oceanfile.csh

Catalyst2682 · August 30, 2019, 3:12pm

Dear @lizadams ,
Thanks for your prompt response. Can you please guide me on this aspect of alloc_surf_zone_to_oceanfile.csh script?:

#set “data” shapefile parameters
#setenv GRIDDESC $DATADIR/GRIDDESC.txt
setenv GRIDDESC /home/catalyst/Desktop/Build_WRF/Megan/megan/megan_china_v2.10/work/GRIDDESC_36km
#set parameters for file being allocated
setenv INPUT_FILE_NAME $DATADIR/surfzone/surfzone_NC_SC
setenv INPUT_FILE_TYPE ShapeFile
setenv INPUT_FILE_MAP_PRJN "+proj=lcc,+lat_1=33,+lat_2=45,+lat_0=40,+lon_0=-97"
setenv INPUT_FILE_ELLIPSOID “+a=6370000.0,+b=6370000.0”
setenv ALLOCATE_ATTRS TYPE
setenv ALLOC_MODE_FILE ALL_AREAPERCENT

#Set this to SURF_ZONE to create the variables needed for CMAQ OCEANfile
setenv ALLOC_ATTR_TYPE SURF_ZONE

Set name and path of resulting shapefile

setenv OUTPUT_FILE_TYPE IoapiFile
#setenv OUTPUT_GRID_NAME NC4KM
setenv OUTPUT_GRID_NAME 36km
setenv OUTPUT_FILE_MAP_PRJN "+proj=lcc,+lat_1=33,+lat_2=45,+lat_0=40,+lon_0=-97"
setenv OUTPUT_FILE_ELLIPSOID “+a=6370000.0,+b=6370000.0”
setenv OUTPUT_FILE_NAME OUTPUT/ocean_file_{OUTPUT_GRID_NAME}.ncf

Those bold lines, I don’t know how to go about them. Attached are the files found in data directory of spatial allocator. How do I make use of them? My model area is China. Can you please share the link to download “spatial_allocator_v4.3_Feb2017.data.tar.gz”, please?

Thanks so much.

Catalystspatial_allocator_v4.3.1_June2017.data.update_only.tar.gz.txt (1.3 KB) spatial_allocator_v4.3_Feb2017.data.tar.gz.txt (36.2 KB)

lizadams · August 30, 2019, 6:27pm

Here are a few more resources on this topic.
FAQ:
https://www.epa.gov/cmaq/frequent-cmaq-questions#OCEAN

Tutorial:
https://github.com/USEPA/CMAQ/blob/master/DOCS/Users_Guide/Tutorials/CMAQ_UG_tutorial_oceanfile.md

The input file name is the name of a shapefile that covers your domain.
In the example, the name of the shapefile was surfzone_NC_SC

Typically, the shapefile name corresponds to 4-5 files that have an extension of .dbf, .prj, .shp, .shx, and sometimes additional extensions.

An example of a shapefile that covers US states is provided in the Spatial Allocator Release:

surfzone_poly_st.dbf
surfzone_poly_st.prj
surfzone_poly_st.sbn
surfzone_poly_st.sbx
surfzone_poly_st.shp
surfzone_poly_st.shp.xml
surfzone_poly_st.shx

The projection information is found in the file with the .prj extension. This file can be examined with a more command.

> more surfzone_poly_st.prj

output of the more command:

PROJCS["Sphere_ARC_INFO_Lambert_Conformal_Conic",GEOGCS["GCS_Sphere_ARC_INFO",DATUM["D_Sphere_ARC_INFO",SPHEROID["Sphere_ARC_INFO",6370997.0,0.0]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Lambert_Conformal_Conic"],PARAMETER["False_Easting",0.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-97.0],PARAMETER["Standard_Parallel_1",33.0],PARAMETER["Standard_Parallel_2",45.0],PARAMETER["Latitude_Of_Origin",40.0],UNIT["Meter",1.0]]

The input file map projection then needs to be specified in the script on this line:

setenv INPUT_FILE_MAP_PRJN "+proj=lcc,+lat_1=33,+lat_2=45,+lat_0=40,+lon_0=-97"

To examine the example shapefile, you can view the shapefile using the website:
https://mapshaper.org/

Import the example shapefiles into your browser.
Click on console
type tips into the console
type info to view the attribute data contained within the shapefile

The attributes for the surfzone_poly_st_shapefile shapefile:

[info] 
=========================
Layer 1 *
-------------------------
Name:     surfzone_poly_st
Type:     polygon
Records:  20,183
Bounds:   -2423753.499559599,-2882042.25,3235764.999559599,2841043.75
CRS:      +proj=lcc +lon_0=-97 +lat_1=33 +lat_2=45 +lat_0=40 +a=6370997 +es=0

Attribute data
------------+----------------
 Field      | First value
------------+----------------
 AREA       | 16138100000000
 DESCRIP    | 'LAND'
 FID_state_ |             -1
 FID_surfzo |              0
 LAND_SUR_1 |              1
 LAND_SURF2 |              2
 PERIMETER  |      106741000
 STATE      | ''
 TYPE       |              2
------------+----------------
$

For China, I believe you will need to create or obtain a shapefile that identifies land, ocean and surf-zone areas. I hope that others can help outlines the steps required to do this.
Page 20 of the following describes a method:

From the following website:
https://www.epa.gov/cmaq/processing-spatial-data-cmaq-inputs
Sea salt emissions** require open ocean and surface zone (50m) buffer fractions for the modeling grid cells in an I/O API file. For most of North American domain, a SA Vector allocation tool can be used to generate the surf zone and open ocean file from a polygon shapefile with land, surf zone buffer, and open ocean in SA data directory. For areas outside U.S., users have to generate a surf zone polygon shapefile with has the same attribute as the file in the SA in order to use the tool.

In the CMAQv5.3 benchmark case, the OCEAN file was provided for the 12US1 domain, and it contains 2 variables OPEN (which has a value of 1 over ocean, and a value of the fraction of the grid cell that is over ocean for the grid cells that are over both surfzone and ocean, and a value of zero over land) and SURF (which has a value of the fraction of the grid cell that is over ocean if the grid cell is over both ocean and surfzone, and a value of zero over land and value zero over ocean. You use VERDI to visualize this example OCEAN file, and use the Control > Probe to determine the values.

If you are running a domain that does not contain a coastline, then I believe there are other options that are described in the tutorials above.

Catalyst2682 · August 31, 2019, 10:11am

Dear @lizadams @cjcoats @bbaek @tlspero run_cctm.log.txt (3.0 MB)

Thanks for your response and guide. I have created a dummy ocean file and no such error again. However, the cctm is yet to be successfully run. Below is the error I got:

—>> WARNING in subroutine FLCHECK on PE 000
Inconsistent header data on input files
M3WARN: DTBUF 1:00:00 Jan. 31, 2016 (2016031:010000)

 = = = = = = = = = = = = = =  End  FLCHECK = = = = = = = = = = = = = =

 Value for CTM_MAXSYNC not defined; returning default:  720
 Maximum Synchronization Time Step (sec)                                         
 Value for CTM_MINSYNC not defined; returning default:  60
 Minimum Synchronization Time Step (sec)                                         
 Value for CTM_ADV_CFL not defined; returning default :  0.75
 Maximum CFL number allowed                                                      
 Value for SIGMA_SYNC_TOP not defined; returning default :  0.7
 Minimum layer limit for which adv = sync                                        
 Value for ADV_HDIV_LIM not defined; returning default :  0.9
 Maximum horiz. div. limit for adv step adjustment                               
 Top layer thru which sync step determined: 12

 From ADVSTEP - date/time:  2016031/010000

 Computed synchronization step (HHMMSS): 001200
 Number of Synchronization steps:    5


 Layer   Advection   per Sync
       Step (HHMMSS)  Step
   17      000600       2
   16      000400       3
   15      000600       2
   14      000600       2
   13      001200       1
   12      001200       1
   11      001200       1
   10      001200       1
    9      001200       1
    8      001200       1
    7      001200       1
    6      001200       1
    5      001200       1
    4      001200       1
    3      001200       1
    2      001200       1
    1      001200       1
 Value for CTM_CKSUM not defined;returning default:   TRUE
 Cksum on flag                                                                   
 Environment variable not set ... Using default:                                     10000
 Value for CTM_ILDEPV:  Y returning TRUE
 Value for CTM_ABFLUX:  N returning FALSE
 Value for CTM_SFC_HONO:  Y returning TRUE
 Value for CTM_MOSAIC:  N returning FALSE
 Value for CTM_HGBIDI:  N returning FALSE
 Value for CTM_DEPV_FILE:  N returning FALSE
 Flag for writing the DEPV diagnostic file                                       
 DEPV_INIT: writes GAS DEPV values to CTM_DEPV_FILE                                                                      
 DEPV_INIT: completed INIT_GAS_DV block 

 Value for KZMIN:  Y returning TRUE
 Value for CTM_GRAV_SETL not defined;returning default:   TRUE
 Value for VDIFF_DIAG_FILE not defined;returning default:   FALSE
 CTM_DRY_DEP_1   :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_DRYDEP_36km_2016031.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 Could not open CTM_DRY_DEP_1    file for update - try to open new

MPI_ABORT was invoked on rank 7 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

[catalyst-Precision-Tower-3620:19100] 5 more processes have sent help message help-mpi-api.txt / mpi-abort
[catalyst-Precision-Tower-3620:19100] Set MCA parameter “orte_base_help_aggregate” to 0 to see all help / error messages
14.867u 19.748s 0:14.68 235.6% 0+0k 0+23512io 20pf+0w
set TASKMAP = /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/machines
cat /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/machines
n001:24
mkdir -p /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/logs/2016031
mv ./CTM_LOG_001.36km_2016031 ./CTM_LOG_002.36km_2016031 ./CTM_LOG_003.36km_2016031 ./CTM_LOG_004.36km_2016031 ./CTM_LOG_005.36km_2016031 ./CTM_LOG_006.36km_2016031 ./CTM_LOG_007.36km_2016031 /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/logs/2016031
@ i = 30 + 1
end
while ( 31 < 31 )
date
Sat Aug 31 18:04:19 CST 2019
exit

Attached is the log file of the cctm run.

Thanks for your usual help.

Catalyst

cjcoats · August 31, 2019, 2:43pm

There’s very probably something wrong with your MET_* files, e.g.:

     "MET_BDY_3D" opened as OLD:READ-ONLY   
     File name "/media/catalyst/Catalyst/mcip/36km/METBDY3D_2016-01-31"
     File type BNDARY3 
     Execution ID "mcip"
     Grid name "METCRO_36km_CROS"
     Dimensions: 79 rows, 95 cols, 17 lays, 18 vbles, 1 cells thick
     NetCDF ID:    917504  opened as READONLY            
     Starting date and time  2016001:010000 (1:00:00   Jan. 1, 2016)
     Timestep                          010000 (1:00:00 hh:mm:ss)
     Maximum current record number       738
     Checking header data for file: MET_BDY_3D
     netCDF error number  -40
     Ending time not on file

and similarly for MET_DOT_3D, MET_CRO_2D, MET_CRO_3D.

That “Maximum current record number 738” is curious; I’m wondering if something went wrong with your MCIP scripting, causing there to be “holes” in the files. It might be enlightening to run something like M3Tools program m3stat on these files, to see exactly what data is there…

Catalyst2682 · September 2, 2019, 9:48am

Dear @cjcoats
I guess its because of the MCIP4.5 that I used for WRF4.0 wrfout file. I hope once the error in MCIP5.0 is corrected by @tlspero, I will be able to run a successful MCIP.

Thanks in anticipation.

Catalyst

esm · September 2, 2019, 11:12am

Hi
I have a problem with running cctm. Could you guide me, please?
This is the log file:
run_cctmm.log.txt (163.4 KB)

The errors are:
“GRID_BDY_2D” opened as OLD:READ-ONLY
File name “/home/ensiyeh/WRF-CMAQ/CMAQ-OFFLINE/CMAQ-5.2/data/cctm/met/mcip/GRIDBDY2D_171213”
File type BNDARY3
Execution ID “mcip”
Grid name “GRIDOUT_SE52BENC”
Dimensions: 64 rows, 73 cols, 1 lays, 31 vbles, 1 cells thick
NetCDF ID: 786432 opened as READONLY
Time-independent data.
Checking header data for file: GRID_BDY_2D
Error opening file at path-name:
netCDF error number -51 processing file “EMIS_1”
NetCDF: Unknown file format
/home/ensiyeh/WRF-CMAQ/CMAQ-OFFLINE/CMAQ-5.2/data/cctm/emis/gridded_area/emission.2017121300.d03.bin
Error opening file at path-name:
netCDF error number -51 processing file “OCEAN_1”
NetCDF: Unknown file format
/home/ensiyeh/WRF-CMAQ/CMAQ-OFFLINE/CMAQ-5.2/data/cctm/land/12US1_surf_bench.nc
MET_BDY_2D :MET_BDY_2D

 *** ERROR ABORT in subroutine DRIVER on PE 000          
 *** Failure defining domain configuration
 Date&time specified as 0

cjcoats · September 3, 2019, 1:43pm

The above says that:

You haven’t done the “setenv” for GRID_BDY_2D correctly;
EMIS_1 and OCEAN_1 are not a correctly-structured netCDF I/O API files.

Catalyst2682 · September 17, 2019, 11:50am

Dear @lizadams , @cjcoats , @bbaek , @tlspero, @hogrefe.christian

This is my log file for cctm. I got errors as shown below and the file is attached too.

—>> WARNING in subroutine FLCHECK on PE 000
Inconsistent header data on input files
M3WARN: DTBUF 0:00:00 Jan. 31, 2016 (2016031:000000)

 = = = = = = = = = = = = = =  End  FLCHECK = = = = = = = = = = = = = =

 Value for CTM_MAXSYNC not defined; returning default:  720
 Maximum Synchronization Time Step (sec)                                         
 Value for CTM_MINSYNC not defined; returning default:  60
 Minimum Synchronization Time Step (sec)                                         
 Value for CTM_ADV_CFL not defined; returning default :  0.75
 Maximum CFL number allowed                                                      
 Value for SIGMA_SYNC_TOP not defined; returning default :  0.7
 Minimum layer limit for which adv = sync                                        
 Value for ADV_HDIV_LIM not defined; returning default :  0.9
 Maximum horiz. div. limit for adv step adjustment                               
 Top layer thru which sync step determined: 12

 From ADVSTEP - date/time:  2016031/000000

 Computed synchronization step (HHMMSS): 001200
 Number of Synchronization steps:    5


 Layer   Advection   per Sync
       Step (HHMMSS)  Step
   17      000600       2
   16      000400       3
   15      000600       2
   14      000600       2
   13      001200       1
   12      001200       1
   11      001200       1
   10      001200       1
    9      001200       1
    8      001200       1
    7      001200       1
    6      001200       1
    5      001200       1
    4      001200       1
    3      001200       1
    2      001200       1
    1      001200       1
 Value for CTM_CKSUM:  N returning FALSE
 Value for CTM_ILDEPV:  Y returning TRUE
 Value for CTM_ABFLUX:  N returning FALSE
 Value for CTM_SFC_HONO not defined;returning default:   TRUE
 Flag for HONO interaction with surfaces

MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

 Environment variable not set ... Using default:                                                                                 0
 Value for CTM_MOSAIC:  N returning FALSE
 Value for CTM_HGBIDI:  N returning FALSE
 Value for CTM_DEPV_FILE:  N returning FALSE
 Flag for writing the DEPV diagnostic file                                       
 DEPV_INIT: writes GAS DEPV values to CTM_DEPV_FILE                                                                      
 DEPV_INIT: completed INIT_GAS_DV block 

 Value for KZMIN:  Y returning TRUE
 Value for CTM_GRAV_SETL:  N returning FALSE
 Using J-,K-mode aerosols gravitational settling                                 
 Value for VDIFF_DIAG_FILE:  N returning FALSE
 Writing the VDIFF diagnostic files                                              
 CTM_DRY_DEP_1   :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_DRYDEP_36km_2016031.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 Could not open CTM_DRY_DEP_1    file for update - try to open new

[catalyst-Precision-Tower-3620:16077] 4 more processes have sent help message help-mpi-api.txt / mpi-abort
[catalyst-Precision-Tower-3620:16077] Set MCA parameter “orte_base_help_aggregate” to 0 to see all help / error messages
19.319u 17.636s 0:12.26 301.3% 0+0k 0+23504io 22pf+0w
mkdir -p /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/logs/2016031
mv ./CTM_LOG_001.36km_2016031 ./CTM_LOG_002.36km_2016031 ./CTM_LOG_003.36km_2016031 ./CTM_LOG_004.36km_2016031 ./CTM_LOG_005.36km_2016031 ./CTM_LOG_006.36km_2016031 ./CTM_LOG_007.36km_2016031 /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/logs/2016031
setenv NEW_START true
@ i = 30 + 1
end
while ( 31 < 31 )
date
Tue Sep 17 19:33:45 CST 2019
exit

Kindly assist me.

cctm.log.txt (2.9 MB) Thanks

Catalyst

Catalyst2682 · September 18, 2019, 12:18pm

Dear @lizadams, @cjcoats, @bbaek , @tlspero , @cgnolte, @hogrefe.christian

Below is the last part of my cctm log file:

—>> WARNING in subroutine FLCHECK on PE 000
Inconsistent header data on input files
M3WARN: DTBUF 0:00:00 Jan. 31, 2016 (2016031:000000)

 = = = = = = = = = = = = = =  End  FLCHECK = = = = = = = = = = = = = =

 Value for CTM_MAXSYNC not defined; returning default:  720
 Maximum Synchronization Time Step (sec)                                         
 Value for CTM_MINSYNC not defined; returning default:  60
 Minimum Synchronization Time Step (sec)                                         
 Value for CTM_ADV_CFL not defined; returning default :  0.75
 Maximum CFL number allowed                                                      
 Value for SIGMA_SYNC_TOP not defined; returning default :  0.7
 Minimum layer limit for which adv = sync                                        
 Value for ADV_HDIV_LIM not defined; returning default :  0.9
 Maximum horiz. div. limit for adv step adjustment                               
 Top layer thru which sync step determined: 12

 From ADVSTEP - date/time:  2016031/000000

 Computed synchronization step (HHMMSS): 001200
 Number of Synchronization steps:    5


 Layer   Advection   per Sync
       Step (HHMMSS)  Step
   17      000600       2
   16      000400       3
   15      000600       2
   14      000600       2
   13      001200       1
   12      001200       1
   11      001200       1
   10      001200       1
    9      001200       1
    8      001200       1
    7      001200       1
    6      001200       1
    5      001200       1
    4      001200       1
    3      001200       1
    2      001200       1
    1      001200       1
 Value for CTM_CKSUM:  N returning FALSE
 Value for CTM_ILDEPV:  Y returning TRUE
 Value for CTM_ABFLUX:  N returning FALSE
 Value for CTM_SFC_HONO not defined;returning default:   TRUE
 Flag for HONO interaction with surfaces                                         
 Environment variable not set ... Using default:                                                                                 0
 Value for CTM_MOSAIC:  N returning FALSE
 Value for CTM_HGBIDI:  N returning FALSE

MPI_ABORT was invoked on rank 7 in communicator MPI_COMM_WORLD
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

 Value for CTM_DEPV_FILE:  N returning FALSE
 Flag for writing the DEPV diagnostic file                                       
 DEPV_INIT: writes GAS DEPV values to CTM_DEPV_FILE                                                                      
 DEPV_INIT: completed INIT_GAS_DV block 

 Value for KZMIN:  Y returning TRUE
 Value for CTM_GRAV_SETL:  N returning FALSE

[catalyst-Precision-Tower-3620:08499] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[catalyst-Precision-Tower-3620:08499] Set MCA parameter “orte_base_help_aggregate” to 0 to see all help / error messages
11.380u 13.851s 0:15.36 164.2% 0+0k 0+23480io 26pf+0w
mkdir -p /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/logs/2016031
mv ./CTM_LOG_001.36km_2016031 ./CTM_LOG_002.36km_2016031 ./CTM_LOG_003.36km_2016031 ./CTM_LOG_004.36km_2016031 ./CTM_LOG_005.36km_2016031 ./CTM_LOG_006.36km_2016031 ./CTM_LOG_007.36km_2016031 /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/logs/2016031
setenv NEW_START true
@ i = 30 + 1
end
while ( 31 < 31 )
date
Wed Sep 18 18:44:58 CST 2019
exit

Kindly assist me in resolving this errors. Attached is the full log file of the simulation.

Thanks in anticipation.

Catalyst

cctm_log.txt (3.0 MB)

hogrefe.christian · September 18, 2019, 6:33pm

Hello Catalyst,

as suggested by @lizadams in a separate thread, could you please check your processor log files for the first day simulation in $OUTDIR/LOGS (likely under /home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/logs/2016001)? Are there any error messages towards the end of one or more of the CTM_LOG* files? You can use the following command to check:

grep -i error CTM_LOG*

Your main cctm.log.txt file reports the following at the end of the first day:
“MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD”

so the error may be documented in greater detail in CTM_LOG_002*

Christian

Catalyst2682 · September 18, 2019, 6:49pm

Dear @hogrefe.christian ,
Below is the log file. The error recorded here is common to all the log files.
Thanks in anticipation.

Catalyst

[CTM_LOG_001.36km_2016002.txt|

attachment]
(upload://1MnLE8v4kXLAzOhs3wdam5bGiGj.txt) (51.3 KB)

 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 Could not open A_CONC_1 file for update - try to open new
 Value for CLD_DIAG:  Y returning TRUE

after INITSCEN G 1.2201706E-01 A 1.2130194E+09 N 9.7443575E-05

 = = = = = = = = = = = = = = Start FLCHECK = = = = = = = = = = = = = =

 Value for FL_ERR_STOP:  N returning FALSE
 Value for CTM_RUNLEN:  240000
  
 "GRID_DOT_2D" opened as OLD:READ-ONLY   
 File name "/media/catalyst/Catalyst/mcip/36km/GRIDDOT2D_2016-01-02"
 File type GRDDED3 
 Execution ID "mcip"
 Grid name "GRIDOUT_36km_DOT"
 Dimensions: 80 rows, 96 cols, 1 lays, 9 vbles
 NetCDF ID:    393216  opened as READONLY            
 Time-independent data.
 Checking header data for file: GRID_DOT_2D
 Checking header data for file: GRID_CRO_2D
 GRID_CRO_3D     :GRID_CRO_3D
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
  
 "GRID_BDY_2D" opened as OLD:READ-ONLY   
 File name "/media/catalyst/Catalyst/mcip/36km/GRIDBDY2D_2016-01-02"
 File type BNDARY3 
 Execution ID "mcip"
 Grid name "GRIDOUT_36km_CRO"
 Dimensions: 79 rows, 95 cols, 1 lays, 28 vbles, 1 cells thick
 NetCDF ID:    458752  opened as READONLY            
 Time-independent data.
 Checking header data for file: GRID_BDY_2D
  
 "EMIS_1" opened as OLD:READ-ONLY   
 File name "/home/catalyst/Desktop/Build_WRF/LIBRARIES/allocation/PM2.5/meic16_reas2_combined/egts_meic16_reas2.36km.aero6.anth.20160102.ncf"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "36km"
 Dimensions: 79 rows, 95 cols, 9 lays, 58 vbles
 NetCDF ID:    524288  opened as READONLY            
 Starting date and time  2016002:000000 (0:00:00   Jan. 2, 2016)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number        25
 Checking header data for file: EMIS_1
     Inconsistent values for NTHIK: 0 versus 1
  
 "OCEAN_1" opened as OLD:READ-ONLY   
 File name "/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/PREP/ocean/ocean_file_36km.ncf"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "36km"
 Dimensions: 79 rows, 95 cols, 1 lays, 2 vbles
 NetCDF ID:    589824  opened as READONLY            
 Time-independent data.
 Checking header data for file: OCEAN_1
 MET_BDY_2D      :MET_BDY_2D
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
  
 "MET_BDY_3D" opened as OLD:READ-ONLY   
 File name "/media/catalyst/Catalyst/mcip/36km/METBDY3D_2016-01-02"
 File type BNDARY3 
 Execution ID "mcip"
 Grid name "METCRO_36km_CROS"
 Dimensions: 79 rows, 95 cols, 17 lays, 18 vbles, 1 cells thick
 NetCDF ID:    655360  opened as READONLY            
 Starting date and time  2016002:000000 (0:00:00   Jan. 2, 2016)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number        25
 Checking header data for file: MET_BDY_3D
  
 "MET_DOT_3D" opened as OLD:READ-ONLY   
 File name "/media/catalyst/Catalyst/mcip/36km/METDOT3D_2016-01-02"
 File type GRDDED3 
 Execution ID "mcip"
 Grid name "METDOT_36km_DOT"
 Dimensions: 80 rows, 96 cols, 17 lays, 6 vbles
 NetCDF ID:    720896  opened as READONLY            
 Starting date and time  2016002:000000 (0:00:00   Jan. 2, 2016)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number        25
 Checking header data for file: MET_DOT_3D
  
 "MET_CRO_2D" opened as OLD:READ-ONLY   
 File name "/media/catalyst/Catalyst/mcip/36km/METCRO2D_2016-01-02"
 File type GRDDED3 
 Execution ID "mcip"
 Grid name "METCRO_36km_CROS"
 Dimensions: 79 rows, 95 cols, 1 lays, 35 vbles
 NetCDF ID:    786432  opened as READONLY            
 Starting date and time  2016002:000000 (0:00:00   Jan. 2, 2016)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number        25
 Checking header data for file: MET_CRO_2D
 Checking header data for file: MET_CRO_3D
  
 "CTM_CONC_1" opened as OLD:READ-ONLY   
 File name "/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_CONC_36km_2016002.nc"
 File type GRDDED3 
 Execution ID "CCTM_36km"
 Grid name "36km"
 Dimensions: 79 rows, 95 cols, 17 lays, 219 vbles
 NetCDF ID:    851968  opened as VOLATILE READONLY   
 Starting date and time  2016002:000000 (0:00:00   Jan. 2, 2016)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number         1
 Checking header data for file: CTM_CONC_1
 CTM_DRY_DEP_1   :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_DRYDEP_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
  
 "CTM_WET_DEP_1" opened as OLD:READ-ONLY   
 File name "/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_WETDEP1_36km_2016002.nc"
 File type GRDDED3 
 Execution ID "CCTM_36km"
 Grid name "36km"
 Dimensions: 79 rows, 95 cols, 1 lays, 138 vbles
 NetCDF ID:    917504  opened as VOLATILE READONLY   
 Starting date and time  2016002:010000 (1:00:00   Jan. 2, 2016)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number         0
 Checking header data for file: CTM_WET_DEP_1
  
 "CTM_WET_DEP_2" opened as OLD:READ-ONLY   
 File name "/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_WETDEP2_36km_2016002.nc"
 File type GRDDED3 
 Execution ID "CCTM_36km"
 Grid name "36km"
 Dimensions: 79 rows, 95 cols, 1 lays, 149 vbles
 NetCDF ID:    983040  opened as VOLATILE READONLY   
 Starting date and time  2016002:010000 (1:00:00   Jan. 2, 2016)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number         0
 Checking header data for file: CTM_WET_DEP_2
 CTM_SSEMIS_1    :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_SSEMIS_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_DUST_EMIS_1 :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_DUSTEMIS_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_VIS_1       :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_PMVIS_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_PMDIAG_1    :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_PMDIAG_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_AOD_1       :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_AOD_DIAG_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_IPR_1       :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_PA_1_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_IPR_2       :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_PA_2_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_IPR_3       :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_PA_3_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_IRR_1       :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_IRR_1_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_IRR_2       :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_IRR_2_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_IRR_3       :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_IRR_3_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
  
 "A_CONC_1" opened as OLD:READ-ONLY   
 File name "/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_ACONC_36km_2016002.nc"
 File type GRDDED3 
 Execution ID "CCTM_36km"
 Grid name "36km"
 Dimensions: 79 rows, 95 cols, 1 lays, 233 vbles
 NetCDF ID:   1048576  opened as VOLATILE READONLY   
 Starting date and time  2016002:000000 (0:00:00   Jan. 2, 2016)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number         0
 Checking header data for file: A_CONC_1
 S_CGRID         :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_CGRID_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 Checking header data for file: INIT_GASC_1
  
 "BNDY_GASC_1" opened as OLD:READ-ONLY   
 File name "/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/PREP/bcon/scripts/bcon/BCON_36km_cb05_profile"
 File type BNDARY3 
 Execution ID "BCON_36km_profile.exe"
 Grid name "36km"
 Dimensions: 79 rows, 95 cols, 17 lays, 76 vbles, 1 cells thick
 NetCDF ID:   1114112  opened as READONLY            
 Time-independent data.
 Checking header data for file: BNDY_GASC_1
 Checking header data for file: INIT_AERO_1
  
 "BNDY_AERO_1" opened as OLD:READ-ONLY   
 File name "/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/PREP/bcon/scripts/bcon/BCON_36km_cb05_profile"
 File type BNDARY3 
 Execution ID "BCON_36km_profile.exe"
 Grid name "36km"
 Dimensions: 79 rows, 95 cols, 17 lays, 76 vbles, 1 cells thick
 NetCDF ID:   1179648  opened as READONLY            
 Time-independent data.
 Checking header data for file: BNDY_AERO_1
 Checking header data for file: INIT_NONR_1
  
 "BNDY_NONR_1" opened as OLD:READ-ONLY   
 File name "/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/PREP/bcon/scripts/bcon/BCON_36km_cb05_profile"
 File type BNDARY3 
 Execution ID "BCON_36km_profile.exe"
 Grid name "36km"
 Dimensions: 79 rows, 95 cols, 17 lays, 76 vbles, 1 cells thick
 NetCDF ID:   1245184  opened as READONLY            
 Time-independent data.
 Checking header data for file: BNDY_NONR_1
  
 "INIT_TRAC_1" opened as OLD:READ-ONLY   
 File name "/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/PREP/icon/scripts/icon/ICON_36km_cb05_profile"
 File type GRDDED3 
 Execution ID "ICON_36km_profile.exe"
 Grid name "36km"
 Dimensions: 79 rows, 95 cols, 17 lays, 76 vbles
 NetCDF ID:   1310720  opened as READONLY            
 Time-independent data.
 Checking header data for file: INIT_TRAC_1
  
 "BNDY_TRAC_1" opened as OLD:READ-ONLY   
 File name "/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/PREP/bcon/scripts/bcon/BCON_36km_cb05_profile"
 File type BNDARY3 
 Execution ID "BCON_36km_profile.exe"
 Grid name "36km"
 Dimensions: 79 rows, 95 cols, 17 lays, 76 vbles, 1 cells thick
 NetCDF ID:   1376256  opened as READONLY            
 Time-independent data.
 Checking header data for file: BNDY_TRAC_1
 DEPV_TRAC_1     :DEPV_TRAC_1
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 EMIS_TRAC_1     :EMIS_TRAC_1
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_DEPV_DIAG   :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_DEPV_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_PT3D_DIAG   :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_PT3D_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_RJ_1        :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_PHOTDIAG1_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_RJ_2        :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_PHOTDIAG2_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 INIT_MEDC_1     :INIT_MEDC_1
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 MEDIA_CONC      :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_MEDIA_CONC_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 REGIONS_1       :REGIONS_1
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 INIT_GASC_S     :INIT_GASC_S
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 INIT_AERO_S     :INIT_AERO_S
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 INIT_NONR_S     :INIT_NONR_S
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 BNDY_GASC_S     :BNDY_GASC_S
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 BNDY_AERO_S     :BNDY_AERO_S
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 BNDY_NONR_S     :BNDY_NONR_S
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_SENS_1      :CTM_SENS_1
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 A_SENS_1        :A_SENS_1
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_SWETDEP_1   :CTM_SWETDEP_1
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_SDRYDEP_1   :CTM_SDRYDEP_1
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
  
 "DUST_LU_1" opened as OLD:READ-ONLY   
 File name "/media/catalyst/Catalyst/mcip/36km/GRIDCRO2D_2016-01-02"
 File type GRDDED3 
 Execution ID "mcip"
 Grid name "GRIDOUT_36km_CRO"
 Dimensions: 79 rows, 95 cols, 1 lays, 28 vbles
 NetCDF ID:   1441792  opened as READONLY            
 Time-independent data.
 Checking header data for file: DUST_LU_1
  
 "DUST_LU_2" opened as OLD:READ-ONLY   
 File name "/media/catalyst/Catalyst/mcip/36km/GRIDCRO2D_2016-01-02"
 File type GRDDED3 
 Execution ID "mcip"
 Grid name "GRIDOUT_36km_CRO"
 Dimensions: 79 rows, 95 cols, 1 lays, 28 vbles
 NetCDF ID:   1507328  opened as READONLY            
 Time-independent data.
 Checking header data for file: DUST_LU_2
 CTM_DEPV_MOS    :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_DEPVFST_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_DRY_DEP_MOS :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_DDMOS_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_DRY_DEP_FST :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_DDFST_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_DEPV_FST    :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_DEPVMOS_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 E2C_FERT        :E2C_FERT
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 E2C_SOIL        :E2C_SOIL
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 BELD4_LU        :BELD4_LU
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_SD_TS       :CTM_SD_TS
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_VDIFF_DIAG  :CTM_VDIFF_DIAG
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_VSED_DIAG   :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_VSED_DIAG_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 BNDY_TRAC_S     :BNDY_TRAC_S
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 INIT_TRAC_S     :INIT_TRAC_S
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_MGEM_1      :CTM_MGEM_1
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_APMDIAG_1   :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_APMDIAG_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_AVIS_1      :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_APMVIS_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_LTNGDIAG_1  :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_LTNGHRLY_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 CTM_LTNGDIAG_2  :/home/catalyst/Desktop/Build_WRF/CMAQ-5.2/CCTM/scripts/cctm_36km/CCTM_LTNGCOL_36km_2016002.nc
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 
 
 >>--->> WARNING in subroutine FLCHECK on PE 001
 Inconsistent header data on input files
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)

 = = = = = = = = = = = = = =  End  FLCHECK = = = = = = = = = = = = = =

 Value for CTM_MAXSYNC not defined; returning default:  720
 Maximum Synchronization Time Step (sec)                                         
 Value for CTM_MINSYNC not defined; returning default:  60
 Minimum Synchronization Time Step (sec)                                         
 Value for CTM_ADV_CFL not defined; returning default :  0.75
 Maximum CFL number allowed                                                      
 Value for SIGMA_SYNC_TOP not defined; returning default :  0.7
 Minimum layer limit for which adv = sync                                        
 Value for ADV_HDIV_LIM not defined; returning default :  0.9
 Maximum horiz. div. limit for adv step adjustment                               
 Top layer thru which sync step determined: 12

 From ADVSTEP - date/time:  2016002/000000

 Computed synchronization step (HHMMSS): 001200
 Number of Synchronization steps:    5


 Layer   Advection   per Sync
       Step (HHMMSS)  Step
   17      000600       2
   16      000600       2
   15      000600       2
   14      000600       2
   13      001200       1
   12      001200       1
   11      001200       1
   10      001200       1
    9      001200       1
    8      001200       1
    7      001200       1
    6      001200       1
    5      001200       1
    4      001200       1
    3      001200       1
    2      001200       1
    1      001200       1
 Value for CTM_CKSUM:  N returning FALSE
 Value for CTM_ILDEPV:  Y returning TRUE
 Value for CTM_ABFLUX:  N returning FALSE
 Value for CTM_SFC_HONO not defined;returning default:   TRUE
 Flag for HONO interaction with surfaces                                         
 Environment variable not set ... Using default:                                                                                 0
 Value for CTM_MOSAIC:  N returning FALSE
 Value for CTM_HGBIDI:  N returning FALSE
 Value for CTM_DEPV_FILE:  N returning FALSE
 Flag for writing the DEPV diagnostic file                                       
 DEPV_INIT: writes GAS DEPV values to CTM_DEPV_FILE                                                                      
 DEPV_INIT: completed INIT_GAS_DV block 

 Value for KZMIN:  Y returning TRUE
 Value for CTM_GRAV_SETL:  N returning FALSE
 Using J-,K-mode aerosols gravitational settling                                 
 Value for VDIFF_DIAG_FILE:  N returning FALSE
 Writing the VDIFF diagnostic files                                              
 Value for CTM_BIOGEMIS:  N returning FALSE
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 GC units incorrect on EMIS_1           for species NO
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species HONO not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species ALD2 not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species ALDX not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species IOLE not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species FORM not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 GC units incorrect on EMIS_1           for species CO
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species PAR not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 GC units incorrect on EMIS_1           for species MEOH
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species OLE not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species ETH not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species TOL not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species XYLMN not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species NAPH not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species ISOP not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species TERP not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 GC units incorrect on EMIS_1           for species SO2
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 GC units incorrect on EMIS_1           for species SULF
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species ETOH not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species ETHA not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species CL2 not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species HCL not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species BENZENE not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 GC units incorrect on EMIS_1           for species SESQ
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species SOAALK not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species FORM_PRIMARY not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species ALD2_PRIMARY not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species BUTADIENE13 not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species ACROLEIN not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species ACROLEIN not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species TOLU not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species MXYL not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species OXYL not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species PXYL not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species APIN not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species BPIN not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species HGNRVA not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)
 
 >>--->> WARNING in subroutine OPEMIS on PE 001
 Emissions species HGIIGAS not found on EMIS_1
 M3WARN:  DTBUF 0:00:00   Jan. 2, 2016  (2016002:000000)


 *** ERROR ABORT in subroutine OPEMIS on PE 001          
 NR units incorrect on EMIS_1

PM3EXIT: DTBUF 0:00:00 Jan. 2, 2016
Date and time 0:00:00 Jan. 2, 2016 (2016002:000000)

CTM_LOG_001.36km_2016002.txt (51.3 KB)

Catalyst2682 · September 19, 2019, 9:33am

Dear @lizadams , @cjcoats , @tlspero, @hogrefe.christian, @CMAQ_user.

Below is the latest common error I got while running cctm:

*** ERROR ABORT in subroutine OPEMIS on PE 003
NR units incorrect on EMIS_1
PM3EXIT: DTBUF 0:00:00 Jan. 2, 2016
Date and time 0:00:00 Jan. 2, 2016 (2016002:000000)

How do I resolve it? Kindly help me out.

Thanks

Catalyst

hogrefe.christian · September 20, 2019, 5:21pm

Hello Catalyst2682,

thank you for posting the error from the processor log files.

Please check the units for all the species in your emissions file that are specified as emission surrogates in your NR namelist file. The units for these NR species should be ‘moles/s’ (alternatively, one can use ‘mole/s’, ‘mol/s’, ‘moles/sec’, ‘mole/sec’, or ‘mol/sec’).

For a typical CMAQv5.2 configuration, the only emitted species specified in the NR namelist (e.g. NR_cb6r3_ae6_aq.nml if you are running cb6r3_ae6_aq) is NH3, so I would check the units for NH3 in your emissions file first.

Christian

esm · September 23, 2019, 3:57pm

Hi
I ran CMAQ-5.2, but I have an error that I don’t know how to solve it. Could you guide me, please?
The error is:

 "GRID_CRO_2D" opened as OLD:READ-ONLY   
 File name "/home/ensiyeh/WRF-CMAQ/CMAQ-OFFLINE/CMAQ-5.2/data/cctm/met/mcip/SE52BENCH/GRIDCRO2D_171213.nc"
 File type GRDDED3 
 Execution ID "mcip"
 Grid name "GRIDOUT_SE52BENC"
 Dimensions: 64 rows, 73 cols, 1 lays, 28 vbles
 NetCDF ID:    131072  opened as READONLY            
 Time-independent data.
 INIT_GASC_1     :/home/ensiyeh/WRF-CMAQ/CMAQ-OFFLINE/CMAQ-5.2/data/output_CCTM_v52_intel_SE52BENCH/CCTM_CGRID_v52_intel_SE52BENCH_20171213
 
 >>--->> WARNING in subroutine OPEN3
 File not available.
 


 *** ERROR ABORT in subroutine LOAD_CGRID on PE 004      
 Could not open INIT_GASC_1      file

PM3EXIT: DTBUF 0:00:00 Dec. 14, 2017
Date and time 0:00:00 Dec. 14, 2017 (2017348:000000)

And I have another question. Why I have these negative numbers in my FLOOR file?
I attached all log files and floor file. Could you guide me, please?
Thank you
Uploading: CTM_LOG_002.txt… Uploading: CTM_LOG_001.txt… CTM_LOG_005.v52_intel_SE52BENCH_20171214.txt (3.2 KB) CTM_LOG_004.v52_intel_SE52BENCH_20171214.txt (4.0 KB) CTM_LOG_003.v52_intel_SE52BENCH_20171214.txt (3.2 KB) CTM_LOG_002.v52_intel_SE52BENCH_20171214.txt (4.0 KB) CTM_LOG_001.v52_intel_SE52BENCH_20171214.txt (3.2 KB) FLOOR_005.v52_intel_SE52BENCH_20171213.txt (18.1 KB) Uploading: run.log.txt… Uploading: CTM_LOG_004.txt… Uploading: CTM_LOG_005.txt… Uploading: CTM_LOG_003.txt…

munshimdrasel · April 15, 2021, 4:34pm

Hy,

I’m currently using CMAQ v5.2. I’m trying to run CCTM without running DDM in my script, however, it still looks for DDM environmental variable. I’ve commented out each of the DDM variable but it didn’t work. How can I run the script completely ignoring DDM in my script?

Please see the attached log file error and script that I ran.
CTM_LOG_001.v52_gcc_AQF5X_20200601.txt (4.4 KB) run_cctm_rasel.csh_sbatch.txt (29.4 KB)

hogrefe.christian · April 15, 2021, 5:24pm

Enabling DDM is a compile time option, not a runtime option. If you compiled your executable with DDM, running it without the necessary input file specified in the run script will lead to a crash. If you want to perform a simulation without DDM, you need to build a new executable without DDM first.

munshimdrasel · February 17, 2022, 12:32pm

Hy,
I’m running CCTM for first 15 days of January 2020 emission data. The job runs for day one fine. When It goes to day two, the model crashes. Can you please take a look into my script and slurm job output file and let me know what’s the issue? (Log file became too large to attach)
jCMAQ-5.2-NODE014-72162.out.txt (2.1 MB)
run_cctm_rasel_all.csh_sbatch.txt (27.9 KB)

 *** ERROR ABORT in subroutine LOAD_CGRID on PE 000
 Could not open INIT_GASC_1      file
 Date and time 12:00:00  Jan. 2, 2020   (2020002:120000)

application called MPI_Abort(MPI_COMM_WORLD, 538976288) - process 0

hogrefe.christian · February 17, 2022, 1:17pm

Looking at your log file, the simulation for your first 24 hour simulation (started at 2020001:120000) did not complete successfully. It failed at 2020002:000000 with these errors in the main log:

application called MPI_Abort(MPI_COMM_WORLD, 538976288) - process 15
application called MPI_Abort(MPI_COMM_WORLD, 538976288) - process 16
application called MPI_Abort(MPI_COMM_WORLD, 538976288) - process 14
srun: Job step aborted: Waiting up to 92 seconds for job step to finish.
slurmstepd-NODE014: error: *** STEP 72162.0 ON NODE014 CANCELLED AT 2022-02-16T18:49:35 ***
Fatal error in PMPI_Recv: Unknown error class, error stack:
PMPI_Recv(171)…: MPI_Recv(buf=0x18147cb0, count=1925, MPI_REAL, src=63, tag=1, comm=0x84000004, status=0x7ffde7ad0e90) failed

Because of that crash of the first 24 hour simulation, the CGRID file needed for initializing the second 24 hour simulation at 2020002:120000 wasn’t written and caused that second simulation to crash right away.

To investigate what caused your first 24 hour simulation to crash at 2020002:000000, you want to examine the CTM_LOG files for that simulation.

Topic		Replies	Views
ERROR ABORT in subroutine FLCHECK on PE 005 Inconsistent header data on input files CMAQ	1	563	March 11, 2020
Error in running CMAQv5.0.2 CMAQ	8	2052	May 10, 2022
Error in runing CCTM-ISAM Run Time Errors and Issues	1	759	April 6, 2020
NetCDF error in CMAQ5.3.1 Run Time Errors and Issues	8	3157	August 11, 2020
Error in CMAQv5.0.2 benchmark run Benchmark Issues and Errors	0	389	October 9, 2019

Error while running CCTM

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them.

Set name and path of resulting shapefile

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them.

Related Topics

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.