Segmentation fault running CCTM 5.3

CMAQ Run Time Errors and Issues
1

Hi all,

I am encountering a segmentation fault when attempting to run CMAQv5.3:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
CCTM_v53.exe 0000000000A2B201 Unknown Unknown Unknown
CCTM_v53.exe 0000000000A2933B Unknown Unknown Unknown
CCTM_v53.exe 00000000009D4494 Unknown Unknown Unknown
CCTM_v53.exe 00000000009D42A6 Unknown Unknown Unknown
CCTM_v53.exe 000000000095E6F9 Unknown Unknown Unknown
CCTM_v53.exe 0000000000965316 Unknown Unknown Unknown
libpthread-2.17.s 00002BA4EFB8F6D0 Unknown Unknown Unknown
CCTM_v53.exe 00000000005E7D45 Unknown Unknown Unknown
CCTM_v53.exe 00000000005E2809 Unknown Unknown Unknown
CCTM_v53.exe 000000000040931E Unknown Unknown Unknown
libc-2.17.so 00002BA4F0463445 __libc_start_main Unknown Unknown
CCTM_v53.exe 0000000000409229 Unknown Unknown Unknown

In my system output log for the run, this occurs immediately after the “Time Integration” label is displayed but before any additional output is created.

In the CTM_LOG_000 file, the simulation stops right after the output for advection time step initialization.

I have tried “unlimit stacksize” without success. I am using intel 2017 and the mpi libraries appear to be linking/referencing correctly. Could this be a result of missing input files? Could you provide any additional suggestions?

Please find the system logfile and the CTM_LOG_000 file attached.

Thank you,
James
CTM_LOG_000.v53_intel2017_d04_bogota_20140101.txt (41.4 KB)

out.cmqbog.777999.txt (118.2 KB)

2

You could probably get more information as to where the code is seg-faulting if you were to add traceback to your compiler options: -traceback for Intel compilers such as you are using, -fbacktrace for gfortran.

3

Thank you.

Recompiling with debug flags results in a slightly different error:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
CCTM_v53.exe       0000000001822D01  Unknown               Unknown  Unknown
CCTM_v53.exe       0000000001820E3B  Unknown               Unknown  Unknown
CCTM_v53.exe       00000000017CBF74  Unknown               Unknown  Unknown
CCTM_v53.exe       00000000017CBD86  Unknown               Unknown  Unknown
CCTM_v53.exe       00000000017561D9  Unknown               Unknown  Unknown
CCTM_v53.exe       000000000175CDF6  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B83BE6D76D0  Unknown               Unknown  Unknown
CCTM_v53.exe       0000000000D59D78  cmaq_driver_              662  driver.F
CCTM_v53.exe       0000000000D55350  MAIN__                     96  cmaq_main.F
CCTM_v53.exe       000000000040929E  Unknown               Unknown  Unknown
libc-2.17.so       00002B83BEFAB445  __libc_start_main     Unknown  Unknown
CCTM_v53.exe       00000000004091A9  Unknown               Unknown  Unknown

Lines 661-662 in driver.F:

 IF ( L_ACONC_WVEL )
&   AVG_WVEL( :,:,: ) = WVEL( :,:,ACONC_BLEV:ACONC_ELEV )

Could problems indexing WVEL cause this sort of error?

Also, the CTM_LOG_000 file gives an inconsistent value warning for vertical grids in the initial and boundary files:

 Checking header data for file: INIT_CONC_1
     Inconsistent values for VGTYP: 7 versus -9999
 Checking header data for file: BNDY_CONC_1
     Inconsistent values for VGTYP: 7 versus -9999

Is there somewhere VGTYP can be defined?

Thank you for any feedback you can give!

4

Could problems indexing WVEL cause this sort of error?

Yes. Especially if the 3rd dimension of AVG_WVEL is less than ACONC_ELEV-ACONC_BLEV+1

Is there somewhere VGTYP can be defined?

Vertical grid types are defined n the I/O API INCLUDE-file PARMS3.EXT:

        !!......  Vertical-coordinate types:

        INTEGER, PARAMETER :: GISLAY3 =  -1   ! "vert" coord type -1:  (arc) GIS layers
        INTEGER, PARAMETER :: VGSGPH3 =   1   !  vert  coord type  1:  hydrostatic sigma-P
        INTEGER, PARAMETER :: VGSGPN3 =   2   !  vert  coord type  2:  non-h sigma-P
        INTEGER, PARAMETER :: VGSIGZ3 =   3   !  vert  coord type  3:  sigma-Z
        INTEGER, PARAMETER :: VGPRES3 =   4   !  vert  coord type  4:  pressure (mb)
        INTEGER, PARAMETER :: VGZVAL3 =   5   !  vert  coord type  5:  Z (m) (above sea lvl)
        INTEGER, PARAMETER :: VGHVAL3 =   6   !  vert  coord type  6:  H (m) (above ground)
        INTEGER, PARAMETER :: VGWRFEM =   7   !  vert  coord type  7:  WRF mass-core sigma
        INTEGER, PARAMETER :: VGWRFNM =   8   !  vert  coord type  8:  WRF NMM
5

Thank you. After adding some write statements, it looks like the problem may be that WVEL has shape (0,0,0).

Write statements:

WRITE(logdev,* ) "ACONC_BLEV, ACONC_ELEV = ", ACONC_BLEV,ACONC_ELEV
WRITE(logdev,* ) "Shape of WVEL = ", SHAPE(WVEL)

Output:

ACONC_BLEV, ACONC_ELEV =            1           1
Shape of WVEL =            0           0           0

Does anyone know what could lead to this? I am using WRF4.1.3 for the meteorology and processed with MCIPv5.

Thanks,
James

6

Hi James,

Did you set CTM_WVEL environment variable to N?
There is a known issue that may be related to your findings.

7

Hi @lizadams, yes, I did and just fixed it. It’s running now.

This was my exact problem. Thank you!

James

1 Like