Segmentation fault occurred when combining multi-layer aconc files

CMAQ-POST
1

Hi all,

I am using POST/combine tool to combine the aconc files, which has 25 layers as what I need. Everything was fine when combine just one layer, defined by the SpecDef:

!#start   YYYYJJJ  010000
!#end     YYYYJJJ  000000
#layer    1

But segmentation fault occurred at the first time step when I tried to combine multi-layer aconc files: (#layer 1 was comment out)

!#start   YYYYJJJ  010000
!#end     YYYYJJJ  000000
!#layer    1

error msg on the log file:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
combine_v533_sc.e  0000000000634B11  Unknown               Unknown  Unknown
combine_v533_sc.e  0000000000632C4B  Unknown               Unknown  Unknown
combine_v533_sc.e  00000000005DBEC4  Unknown               Unknown  Unknown
combine_v533_sc.e  00000000005DBCD6  Unknown               Unknown  Unknown
combine_v533_sc.e  000000000057C849  Unknown               Unknown  Unknown
combine_v533_sc.e  0000000000583096  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B24FF2D45E0  Unknown               Unknown  Unknown
combine_v533_sc.e  0000000000559AFE  Unknown               Unknown  Unknown
combine_v533_sc.e  00000000005648D1  Unknown               Unknown  Unknown
combine_v533_sc.e  0000000000561838  Unknown               Unknown  Unknown
combine_v533_sc.e  0000000000560295  Unknown               Unknown  Unknown
combine_v533_sc.e  00000000005504F1  Unknown               Unknown  Unknown
combine_v533_sc.e  000000000054E860  Unknown               Unknown  Unknown
combine_v533_sc.e  0000000000546BD0  Unknown               Unknown  Unknown
combine_v533_sc.e  00000000004405E4  rdvars_                   346  rdvars.F90
combine_v533_sc.e  0000000000506476  rdgrdded_                 107  rdgrdded.f
combine_v533_sc.e  0000000000433B1E  lambert_                  497  lambert.f
combine_v533_sc.e  000000000040738B  Unknown               Unknown  Unknown
combine_v533_sc.e  000000000041AE13  Unknown               Unknown  Unknown
combine_v533_sc.e  000000000041260E  Unknown               Unknown  Unknown
combine_v533_sc.e  000000000040FE86  Unknown               Unknown  Unknown
combine_v533_sc.e  000000000040AD8B  Unknown               Unknown  Unknown
combine_v533_sc.e  000000000040A778  Unknown               Unknown  Unknown
combine_v533_sc.e  0000000000422EAB  Unknown               Unknown  Unknown
combine_v533_sc.e  00000000004037DE  Unknown               Unknown  Unknown
libc-2.17.so       00002B24FF804C05  __libc_start_main     Unknown  Unknown
combine_v533_sc.e  00000000004036E9  Unknown               Unknown  Unknown

Can anyone give me some advice?

Thanks a lot!

Tristan

2

It would be useful to have a “good” complete trace-back: re-compiling everything with flags compatible with those in ioapi/Makeinclude.Linux2_x86_64gfortdbg would do this.

Still. The fact that the seg-fault is coming from I/O API SUBROUTINE RDVARS says that the routine is trying to read more data than the ALLOCATEd size of its input-buffer – most probably, the input-buffer does not have as many layer allocated as the RDVARS-call is expecting. Possibly a script-error, or else a programmer-error.

3

Thanks for your quick reply!

I am now using IOAPI3.2_x86_64ifort_medium, is “re-compiling everything” means re-compiling IOAPI (change to _x86_64ifort_mediumdbg) and everything based on IOAPI?

Thanks!

4

Yes, sort of: new I/O API library, and new COMBINE object and executable files. Unforetunately, the CMAQ build-system does not lend itself well to per-program builds, but…

Needing medium binary-type says that you’re expecting huge (> 2GB) arrays – which happens for large CMAQ DDM or ISAM cases; is that the case for your COMBINE runs?

5

Please post your combine run script, full SpecDef file, full log file from the run leading to the crash, and ncdump -h of any file(s) specified in the run script and referenced in the SpecDef file.

6

The most likely source of the error is an inconsistency in the number of layers across the input files.

If you instruct combine to operate on all layers rather than just layer 1 by commenting out the #LAYER 1 line, the number of layers must match across all input files.

7

I do use ISAM, thus the medium type.
But this combine run with error is just the base model outputs.

8

Yes!
I set the cctm run scripts to only output 25 out of 34 layers in aconc, didn’t need the higher layers (trying to save some space). But all the MCIP files are 34 layers. I know where the problem is.
Thanks!