Running SMOKE error _Program received signal SIGSEGV

Dear all,

I run smoke-v4.7-gfort. There are 4 nests (1km, 3km,9km,27km) in my model. As for 1km and 3km parts, I successfully run the programs in the model. From my 9km and 27km running logfile, there is errors.
* Released under the GNU LGPL License, version 2.1. See URL

*          https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html

*      for conditions of use.

*      ioapi-3.2: $Id: init3.F90 185 2020-08-28 16:49:45Z coats $
*      netCDF version 4.7.4 of Feb 25 2021 01:11:06 $


* Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

* Backtrace for this error:
* #0  0x7f7a981f5131 in ???
* #1  0x7f7a981f42d3 in ???
* #2  0x7f7a978a978f in ???
* #3  0x7f7a97915852 in ???
* #4  0x7f7a9840ee33 in ???
* #5  0x404618 in ???
* #6  0x4489b7 in ???
* #7  0x40318c in ???
* #8  0x7f7a978956a2 in ???
* #9  0x4031ed in ???
* #10  0xffffffffffffffff in ???
* Segmentation fault (core dumped)

Best regards,
Chang

Which SMOKE program? …and how big are your grids and your inventories?

It might be interesting to run the Linux2_x86_64gfort_mediumdbg version, to see:

  • the traceback information if if fails; and
  • that its inventory or grid is so large that it requires medium memory model if it succeeds (in which case, build the Linux2_x86_64gfort_medium version, and run with it.

Thanks for your reply. Here is my anwser,

  1. SMKREPORT program. In my 27km domain, cols 138 × cols 182. In my model, all ge_dat and inventory files are about 250MB.
  2. Where could I get the Linux2_x86_64gfort_medium version?

Best regards,
Chang

You build it – it’s not hard:

  • Check out SMOKE from GitHub.
  • setenv BIN Linux2_x86_64gfort_medium [or Linux2_x86_64gfort_mediumdbg]
  • Make the corresponding I/O API library
  • cd SMOKE
  • mkdir $BIN
  • cd src
  • Customize Makeinclude: BASEDIR is the full SMOKE/src path; IOBASE is the path for the root of the I/O API installation.
  • make

Dear,
I recompiled netcdf, IOAPI and smoke. In SMOKE, I set BIN Linux2_x86_64gfort_medium and recompiled. And I also reset the environment variables of IOAPI and netcdf.
But When I run Smoke, In the 27km and 9km logfile recording processing the area inventory, I still meet the error.

OK, that says it’s not a memory-size related error.

Running a debug-build (as I suggested to begin with) would give a traceback that says at which line the SEGFAULT is occurring.

@cicy996 Which program is giving you a segmentation fault? Can you first identify the program first and then attached the log file from that program for me to review please?

I have recompiled the Linux2_x86_64gfort_medium, and I run the SMOKE-medium model. Is this the debug-build? Which directory could I find the traceback?

Thanks for your reply.
It occured that I ran the SMK_area.csh to process the D1 and D2 area emission. And it was running SMKREPORT program and went wrong. Here is the log file,

/disk/xxx/cmaq/smoke/cmaq/temp/PRD/grd_Alpine_test/CMAQ/201501.20160317
/disk/xxx/cmaq/smoke/cmaq/temp/PRD/grd_Alpine_test/CMAQ/201501.20160317
/bin/rm: No match.
/bin/rm: No match.
Running part 1...
                                                                 
 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).                                 
 Copyright (C) 1992-2002 MCNC,                                   
 (C) 1992-2018 Carlie J. Coats, Jr.,                             
 (C) 2003-2012 Baron Advanced Meteorological Systems, LLC, and   
 (C) 2014-2020 UNC Institute for the Environment.                
 Released under the GNU LGPL  License, version 2.1.  See URL     
                                                                 
     https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html     
                                                                 
 for conditions of use.                                          
                                                                 
 ioapi-3.2: $Id: init3.F90 185 2020-08-28 16:49:45Z coats $
 netCDF version 4.7.4 of Mar  9 2021 15:22:14 $
  
 SMOKE ---------------
 Copyright (c)2004 Environmental Modeling for Policy Development
 All rights reserved

 Program SMKINVEN, Version SMOKEv4.7_Oct2019
 Online documentation 
     http://www.cep.unc.edu/empd/products/smoke                 
  
 Program SMKINVEN to take ASCII area or point source files
 in IDA, EMS-95, or SMOKE list format, or mobile files
 in IDA format, and produce the I/O API and ASCII SMOKE
 inventory files and list of unique SCCs in the inventory.
  
  
 You will need to enter the logical names for the input and
 output files (and to have set them prior to program start,
 using "setenv <logicalname> <pathname>").
  
 You may use END_OF-FILE (control-D) to quit the program
 during logical-name entry. Default responses are given in
 brackets [LIKE THIS] and can be accepted by hitting the
 <RETURN> key.

0.309u 0.101s 0:02.57 15.5% 0+0k 0+864io 0pf+0w

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).                                 
 Copyright (C) 1992-2002 MCNC,                                   
 (C) 1992-2018 Carlie J. Coats, Jr.,                             
 (C) 2003-2012 Baron Advanced Meteorological Systems, LLC, and   
 (C) 2014-2020 UNC Institute for the Environment.                
 Released under the GNU LGPL  License, version 2.1.  See URL     
                                                                 
     https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html     
                                                                 
 for conditions of use.                                          
                                                                 
 ioapi-3.2: $Id: init3.F90 185 2020-08-28 16:49:45Z coats $
 netCDF version 4.7.4 of Mar  9 2021 15:22:14 $
  
 SMOKE ---------------
 Copyright (c)2004 Environmental Modeling for Policy Development
 All rights reserved

 Program SPCMAT, Version SMOKEv4.7_Oct2019
 Online documentation 
     http://www.cep.unc.edu/empd/products/smoke                 
  
 Program SPCMAT to take a SMOKE area, mobile, or point source
 inventory file, a speciation profiles file, a speciation
 cross-reference file, an optional pollutant-to-pollutant,
 conversion file, and produce mass-based and/or mole-based
 SMOKE speciation matrices for all inventory pollutants
 using run-time defined combinations of pollutants and model
 species. The output species are defined at run time by the
 speciation profiles file, permitting support of any chemical
 mechanism.
  
  
 You will need to enter the logical names for the input and
 output files (and to have set them prior to program start,
 using "setenv <logicalname> <pathname>").
  
 You may use END_OF-FILE (control-D) to quit the program
 during logical-name entry. Default responses are given in
 brackets [LIKE THIS] and can be accepted by hitting the
 <RETURN> key.

0.772u 0.020s 0:00.84 94.0% 0+0k 0+1696io 0pf+0w

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).                                 
 Copyright (C) 1992-2002 MCNC,                                   
 (C) 1992-2018 Carlie J. Coats, Jr.,                             
 (C) 2003-2012 Baron Advanced Meteorological Systems, LLC, and   
 (C) 2014-2020 UNC Institute for the Environment.                
 Released under the GNU LGPL  License, version 2.1.  See URL     
                                                                 
     https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html     
                                                                 
 for conditions of use.                                          
                                                                 
 ioapi-3.2: $Id: init3.F90 185 2020-08-28 16:49:45Z coats $
 netCDF version 4.7.4 of Mar  9 2021 15:22:14 $
  
 SMOKE ---------------
 Copyright (c)2004 Environmental Modeling for Policy Development
 All rights reserved

 Program GRDMAT, Version SMOKEv4.7_Oct2019
 Online documentation 
     http://www.cep.unc.edu/empd/products/smoke                 
  
 Program GRDMAT to take a SMOKE area, mobile, or point source
 inventory file, gridding surrogates,  surrogate cross-
 reference, and an optional link definitions file, and produce
 a SMOKE gridding matrix for a grid defined at run time. For
 mobile sources, an "ungridding" matrix is also created to
 allow the use of gridded temperature data in assigning
 factors to mobile sources.
  
  
 You will need to enter the logical names for the input and
 output files (and to have set them prior to program start,
 using "setenv <logicalname> <pathname>").
  
 You may use END_OF-FILE (control-D) to quit the program
 during logical-name entry. Default responses are given in
 brackets [LIKE THIS] and can be accepted by hitting the
 <RETURN> key.

0.554u 0.024s 0:00.44 129.5% 0+0k 0+792io 0pf+0w
/disk/r004/cliucv/cmaq/smoke/cmaq/temp/PRD/grd_Alpine_test/CMAQ/201501.20160317
/disk/r004/cliucv/cmaq/smoke/cmaq/temp/PRD/grd_Alpine_test/CMAQ/201501.20160317
Running part 2, for 20150102 …
Running part 4, for 20150102…

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).                                 
 Copyright (C) 1992-2002 MCNC,                                   
 (C) 1992-2018 Carlie J. Coats, Jr.,                             
 (C) 2003-2012 Baron Advanced Meteorological Systems, LLC, and   
 (C) 2014-2020 UNC Institute for the Environment.                
 Released under the GNU LGPL  License, version 2.1.  See URL     
                                                                 
     https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html     
                                                                 
 for conditions of use.                                          
                                                                 
 ioapi-3.2: $Id: init3.F90 185 2020-08-28 16:49:45Z coats $
 netCDF version 4.7.4 of Mar  9 2021 15:22:14 $
  
 SMOKE ---------------
 Copyright (c)2004 Environmental Modeling for Policy Development
 All rights reserved

 Program TEMPORAL, Version SMOKEv4.7_Oct2019
 Online documentation 
     http://www.cep.unc.edu/empd/products/smoke                 
  
 Program TEMPORAL to take a SMOKE area, mobile, or point
 source inventory file, a temporal profiles file, a temporal
 cross-reference file, an optional SMOKE day-specific file, 
 and an optional SMOKE hour-specific point source file, and
 produce hourly low-level and optionally hourly elevated point
 source emissions for the requested episode.
  
  
 You will need to enter the logical names for the input and
 output files (and to have set them prior to program start,
 using "setenv <logicalname> <pathname>").
  
 You may use END_OF-FILE (control-D) to quit the program
 during logical-name entry. Default responses are given in
 brackets [LIKE THIS] and can be accepted by hitting the
 <RETURN> key.

0.075u 0.015s 0:00.09 88.8% 0+0k 0+2960io 0pf+0w

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).                                 
 Copyright (C) 1992-2002 MCNC,                                   
 (C) 1992-2018 Carlie J. Coats, Jr.,                             
 (C) 2003-2012 Baron Advanced Meteorological Systems, LLC, and   
 (C) 2014-2020 UNC Institute for the Environment.                
 Released under the GNU LGPL  License, version 2.1.  See URL     
                                                                 
     https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html     
                                                                 
 for conditions of use.                                          
                                                                 
 ioapi-3.2: $Id: init3.F90 185 2020-08-28 16:49:45Z coats $
 netCDF version 4.7.4 of Mar  9 2021 15:22:14 $
  
 SMOKE ---------------
 Copyright (c)2004 Environmental Modeling for Policy Development
 All rights reserved

 Program SMKMERGE, Version SMOKEv4.7_Oct2019
 Online documentation 
     http://www.cep.unc.edu/empd/products/smoke                 
  
 Program SMKMERGE to merge the inventory or hourly emission
 files with gridding matrices and with optionally any
 combination of speciation matrices, multiplicative control
 matrices, additive control matrices, or reactivity control
 matrices. The program can operate on one to four source 
 categories (area, biogenic, mobile, or point sources), or any
 combination of these.  Gridded and/or state reports and/or
 county reports can be written from this program. If a layer-
 fractions file is input, then the total emissions output file
 is three-dimensional.
  
  
 You will need to enter the logical names for the input and
 output files (and to have set them prior to program start,
 using "setenv <logicalname> <pathname>").
  
 You may use END_OF-FILE (control-D) to quit the program
 during logical-name entry. Default responses are given in
 brackets [LIKE THIS] and can be accepted by hitting the
 <RETURN> key.
  
    at time 12:00:00
    at time 13:00:00
    at time 14:00:00
    at time 15:00:00
    at time 16:00:00
    at time 17:00:00
    at time 18:00:00
    at time 19:00:00
    at time 20:00:00
    at time 21:00:00
    at time 22:00:00
    at time 23:00:00
    at time 0:00:00 
    at time 1:00:00 
    at time 2:00:00 
    at time 3:00:00 
    at time 4:00:00 
    at time 5:00:00 
    at time 6:00:00 
    at time 7:00:00 
    at time 8:00:00 
    at time 9:00:00 
    at time 10:00:00
    at time 11:00:00
    at time 12:00:00

0.456u 0.090s 0:01.85 29.1% 0+0k 0+167024io 0pf+0w

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).                                 
 Copyright (C) 1992-2002 MCNC,                                   
 (C) 1992-2018 Carlie J. Coats, Jr.,                             
 (C) 2003-2012 Baron Advanced Meteorological Systems, LLC, and   
 (C) 2014-2020 UNC Institute for the Environment.                
 Released under the GNU LGPL  License, version 2.1.  See URL     
                                                                 
     https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html     
                                                                 
 for conditions of use.                                          
                                                                 
 ioapi-3.2: $Id: init3.F90 185 2020-08-28 16:49:45Z coats $
 netCDF version 4.7.4 of Mar  9 2021 15:22:14 $
  

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7fa24bbcd131 in ???
#1  0x7fa24bbcc2d3 in ???
#2  0x7fa24b28178f in ???
#3  0x7fa24b2ed852 in ???
#4  0x7fa24bde6e33 in ???
#5  0x404648 in ???
#6  0x448a07 in ???
#7  0x4031bc in ???
#8  0x7fa24b26d6a2 in ???
#9  0x40321d in ???
#10  0xffffffffffffffff in ???
Segmentation fault (core dumped)
0.099u 0.065s 0:06.29 2.3%	0+0k 0+912io 0pf+0w

For traceback, use one of the BIN choices that ends in dbg – e.g., BIN=Linux2_x86_64gfort_mediumdbg.

Also, what does the limit command say? …and are you running with limit stacksize unlimited and limit memoryuse unlimited ?

@cjcoats @cicy996 I am not so sure whether this segmentation fault is coming from SMOKE or not. Based on your screen output, Smkmerge ran successfully and generated the final output files! The segmentation fault is occurred after that. It could be one of IOAPI tools used in the run scripts? I do not see any SMOKE program name mentioned in that screen output.

Yes, I got the final emission file. But I compared the log file with my classmate who ran the same model, and he just ran in different Linux machine, in his log file there was Smkreport program and at the end of log file a successfully appeared. So how to tell the error from smoke or ioapi?

All of the I/O API M3Tools programs begin with a “flash screen” of the form below, giving (among other things) the program-name, its usage-summary, and copyright and license notices (and the SMOKE modeling programs do similarly):

>      This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
>      I/O Applications Programming Interface...                                        
>                                                                      
>      $Id:: init3.F 1 2014-03-14 20:22:54Z coats    $
>      Version with PARMS3.EXT/PARAMETER::MXVARS3= 2048
>      netCDF version "3.6.2" of Dec 13 2013 03:51:05 $      
> 
> 
>      Program M3STAT to compute...     
>      
>      Program version: 
>      $Id:: m3stat.f 44 2014-09-12 18:03:16Z coats $

In the absence of such a “splash screen”, I don’t know what is causing this seg-fault.

Again, what does limit say?

By the way, in order to run with different choices of binary-type BIN, you must edit the ASSIGNS file accordingly. Did you do that?

When I typed the ulimit -n, it returned 1024. Do you mean that?
Everytime I compiled the SMOKE, I edit the BIN in the ASSIGNS file.

Your ulimit -n result says that you are allowed 1024 open files.
I want to know about stacksize and memory use, particularly; try ulimit -a to get all the limits.

Here is the (ulimit -a) returned information,

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 255330
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 255330
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

8 MB (8192 KB) stack size may be much too small.; set it to unlimited and try again.

Sorry I am not familiar with linux. :joy: Would you mind tell me how to type the code to change the pipe size?

see man ulimit:
ulimit -s unlimited
sets stack-size to unlimited.

I set the stack size to unlimited, and it still shows the Segmentation fault - invalid memory reference error. :sleepy:

>    ioapi-3.2: $Id: init3.F90 185 2020-08-28 16:49:45Z coats $
>      netCDF version 4.7.4 of Mar  9 2021 15:22:14 $
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x148708f2c131 in ???
#1  0x148708f2b2d3 in ???
#2  0x1487085e078f in ???
#3  0x14870864c852 in ???
#4  0x148709145e33 in ???
#5  0x404648 in ???
#6  0x448a07 in ???
#7  0x4031bc in ???
#8  0x1487085cc6a2 in ???
#9  0x40321d in ???
#10  0xffffffffffffffff in ???
Segmentation fault (core dumped)
0.108u 0.059s 0:08.93 1.6%      0+0k 2400+920io 6pf+0w