SMOKE issue "Program received signal SIGSEGV: Segmentation fault - invalid memory reference"

Hi,
I have installed SMOKE 4.8.1 using gfortran compiler. And I have successfully run test cases.

Now, I am trying to run a 12km ptegu onetime script 2016v2 platform using the same setup as https://gaftp.epa.gov/Air/emismod/2016/v2/.

But I am getting this error:
Program received signal SIGSEGV: Segmentation fault - invalid memory reference
Backtrace for this error:

Note: 2016v1 SMOKE run is successfully done using same gfortran compiler with SMOKE4.7

Can you provide a log / standard output file? Also which program did it crash on?

Do you have the stacksize unlimited setting on?

Alison

Looks like SMKREPORT program is crashing (please see the terminal output below). Yes, I have stacksize unlimited setting on.

[Iqbal@ADEQPlanning point]$ ./trial_Annual_ptegu_wintershld_onetime_12US1_2032fj_16j.csh
testing SRGPRO_PATH set by SRGPRO input: /home/Michael/RAID1/SMOKE/2016v2/ge_dat/gridding/surrogates/CONUS12_2017NEI_04mar2021/
Linux2_x86_64ifort
/home/Iqbal/Documents/SMOKE481/subsys/ioapi/Linux2_x86_64gfort/juldate: Permission denied.
/home/Iqbal/Documents/SMOKE481/subsys/ioapi/Linux2_x86_64gfort/juldate: Permission denied.
NOTE: Default MONTH setting to ‘jan’
cut: /home/Michael/RAID1/SMOKE/output2016v2/2032fj_16j/scripts/sectorlist_2032fj: No such file or directory
/bin/ls: No match.
/bin/ls: No match.
/bin/ls: No match.
/bin/ls: No match.
Creating dataset /home/Michael/RAID1/SMOKE/output2016v2/ge_dat/speciation/2032fj_16j/gspro_ptegu_wintershld_cmaq_cb6ae7_2032fj_16j_16j.txt
using script combine_data.csh
grep: 15923: No such file or directory
Processing environment variables GSPROTMP_A
Including file: /home/Michael/RAID1/SMOKE/2016v2/ge_dat/speciation/gspro_ptegu_wintershld_cmaq_cb6ae7_2032fj_16j_01may2019.txt
Creating dataset /home/Michael/RAID1/SMOKE/output2016v2/ge_dat/speciation/2032fj_16j/gsref_ptegu_wintershld_cmaq_cb6ae7_2032fj_16j_16j.txt
using script combine_data.csh
grep: 16068: No such file or directory
Processing environment variables GSREFTMP_A
Including file: /home/Michael/RAID1/SMOKE/2016v2/ge_dat/speciation/gsref_ptegu_wintershld_cmaq_cb6ae7_2032fj_16j_nf.txt
SCRIPT NOTE: Scanning GSREF for duplicate records
SCRIPT NOTE: No duplicates found in /home/Michael/RAID1/SMOKE/output2016v2/ge_dat/speciation/2032fj_16j/gsref_ptegu_wintershld_cmaq_cb6ae7_2032fj_16j_16j.txt
testing SRGPRO_PATH set by SRGPRO input: /home/Michael/RAID1/SMOKE/2016v2/ge_dat/gridding/surrogates/CONUS12_2017NEI_04mar2021/
Linux2_x86_64ifort
/home/Iqbal/Documents/SMOKE481/subsys/ioapi/Linux2_x86_64gfort/juldate: Permission denied.
/home/Iqbal/Documents/SMOKE481/subsys/ioapi/Linux2_x86_64gfort/juldate: Permission denied.
cut: /home/Michael/RAID1/SMOKE/output2016v2/2032fj_16j/scripts/sectorlist_2032fj: No such file or directory
Creating dataset /home/Michael/RAID1/SMOKE/output2016v2/2032fj_16j/inputs/ptegu_wintershld/ptinv_ptegu_wintershld_2032fj_16j.lst
using script combine_data.csh
Processing environment variables EMISINV_A EMISINV_B
Running part 1…

SCRIPT NOTE: Automatically deleting log file.
/home/Michael/RAID1/SMOKE/output2016v2/2032fj_16j/intermed/ptegu_wintershld/logs/smkinven_ptegu_wintershld_2032fj_16j.log

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).                                 
 Copyright (C) 1992-2002 MCNC,                                   
 (C) 1992-2018 Carlie J. Coats, Jr.,                             
 (C) 2003-2012 Baron Advanced Meteorological Systems, LLC, and   
 (C) 2014-2020 UNC Institute for the Environment.                
 Released under the GNU LGPL  License, version 2.1.  See URL     
                                                                 
     https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html     
                                                                 
 for conditions of use.                                          
                                                                 
 ioapi-3.2: $Id: init3.F90 185 2020-08-28 16:49:45Z coats $
 netCDF version 4.7.4 of Apr  7 2021 16:50:09 $
  
 SMOKE ---------------
 Copyright (c)2004 Environmental Modeling for Policy Development
 All rights reserved

 Program SMKINVEN, Version SMOKEv4.8.1_Jan2021
 Online documentation 
     http://www.cep.unc.edu/empd/products/smoke                 
  
 Program SMKINVEN to take ASCII area or point source files
 in IDA, EMS-95, or SMOKE list format, or mobile files
 in IDA format, and produce the I/O API and ASCII SMOKE
 inventory files and list of unique SCCs in the inventory.
  
  
 You will need to enter the logical names for the input and
 output files (and to have set them prior to program start,
 using "setenv <logicalname> <pathname>").
  
 You may use END_OF-FILE (control-D) to quit the program
 during logical-name entry. Default responses are given in
 brackets [LIKE THIS] and can be accepted by hitting the
 <RETURN> key.

3.420u 0.083s 0:03.51 99.7% 0+0k 0+13760io 0pf+0w
SCRIPT NOTE: timetracker is replacing line 2 of the TIMELOG file
now checking log file /home/Michael/RAID1/SMOKE/output2016v2/2032fj_16j/intermed/ptegu_wintershld/logs/smkinven_ptegu_wintershld_2032fj_16j.log

SCRIPT NOTE: Automatically deleting log file.
/home/Michael/RAID1/SMOKE/output2016v2/2032fj_16j/intermed/ptegu_wintershld/logs/spcmat_ptegu_wintershld_2032fj_16j_cmaq_cb6ae7.log

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).                                 
 Copyright (C) 1992-2002 MCNC,                                   
 (C) 1992-2018 Carlie J. Coats, Jr.,                             
 (C) 2003-2012 Baron Advanced Meteorological Systems, LLC, and   
 (C) 2014-2020 UNC Institute for the Environment.                
 Released under the GNU LGPL  License, version 2.1.  See URL     
                                                                 
     https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html     
                                                                 
 for conditions of use.                                          
                                                                 
 ioapi-3.2: $Id: init3.F90 185 2020-08-28 16:49:45Z coats $
 netCDF version 4.7.4 of Apr  7 2021 16:50:09 $
  
 SMOKE ---------------
 Copyright (c)2004 Environmental Modeling for Policy Development
 All rights reserved

 Program SPCMAT, Version SMOKEv4.8.1_Jan2021
 Online documentation 
     http://www.cep.unc.edu/empd/products/smoke                 
  
 Program SPCMAT to take a SMOKE area, mobile, or point source
 inventory file, a speciation profiles file, a speciation
 cross-reference file, an optional pollutant-to-pollutant,
 conversion file, and produce mass-based and/or mole-based
 SMOKE speciation matrices for all inventory pollutants
 using run-time defined combinations of pollutants and model
 species. The output species are defined at run time by the
 speciation profiles file, permitting support of any chemical
 mechanism.
  
  
 You will need to enter the logical names for the input and
 output files (and to have set them prior to program start,
 using "setenv <logicalname> <pathname>").
  
 You may use END_OF-FILE (control-D) to quit the program
 during logical-name entry. Default responses are given in
 brackets [LIKE THIS] and can be accepted by hitting the
 <RETURN> key.

3.090u 0.058s 0:03.15 99.6% 0+0k 0+23904io 0pf+0w
SCRIPT NOTE: timetracker is replacing line 2 of the TIMELOG file
now checking log file /home/Michael/RAID1/SMOKE/output2016v2/2032fj_16j/intermed/ptegu_wintershld/logs/spcmat_ptegu_wintershld_2032fj_16j_cmaq_cb6ae7.log

SCRIPT NOTE: Automatically deleting log file.
/home/Michael/RAID1/SMOKE/output2016v2/2032fj_16j/intermed/ptegu_wintershld/logs/grdmat_ptegu_wintershld_2032fj_16j_12US2.log

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).                                 
 Copyright (C) 1992-2002 MCNC,                                   
 (C) 1992-2018 Carlie J. Coats, Jr.,                             
 (C) 2003-2012 Baron Advanced Meteorological Systems, LLC, and   
 (C) 2014-2020 UNC Institute for the Environment.                
 Released under the GNU LGPL  License, version 2.1.  See URL     
                                                                 
     https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html     
                                                                 
 for conditions of use.                                          
                                                                 
 ioapi-3.2: $Id: init3.F90 185 2020-08-28 16:49:45Z coats $
 netCDF version 4.7.4 of Apr  7 2021 16:50:09 $
  
 SMOKE ---------------
 Copyright (c)2004 Environmental Modeling for Policy Development
 All rights reserved

 Program GRDMAT, Version SMOKEv4.8.1_Jan2021
 Online documentation 
     http://www.cep.unc.edu/empd/products/smoke                 
  
 Program GRDMAT to take a SMOKE area, mobile, or point source
 inventory file, gridding surrogates,  surrogate cross-
 reference, and an optional link definitions file, and produce
 a SMOKE gridding matrix for a grid defined at run time. For
 mobile sources, an "ungridding" matrix is also created to
 allow the use of gridded temperature data in assigning
 factors to mobile sources.
  
  
 You will need to enter the logical names for the input and
 output files (and to have set them prior to program start,
 using "setenv <logicalname> <pathname>").
  
 You may use END_OF-FILE (control-D) to quit the program
 during logical-name entry. Default responses are given in
 brackets [LIKE THIS] and can be accepted by hitting the
 <RETURN> key.

0.612u 0.015s 0:00.30 206.6% 0+0k 0+880io 0pf+0w
SCRIPT NOTE: timetracker is replacing line 2 of the TIMELOG file
now checking log file /home/Michael/RAID1/SMOKE/output2016v2/2032fj_16j/intermed/ptegu_wintershld/logs/grdmat_ptegu_wintershld_2032fj_16j_12US2.log

SCRIPT NOTE: Automatically deleting log file.
/home/Michael/RAID1/SMOKE/output2016v2/2032fj_16j/intermed/ptegu_wintershld/logs/smkreport_ptegu_wintershld_2032fj_16j_inv.log

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).                                 
 Copyright (C) 1992-2002 MCNC,                                   
 (C) 1992-2018 Carlie J. Coats, Jr.,                             
 (C) 2003-2012 Baron Advanced Meteorological Systems, LLC, and   
 (C) 2014-2020 UNC Institute for the Environment.                
 Released under the GNU LGPL  License, version 2.1.  See URL     
                                                                 
     https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html     
                                                                 
 for conditions of use.                                          
                                                                 
 ioapi-3.2: $Id: init3.F90 185 2020-08-28 16:49:45Z coats $
 netCDF version 4.7.4 of Apr  7 2021 16:50:09 $
  
 SMOKE ---------------
 Copyright (c)2004 Environmental Modeling for Policy Development
 All rights reserved

 Program SMKREPORT, Version SMOKEv4.8.1_Jan2021
 Online documentation 
     http://www.cep.unc.edu/empd/products/smoke                 
  
 Program SMKREPORT to take any combination of SMOKE intermed-
 iate files (but at least an inventory file) and generate
 one or more user-defined reports in any combination of output
 files.
  
  
 You will need to enter the logical names for the input and
 output files (and to have set them prior to program start,
 using "setenv <logicalname> <pathname>").
  
 You may use END_OF-FILE (control-D) to quit the program
 during logical-name entry. Default responses are given in
 brackets [LIKE THIS] and can be accepted by hitting the
 <RETURN> key.

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Note: During my SMOKE4.8.1 installation , I couldn’t find SMKROOT/doc directory. Please see the 4th comment of this thread SMOKE v4.81 installtion issue: make: *** [rdemspd.o] Error 1

This might be an issue specific to SMOKE 4.8.1 Smkreport. Is it possible for you to try an earlier version of Smkreport from 4.8 or 4.7?

Also, did you mention that you could run the same report with the 2016v1 platform?

We may have some other suggestions a little later…

In addition to the suggestion of trying SMOKE 4.8 instead of 4.8.1, if it works on your system you could try the executable from this package which includes Smkreport from 4.8:

https://gaftp.epa.gov/Air/emismod/2016/v2/smoke_2016v2_platform_core_01oct2021.zip

In addition to the standard output you posted, could you also provide the Smkreport log?

Unrelated to Smkreport, we see this in your standard output:

/home/Iqbal/Documents/SMOKE481/subsys/ioapi/Linux2_x86_64gfort/juldate: Permission denied.

You might need to give the juldate executable execute permissions?

That wouldn’t cause Smkreport any problems but might cause issues down the line in Temporal/Smkmerge.

Please see the Smkreport log from v4.8.1
smkreport_ptegu_wintershld_2032fj_16j_inv.txt (13.4 KB), the report ends with following lines

Value for REPLABEL:  'ptegu_wintershld'
 Value for REPLABEL:  'ptegu_wintershld'

However, I have tried copying SMKREPT program from my 2016v1 platform’s SMOKEV4.7 installation to the v4.8.1 directory. And IT WORKS.

Do I need to install v4.8 and copy the SMKREPT to v4.8.1 for the consistency of the version ( 4.8) ? Or It does not matter if I use SMKREPT from v4.7?

Thank you

Good news – we have some known issues with Smkreport in v4.8.1 that we haven’t quite been able to track down.

You should be OK to use the v4.7 Smkreport version unless you need a Smkreport feature that was added in SMOKE v4.8 – you should be able to see what was added in v4.8 in the release notes for that version.

1 Like

Hi @eyth.alison ,

So, I just installed v4.8. And copied the Smkreport to V4.8.1 which is giving me same issue

Please see the log file
smkreport_ptegu_wintershld_mar_2032fj_16j_temporal.txt (13.9 KB)

Are there any known issues for the Smkreport in v4.8?

Thanks

So, it works with 4.7 but not with v4.8 or v4.8.1?

Yes, It does work on v4.7

Hi,

I have installed 4.8.1 with ifort and Linux2_x86_64ifort_medium ioapi, and I am able to run your scenario without problems.

You may want to try with the latest release of 4.8.1 from github as we have fixed a potential issue in ptegu with gfortran.

Hi @chef ,

I don’t have an intel compiler in my system. So, I used gfortran compiler. Have you tried with gfortran?

Thank you

Not yet in my system, it will take me some time to setup everything and replicate.

But if in the meantime you want to download and try the very latest 4.8.1 that is on github now in your system, it would help verify if these changes work for you or the problem comes from somewhere else.

1 Like