Segmentation Fault in Temporal Process (SMOKE v4.8.1)

Hi everyone,

I’m trying to run SMOKE (v4.8.1/Ubuntu 18.04) and I keep getting the error below when running the Temporal Process, I saw an earlier post about a similar issue and followed the solution proposed in the post but I’m still getting this error:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
temporal           00000000005B43AD  Unknown               Unknown  Unknown
libpthread-2.27.s  00007FF4D920C980  Unknown               Unknown  Unknown
temporal           00000000004172B0  proctpro_                 270  proctpro.f
temporal           0000000000429B84  MAIN__                    477  temporal.f
temporal           000000000040371E  Unknown               Unknown  Unknown
libc-2.27.so       00007FF4D8C26BF7  __libc_start_main     Unknown  Unknown
temporal           0000000000403629  Unknown               Unknown  Unknown

I’ve already done the following trying to solve this issue:

  • I’ve installed the model on a Virtual Machine, I’ve increased the resources from its original 8 cores and 8GB RAM to 16 cores and 42GB RAM
  • Ran the benchmark test case and confirmed processes ran OK, and output was generated (the benchmark case ran fine with the original assigned resources)
  • Increased the default limits settings to unlimited (see my conf here)

The script run file I’m using is available here and the temporal log file is available here.

I ran out of ideas as to what I’m doing wrong, greatly appreciate if someone can point me in the right path?

Are you using the precompiled executables from the CMAS download package or did you compile the SMOKE codes and generate your own version of SMOKE executables?

I don’t remember exactly how much memory is required for EDGAR processing in SMOKE but 42GB does not seem enough since there are more than 2 million grid cells/sources which requires a lot of memory to process in SMOKE. Try to run the case with a single EDGAR file (one pollutant) if you are processing multiple EDGAR files for multiple pollutants to reduce the usage of memory.

You could try is setting G_RUNLEN to 250000 instead of 240000.

Beyond that, the way you define their dates is different than what we do in our scripts. You define G_STDATE / G_STTIME / G_RUNLEN, probably in the assigns file, and run Temporal for a single day, whereas we use a PROCDATES file (https://www.cmascenter.org/smoke/documentation/4.8.1/html/ch08s03s04.html) and run multiple days in a single Temporal instance. Here is a sample PROCDATES for representative dates (7 days per month) in April:

20160404 000000 250000

20160405 000000 250000

20160406 000000 250000

20160407 000000 250000

20160408 000000 250000

20160409 000000 250000

20160410 000000 250000

Hi @bbaek, I’ve increased the assigned resources to this server (it’s a virtual machine) and it now has 72 cores and 220GB of RAM, and I’m still getting the same error:

To answer your question, yes I’m using the precompiled executables from CMAS, I did not compiled my own version of the SMOKE executables, I’m also using the most recent version of the Intel Fortran compiler which is now shipped as Intel oneAPI, the two toolkits I installed were the Base Toolkit and the HPC Toolkit.

The NetCDF-C, NetCDF-Fortran and IOAPI v.3.2 were also installed using the Intel Compilers.

My understanding is that we are already trying to run SMOKE for only one pollutant (see below), but I’ll confirm this with the Professor I’m working with, if it helps, this is the ASSIGNS file we are using.


Hi @eyth.alison – we actually reduced the value of G_RUNLEN from 240000 to 010000, you can see it here, this is how those variables are currently set:

## Per-period environment variables
   setenv G_STDATE  2019213      # Julian start date
   setenv G_STTIME   000000      # start time (HHMMSS)
   setenv G_TSTEP     10000      # time step  (HHMMSS)
   setenv G_RUNLEN   010000      # run length (HHMMSS)
   setenv ESDATE   20190801      # Start date of emis time-based files/dirs
   setenv MSDATE   20190801      # Start date of met  time-based files
   setenv NDAYS           1      # Duration in days of each emissions file
   setenv MDAYS           1      # Duration in days of met  time-based files
   setenv YEAR         2019      # Base year for year-specific files

Regarding the way the dates are defined, my understanding is that we set it up the way the test case is setup? regardless, where we should put this PROCDATES file? maybe I’m looking at the wrong place but I’m not seeing it in the documentation.

So you also tried 240000 and 250000 run length? I would think that a length of 010000 wouldn’t provide any meaningful results and likely could introduce other issues that could cause a crash.

Try setting G_RUNLEN to 250000 to be consistent with our tried-and-true methods.

Note: we put our PROCDATES files in their own directory ($CASE/intermed/datefiles), but they can go anywhere as long as the location of the file is defined in the ASSIGNS file or somewhere else in the SMOKE setup. The SMOKE documentation doesn’t say too much about the PROCDATES file, but there is a page for it in the documentation: https://www.cmascenter.org/smoke/documentation/4.8.1/html/ch08s03s04.html

Hi @eyth.alison – thank you for taking the time to help me with this issue.

I just made the adjustment as you suggested, and changed the value of the G_RUNLEN variable to 250000, you can see my new assigns file here, however, I’m still getting the same error (see also log here):

What other thing I could do or you suggest to do to troubleshoot this?

@eyth.alison and @bbaek happy to report that we were finally able to solve this issue, we increased the max number of open files however, we needed to also increase the max stack size (KB), we did this by executing the following:

limit stacksize unlimited

to make it “permanent”, we added this to our ~/.cshrc file.

Glad to hear that you were able to resolve the issue and moving forward. Please let us know if you run into any other issues.