# Segmentation Fault in Temporal Process (SMOKE v4.8.1) **URL:** \ **Category:** Run Errors and Warnings\ **Tags:** smoke, smoke-481\ **Created:** [July 8, 2021, 1:16am UTC](https://forum.cmascenter.org/t/segmentation-fault-in-temporal-process-smoke-v4-8-1/2484 "2021-07-08T01:16:07Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![ever.barreto](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/ever.barreto/32/778_2.png) [@ever.barreto](https://forum.cmascenter.org/u/ever.barreto)\ **Post date:** [July 8, 2021, 1:16am UTC](https://forum.cmascenter.org/t/segmentation-fault-in-temporal-process-smoke-v4-8-1/2484/1 "2021-07-08T01:16:07Z")
Hi everyone, I’m trying to run SMOKE (v4.8.1/Ubuntu 18.04) and I keep getting the error below when running the Temporal Process, [I saw an earlier post about a similar issue and followed the solution proposed in the post](https://forum.cmascenter.org/t/segmentation-fault-in-temporal-process/2310) but I’m still getting this error: ```auto forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source temporal 00000000005B43AD Unknown Unknown Unknown libpthread-2.27.s 00007FF4D920C980 Unknown Unknown Unknown temporal 00000000004172B0 proctpro_ 270 proctpro.f temporal 0000000000429B84 MAIN__ 477 temporal.f temporal 000000000040371E Unknown Unknown Unknown libc-2.27.so 00007FF4D8C26BF7 __libc_start_main Unknown Unknown temporal 0000000000403629 Unknown Unknown Unknown ``` I’ve already done the following trying to solve this issue: - I’ve installed the model on a Virtual Machine, I’ve increased the resources from its original 8 cores and 8GB RAM to 16 cores and 42GB RAM - Ran the benchmark test case and confirmed processes ran OK, and output was generated (the benchmark case ran fine with the original assigned resources) - Increased the default limits settings to `unlimited` ([see my conf here](https://gist.github.com/everdaniel/1b6f5eac23427a00bedf9c70158440b7)) The [script run file I’m using is available here](https://gist.github.com/everdaniel/54fa431003b4878f7a1718f6deb0432f) and [the temporal log file is available here](https://gist.github.com/everdaniel/3f6f4cf4adcc951c4b77a2e7a528dce0). I ran out of ideas as to what I’m doing wrong, greatly appreciate if someone can point me in the right path? ---
**Author:** ![bbaek](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/bbaek/32/22_2.png) [@bbaek](https://forum.cmascenter.org/u/bbaek)\ **Post date:** [July 8, 2021, 2:36pm UTC](https://forum.cmascenter.org/t/segmentation-fault-in-temporal-process-smoke-v4-8-1/2484/2 "2021-07-08T14:36:56Z")
Are you using the precompiled executables from the CMAS download package or did you compile the SMOKE codes and generate your own version of SMOKE executables? I don’t remember exactly how much memory is required for EDGAR processing in SMOKE but 42GB does not seem enough since there are more than 2 million grid cells/sources which requires a lot of memory to process in SMOKE. Try to run the case with a single EDGAR file (one pollutant) if you are processing multiple EDGAR files for multiple pollutants to reduce the usage of memory. ---
**Author:** ![eyth.alison](https://avatars.discourse-cdn.com/v4/letter/e/7ab992/32.png) [@eyth.alison](https://forum.cmascenter.org/u/eyth.alison)\ **Post date:** [July 8, 2021, 9:20pm UTC](https://forum.cmascenter.org/t/segmentation-fault-in-temporal-process-smoke-v4-8-1/2484/3 "2021-07-08T21:20:17Z")
You could try is setting G\_RUNLEN to 250000 instead of 240000. Beyond that, the way you define their dates is different than what we do in our scripts. You define G\_STDATE / G\_STTIME / G\_RUNLEN, probably in the assigns file, and run Temporal for a single day, whereas we use a PROCDATES file ([https://www.cmascenter.org/smoke/documentation/4.8.1/html/ch08s03s04.html](https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cmascenter.org%2Fsmoke%2Fdocumentation%2F4.8.1%2Fhtml%2Fch08s03s04.html&data=04%7C01%7CEyth.Alison%40epa.gov%7C2f4063bab4314c5bf1e808d9424a97be%7C88b378b367484867acf976aacbeca6a7%7C0%7C0%7C637613710396965283%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SII7FZ%2Bl%2FIxwGEElDt7xNGas4NpUIUves5Q9%2F2re4nk%3D&reserved=0)) and run multiple days in a single Temporal instance. Here is a sample PROCDATES for representative dates (7 days per month) in April: 20160404 000000 250000 20160405 000000 250000 20160406 000000 250000 20160407 000000 250000 20160408 000000 250000 20160409 000000 250000 20160410 000000 250000 ---
**Author:** ![ever.barreto](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/ever.barreto/32/778_2.png) [@ever.barreto](https://forum.cmascenter.org/u/ever.barreto)\ **Post date:** [July 15, 2021, 4:41pm UTC](https://forum.cmascenter.org/t/segmentation-fault-in-temporal-process-smoke-v4-8-1/2484/4 "2021-07-15T16:41:39Z")
Hi @bbaek, I’ve increased the assigned resources to this server (it’s a virtual machine) and it now has 72 cores and 220GB of RAM, and I’m still getting the same error: ![Screen Shot 2021-07-15 at 00.27.46](https://canada1.discourse-cdn.com/flex027/uploads/cmas/original/2X/0/0ebdb26071f05d2699eeaba6f6e1baf5fb4c50c0.jpeg) To answer your question, yes I’m using the precompiled executables from CMAS, _I did not compiled my own version of the SMOKE executables_, I’m also using the most recent version of the Intel Fortran compiler which is now shipped as Intel oneAPI, the two toolkits I installed were the [Base Toolkit](https://software.intel.com/content/www/us/en/develop/tools/oneapi/base-toolkit/download.html) and the [HPC Toolkit](https://software.intel.com/content/www/us/en/develop/tools/oneapi/hpc-toolkit/download.html). The NetCDF-C, NetCDF-Fortran and IOAPI v.3.2 were also installed using the Intel Compilers. My understanding is that we are already trying to run SMOKE for only one pollutant (see below), but I’ll confirm this with the Professor I’m working with, if it helps, [this is the ASSIGNS file we are using](https://gist.github.com/everdaniel/d7d516c17149a728e536bb2304766c72). ![Screen Shot 2021-07-15 at 12.22.25](https://canada1.discourse-cdn.com/flex027/uploads/cmas/original/2X/b/bfd24bccb6d2a5ae174f045fa998868b5dea7649.jpeg) ![Screen Shot 2021-07-15 at 12.23.47](https://canada1.discourse-cdn.com/flex027/uploads/cmas/original/2X/c/c2788e34af6f51eb495448f25243b931f2b817e8.jpeg) ---
**Author:** ![ever.barreto](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/ever.barreto/32/778_2.png) [@ever.barreto](https://forum.cmascenter.org/u/ever.barreto)\ **Post date:** [July 15, 2021, 4:47pm UTC](https://forum.cmascenter.org/t/segmentation-fault-in-temporal-process-smoke-v4-8-1/2484/5 "2021-07-15T16:47:38Z")
Hi @eyth.alison – we actually reduced the value of `G_RUNLEN` from `240000` to `010000`, [you can see it here](https://gist.github.com/everdaniel/d7d516c17149a728e536bb2304766c72#file-assigns-biogenicas-L56), this is how those variables are currently set: ```auto ## Per-period environment variables setenv G_STDATE 2019213 # Julian start date setenv G_STTIME 000000 # start time (HHMMSS) setenv G_TSTEP 10000 # time step (HHMMSS) setenv G_RUNLEN 010000 # run length (HHMMSS) setenv ESDATE 20190801 # Start date of emis time-based files/dirs setenv MSDATE 20190801 # Start date of met time-based files setenv NDAYS 1 # Duration in days of each emissions file setenv MDAYS 1 # Duration in days of met time-based files setenv YEAR 2019 # Base year for year-specific files ``` Regarding the way the dates are defined, my understanding is that we set it up the way the test case is setup? regardless, where we should put this `PROCDATES` file? maybe I’m looking at the wrong place but I’m not seeing it in the documentation. ---
**Author:** ![eyth.alison](https://avatars.discourse-cdn.com/v4/letter/e/7ab992/32.png) [@eyth.alison](https://forum.cmascenter.org/u/eyth.alison)\ **Post date:** [July 15, 2021, 5:24pm UTC](https://forum.cmascenter.org/t/segmentation-fault-in-temporal-process-smoke-v4-8-1/2484/6 "2021-07-15T17:24:40Z")
So you also tried 240000 and 250000 run length? I would think that a length of 010000 wouldn’t provide any meaningful results and likely could introduce other issues that could cause a crash. Try setting G\_RUNLEN to 250000 to be consistent with our tried-and-true methods. Note: we put our PROCDATES files in their own directory ($CASE/intermed/datefiles), but they can go anywhere as long as the location of the file is defined in the ASSIGNS file or somewhere else in the SMOKE setup. The SMOKE documentation doesn’t say too much about the PROCDATES file, but there is a page for it in the documentation: [https://www.cmascenter.org/smoke/documentation/4.8.1/html/ch08s03s04.html](https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cmascenter.org%2Fsmoke%2Fdocumentation%2F4.8.1%2Fhtml%2Fch08s03s04.html&data=04%7C01%7CEyth.Alison%40epa.gov%7C7122c9d556154b584cd608d947b52ccf%7C88b378b367484867acf976aacbeca6a7%7C0%7C0%7C637619665738603145%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9%2BQ6%2BmiJCBBnrmh5qusAF2rys7lRGQhuA%2BMEuLLuK2w%3D&reserved=0) ---
**Author:** ![ever.barreto](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/ever.barreto/32/778_2.png) [@ever.barreto](https://forum.cmascenter.org/u/ever.barreto)\ **Post date:** [July 15, 2021, 9:56pm UTC](https://forum.cmascenter.org/t/segmentation-fault-in-temporal-process-smoke-v4-8-1/2484/7 "2021-07-15T21:56:01Z")
Hi @eyth.alison – thank you for taking the time to help me with this issue. I just made the adjustment as you suggested, and changed the value of the `G_RUNLEN` variable to `250000`, [you can see my new assigns file here](https://gist.github.com/everdaniel/2c0ba6270ed1d083a0d823348b91fb37), however, I’m still getting the same error ([see also log here](https://gist.github.com/everdaniel/1073c52c9a398a86f2caf972e887f944)): ![Screen Shot 2021-07-15 at 17.53.33](https://canada1.discourse-cdn.com/flex027/uploads/cmas/original/2X/2/29c6886a780a4831c1e37074fbf8bbaed094c04b.jpeg) What other thing I could do or you suggest to do to troubleshoot this? ---
**Author:** ![ever.barreto](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/ever.barreto/32/778_2.png) [@ever.barreto](https://forum.cmascenter.org/u/ever.barreto)\ **Post date:** [July 16, 2021, 12:43am UTC](https://forum.cmascenter.org/t/segmentation-fault-in-temporal-process-smoke-v4-8-1/2484/8 "2021-07-16T00:43:02Z")
@eyth.alison and @bbaek happy to report that we were finally able to solve this issue, we increased the `max number of open files` however, we needed to also increase the `max stack size (KB)`, we did this by executing the following: ```auto limit stacksize unlimited ``` to make it “permanent”, we added this to our `~/.cshrc` file. ---
**Author:** ![bbaek](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/bbaek/32/22_2.png) [@bbaek](https://forum.cmascenter.org/u/bbaek)\ **Post date:** [July 16, 2021, 2:29am UTC](https://forum.cmascenter.org/t/segmentation-fault-in-temporal-process-smoke-v4-8-1/2484/9 "2021-07-16T02:29:17Z")
Glad to hear that you were able to resolve the issue and moving forward. Please let us know if you run into any other issues.