I’ve already done the following trying to solve this issue:
I’ve installed the model on a Virtual Machine, I’ve increased the resources from its original 8 cores and 8GB RAM to 16 cores and 42GB RAM
Ran the benchmark test case and confirmed processes ran OK, and output was generated (the benchmark case ran fine with the original assigned resources)
Increased the default limits settings to unlimited (see my conf here)
Are you using the precompiled executables from the CMAS download package or did you compile the SMOKE codes and generate your own version of SMOKE executables?
I don’t remember exactly how much memory is required for EDGAR processing in SMOKE but 42GB does not seem enough since there are more than 2 million grid cells/sources which requires a lot of memory to process in SMOKE. Try to run the case with a single EDGAR file (one pollutant) if you are processing multiple EDGAR files for multiple pollutants to reduce the usage of memory.
You could try is setting G_RUNLEN to 250000 instead of 240000.
Beyond that, the way you define their dates is different than what we do in our scripts. You define G_STDATE / G_STTIME / G_RUNLEN, probably in the assigns file, and run Temporal for a single day, whereas we use a PROCDATES file (https://www.cmascenter.org/smoke/documentation/4.8.1/html/ch08s03s04.html) and run multiple days in a single Temporal instance. Here is a sample PROCDATES for representative dates (7 days per month) in April:
Hi @bbaek, I’ve increased the assigned resources to this server (it’s a virtual machine) and it now has 72 cores and 220GB of RAM, and I’m still getting the same error:
To answer your question, yes I’m using the precompiled executables from CMAS, I did not compiled my own version of the SMOKE executables, I’m also using the most recent version of the Intel Fortran compiler which is now shipped as Intel oneAPI, the two toolkits I installed were the Base Toolkit and the HPC Toolkit.
The NetCDF-C, NetCDF-Fortran and IOAPI v.3.2 were also installed using the Intel Compilers.
My understanding is that we are already trying to run SMOKE for only one pollutant (see below), but I’ll confirm this with the Professor I’m working with, if it helps, this is the ASSIGNS file we are using.
Hi @eyth.alison – we actually reduced the value of G_RUNLEN from 240000 to 010000, you can see it here, this is how those variables are currently set:
## Per-period environment variables
setenv G_STDATE 2019213 # Julian start date
setenv G_STTIME 000000 # start time (HHMMSS)
setenv G_TSTEP 10000 # time step (HHMMSS)
setenv G_RUNLEN 010000 # run length (HHMMSS)
setenv ESDATE 20190801 # Start date of emis time-based files/dirs
setenv MSDATE 20190801 # Start date of met time-based files
setenv NDAYS 1 # Duration in days of each emissions file
setenv MDAYS 1 # Duration in days of met time-based files
setenv YEAR 2019 # Base year for year-specific files
Regarding the way the dates are defined, my understanding is that we set it up the way the test case is setup? regardless, where we should put this PROCDATES file? maybe I’m looking at the wrong place but I’m not seeing it in the documentation.
So you also tried 240000 and 250000 run length? I would think that a length of 010000 wouldn’t provide any meaningful results and likely could introduce other issues that could cause a crash.
Try setting G_RUNLEN to 250000 to be consistent with our tried-and-true methods.
Note: we put our PROCDATES files in their own directory ($CASE/intermed/datefiles), but they can go anywhere as long as the location of the file is defined in the ASSIGNS file or somewhere else in the SMOKE setup. The SMOKE documentation doesn’t say too much about the PROCDATES file, but there is a page for it in the documentation: https://www.cmascenter.org/smoke/documentation/4.8.1/html/ch08s03s04.html
@eyth.alison and @bbaek happy to report that we were finally able to solve this issue, we increased the max number of open files however, we needed to also increase the max stack size (KB), we did this by executing the following:
limit stacksize unlimited
to make it “permanent”, we added this to our ~/.cshrc file.
