# CCTM stops running on long simulation **URL:** \ **Category:** Run Time Errors and Issues\ **Created:** [September 12, 2018, 8:36pm UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292 "2018-09-12T20:36:30Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![ever.barreto](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/ever.barreto/32/778_2.png) [@ever.barreto](https://forum.cmascenter.org/u/ever.barreto)\ **Post date:** [September 12, 2018, 8:36pm UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/1 "2018-09-12T20:36:30Z")
Hi all, ok, so on a different topic I asked [how to setup a long simulation](https://forum.cmascenter.org/t/cctm-nsteps-for-long-period-20-days/290/3) so I then proceeded to update my `run_cctm.csh` script and launched the simulation, however, the simulation halts after a few hours of simulation, before this run, I was able to run the model for 24 hours just fine. I can’t find any immediate errors on the `run_cctm.csh` output and on the `CCTM_LOG` files, this is my CCTM configuration: [http://demo-staging.com/cmaq\_long\_run/run\_cctm.csh](http://demo-staging.com/cmaq_long_run/run_cctm.csh) and these are the output/logs: [http://demo-staging.com/cmaq\_long\_run/run.benchmark.log](http://demo-staging.com/cmaq_long_run/run.benchmark.log) [http://demo-staging.com/cmaq\_long\_run/CTM\_LOG\_001.v521\_gcc\_CMAQ-ASU1K\_20150601](http://demo-staging.com/cmaq_long_run/CTM_LOG_001.v521_gcc_CMAQ-ASU1K_20150601) [http://demo-staging.com/cmaq\_long\_run/CTM\_LOG\_002.v521\_gcc\_CMAQ-ASU1K\_20150601](http://demo-staging.com/cmaq_long_run/CTM_LOG_002.v521_gcc_CMAQ-ASU1K_20150601) [http://demo-staging.com/cmaq\_long\_run/CTM\_LOG\_003.v521\_gcc\_CMAQ-ASU1K\_20150601](http://demo-staging.com/cmaq_long_run/CTM_LOG_003.v521_gcc_CMAQ-ASU1K_20150601) [http://demo-staging.com/cmaq\_long\_run/CTM\_LOG\_004.v521\_gcc\_CMAQ-ASU1K\_20150601](http://demo-staging.com/cmaq_long_run/CTM_LOG_004.v521_gcc_CMAQ-ASU1K_20150601) [http://demo-staging.com/cmaq\_long\_run/CTM\_LOG\_005.v521\_gcc\_CMAQ-ASU1K\_20150601](http://demo-staging.com/cmaq_long_run/CTM_LOG_005.v521_gcc_CMAQ-ASU1K_20150601) Thanks in advance for the help! ---
**Author:** ![lizadams](https://avatars.discourse-cdn.com/v4/letter/l/49beb7/32.png) [@lizadams](https://forum.cmascenter.org/u/lizadams)\ **Post date:** [September 12, 2018, 8:57pm UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/2 "2018-09-12T20:57:05Z")
For the CMAQv5.2.1 version, it is necessary to change the END\_DATE if you want to do a longer duration run. Keep NSTEPS set to one day (24 hours), as this is the number of hours in one day. > set NSTEPS = 240000 #\> time duration (HHMMSS) for this run The run script has a `while loop` that is set up to run the model from the `START_DATE` to the `END_DATE`. As long as the current day is less than the `STOP_DAY`, then the model will continue to run. I’ve pulled out the lines of code that control the `WHILE` loop for you to review: > #\> Set Start and End Days for looping > setenv NEW\_START TRUE #\> Set to FALSE for model restart > set START\_DATE = “2011-07-01” #\> beginning date (July 1, 2011) > set END\_DATE = “2011-07-01” #\> ending date (July 14, 2011) > set STOP\_DAY = `date -ud "${END_DATE}" +%Y%j` #\> Convert YYYY-MM-DD to YYYYJJJ This is the start of the loop: > while ($TODAYJ \<= $STOP\_DAY ) #\>Compare dates in terms of YYYYJJJ Before the end loop, the script increments both the Gregorian and Julian Days. > #\> The next simulation day will, by definition, be a restart > setenv NEW\_START false > #\> Increment both Gregorian and Julian Days > set TODAYG = `date -ud "${TODAYG}+1days" +%Y-%m-%d` #\> Add a day for tomorrow > set TODAYJ = `date -ud "${TODAYG}" +%Y%j` #\> Convert YYYY-MM-DD to YYYYJJJ > end #Loop to the next Simulation Day ---
**Author:** ![ever.barreto](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/ever.barreto/32/778_2.png) [@ever.barreto](https://forum.cmascenter.org/u/ever.barreto)\ **Post date:** [September 13, 2018, 2:43am UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/3 "2018-09-13T02:43:42Z")
Hi @lizadams made changes to my run `cctm_script.csh` as advised, NSTEPS was set to 24 hours: ``` set NSTEPS = 240000 ``` and `START_DATE` / `END_DATE` are set to the simulation period, in my case, June 1st 2015 to June 20 2015: ``` set START_DATE = "2015-06-01" #> beginning date (July 1, 2011) set END_DATE = "2015-06-20" #> ending date (July 14, 2011) ``` however, the model still doesn’t go past the first 24 hours. I actually think it’s crashing but my biggest issue is that the model isn’t outputting any sort of error and I’m having a hard time troubleshooting what’s the issue, anything you can think of that I could turn on or something to gather more info? This is my current `run_cctm.csh` script: > In case it helps, here are the output logs: From `run_ctm.csh` [http://www.demo-staging.com/cctm\_run/run.benchmark.txt](http://www.demo-staging.com/cctm_run/run.benchmark.txt) From CCTM [http://www.demo-staging.com/cctm\_run/CTM\_LOG\_001.v521\_gcc\_CMAQ-ASU1K\_20150601.txt](http://www.demo-staging.com/cctm_run/CTM_LOG_001.v521_gcc_CMAQ-ASU1K_20150601.txt) [http://www.demo-staging.com/cctm\_run/CTM\_LOG\_002.v521\_gcc\_CMAQ-ASU1K\_20150601.txt](http://www.demo-staging.com/cctm_run/CTM_LOG_002.v521_gcc_CMAQ-ASU1K_20150601.txt) [http://www.demo-staging.com/cctm\_run/CTM\_LOG\_003.v521\_gcc\_CMAQ-ASU1K\_20150601.txt](http://www.demo-staging.com/cctm_run/CTM_LOG_003.v521_gcc_CMAQ-ASU1K_20150601.txt) [http://www.demo-staging.com/cctm\_run/CTM\_LOG\_004.v521\_gcc\_CMAQ-ASU1K\_20150601.txt](http://www.demo-staging.com/cctm_run/CTM_LOG_004.v521_gcc_CMAQ-ASU1K_20150601.txt) [http://www.demo-staging.com/cctm\_run/CTM\_LOG\_005.v521\_gcc\_CMAQ-ASU1K\_20150601.txt](http://www.demo-staging.com/cctm_run/CTM_LOG_005.v521_gcc_CMAQ-ASU1K_20150601.txt) Thanks in advance for your help! ---
**Author:** ![lizadams](https://avatars.discourse-cdn.com/v4/letter/l/49beb7/32.png) [@lizadams](https://forum.cmascenter.org/u/lizadams)\ **Post date:** [September 13, 2018, 1:45pm UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/4 "2018-09-13T13:45:15Z")
@bbaek and @ever.barreto I am seeing this WARNING at the end of this CMAQ run in the log files: > > > —\>\> WARNING in subroutine EMIS\_SPC\_CHECK on PE 005 > > > For optimal predictions, species with the missing surrogates should have a surrogate found in at least one source. > > > M3WARN: DTBUF 0:00:00 June 1, 2015 (2015152:000000) > > > It appears that the model doesn’t progress beyond that point, so perhaps it is crashing right away. I also see the following warning earlier in the log file. > Checking header data for file: EMIS\_1 > Inconsistent value for vertical level ---
**Author:** ![ever.barreto](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/ever.barreto/32/778_2.png) [@ever.barreto](https://forum.cmascenter.org/u/ever.barreto)\ **Post date:** [September 13, 2018, 4:20pm UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/5 "2018-09-13T16:20:11Z")
Hi @lizadams, yes I did saw those warnings, however, the are the same warnings that I got when I ran the simulation for 1 day, and the simulation completed successfully. That’s why I think it’s something related to the simulation being more than 1 day. Is there maybe like a debug option that I can turn to get more info on what the model is doing before it crashes/finishes? Thanks for your time and patience! ---
**Author:** ![ernesto.pinoc](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/ernesto.pinoc/32/43_2.png) [@ernesto.pinoc](https://forum.cmascenter.org/u/ernesto.pinoc)\ **Post date:** [September 13, 2018, 5:53pm UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/6 "2018-09-13T17:53:43Z")
Hello Ever: Have you tried to run 3 days for example changing the “set NSTEPS = 240000” to 7200000?? Maybe one of your inputs do not have the information for the periods you want. ---
**Author:** ![ernesto.pinoc](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/ernesto.pinoc/32/43_2.png) [@ernesto.pinoc](https://forum.cmascenter.org/u/ernesto.pinoc)\ **Post date:** [September 13, 2018, 5:54pm UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/7 "2018-09-13T17:54:27Z")
Sorry, set NSTEPS = 720000 ---
**Author:** ![ever.barreto](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/ever.barreto/32/778_2.png) [@ever.barreto](https://forum.cmascenter.org/u/ever.barreto)\ **Post date:** [September 13, 2018, 6:29pm UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/8 "2018-09-13T18:29:42Z")
Hi @ernesto.pinoc, I didn’t tried running it for 3 days, I did tried to run it for 20 days by increasing NSTEPS to 4800000 as you advised here: > [@CCTM NSTEPS for long period (20 days)](https://forum.cmascenter.org/t/cctm-nsteps-for-long-period-20-days/290/2): > > Hello: Just put set NSTEPS = 4800000 Don´t worry about if you insert another digit for HH. It will run for 20 days. I ran 15 days using 3600000 and it finished successfully. Hope this help you but I will try to run it for 3 days and report back. Thanks! ---
**Author:** ![ever.barreto](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/ever.barreto/32/778_2.png) [@ever.barreto](https://forum.cmascenter.org/u/ever.barreto)\ **Post date:** [September 14, 2018, 5:41pm UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/9 "2018-09-14T17:41:45Z")
Hi @ernesto.pinoc, did another test run with NSTEPS set to 720000 but got the same results again, model seems to halt after day one. I saw you did a run for 15 days here: > [@BCON m3conc from CCTM outputs](https://forum.cmascenter.org/t/bcon-m3conc-from-cctm-outputs/299): > > Hello: I did a run in CCTM for 15 days. Now I want to continue for 16 days to complete 1 month (January) and I was wondering if I have to use the same BCON file as used for the first run, or instead, I could use a bcon file but using module m3conc. I reviewed the bcon script and the input required is a file called CCTM\_d1bCONC.d1b. I was looking for it in the output file and I did not find it. Could you please help me to understand what file is required? Thanks for your time any chance you can share with me your `run_cctm.csh` script? Just want to double check and compare my settings against yours, maybe something pops up, think is possible? Thank you! ---
**Author:** ![cjcoats](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@cjcoats](https://forum.cmascenter.org/u/cjcoats)\ **Post date:** [September 14, 2018, 8:48pm UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/10 "2018-09-14T20:48:23Z")
Looking at this, and looking at "initscen.F in the source code I have a response, and additionally a problem-report. The relevant environment variable is CTM\_RUNLEN, which should give the duration of the run in format H\*MMSS, i.e., ``` 100*(100*HOURS + MINUTES) + SECONDS ``` The script needs to deal with this correctly; moreover, this value should NOT be over-written elsewhere. Doing so is metadata perjury. And then when I start looking at the code and at your log, I find that this routine is improperly using variables STDATE, STTIME before they are defined: e.g., its very first “clause”, before they have been retrieved from the environment, uses them: ``` VARDESC = 'Main Program Name' CALL M3MESG( VARDESC ) CALL ENVSTR( CTM_PROGNAME, VARDESC, 'DRIVER', PROGNAME, STATUS ) IF ( STATUS .EQ. 1 ) THEN MSG = 'Environment variable improperly formatted' CALL M3EXIT( PNAME, STDATE, STTIME, MSG, XSTAT2 ) ELSE IF ( STATUS .EQ. -1 ) THEN MSG = 'Environment variable set, but empty ... Using default:' WRITE( LOGDEV, '(5X, A, I9)' ) MSG, STTIME ELSE IF ( STATUS .EQ. -2 ) THEN MSG = 'Environment variable not set ... Using default:' WRITE( LOGDEV, '(5X, A, I9)' ) MSG, STTIME END IF ``` That’s the reason for the “\*\*\*\*\*\*\*\*” in your log. By convention, errors before the start of the time-stepped execution of the model should be reported as having date&time 0:0. Moreover, the “Using default:’…STTIME” messages do NOT report the correct default – for ANY of the environment-variable calls. For what it’s worth. Carlie J. Coats, Jr. I/O API Author/Maintainer ---
**Author:** ![ever.barreto](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/ever.barreto/32/778_2.png) [@ever.barreto](https://forum.cmascenter.org/u/ever.barreto)\ **Post date:** [September 15, 2018, 1:34am UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/11 "2018-09-15T01:34:31Z")
Hi @cjcoats, really appreciate your time looking into this! From your comment however, I am not sure how to proceed, is there something I can do to get it fixed? Or should I go one version down for the model (CMAQ 5.1)? Thank you! ---
**Author:** ![cgnolte](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/cgnolte/32/53_2.png) [@cgnolte](https://forum.cmascenter.org/u/cgnolte)\ **Post date:** [September 17, 2018, 1:43pm UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/12 "2018-09-17T13:43:48Z")
This block in the logfile: “EMIS\_1” opened as OLD:READ-ONLY File name “/root/CMAQ-5.2.1/data/CMAQ-ASU1K/cctm\_input/emis/HERMESv3\_par\_20150601.nc” File type GRDDED3 Execution ID “0.1alpha” Grid name “” Dimensions: 53 rows, 53 cols, 18 lays, 31 vbles NetCDF ID: 524288 opened as READONLY Starting date and time 2015152:000000 (0:00:00 June 1, 2015) Timestep 010000 (1:00:00 hh:mm:ss) Maximum current record number 25 Checking header data for file: EMIS\_1 Inconsistent value for vertical level shows that your emissions file has 18 layers, which differs from your meteorology, which has 29 layers. In your run script, below where you set EMISfile, also set CTM\_EMLAYS: set EMISfile = HERMESv3\_par\_${YYYYMMDD}.nc setenv CTM\_EMLAYS 18 This block in your log file: “OCEAN\_1” opened as OLD:READ-ONLY File name “/root/CMAQ-5.2.1/data/CMAQ-ASU1K/cctm\_input/land/ocean\_file.dummy.CMAQ-ASU1K.nc” File type GRDDED3 Execution ID “???” Grid name “CMAQ-ASU1K” Dimensions: 53 rows, 53 cols, 29 lays, 2 vbles NetCDF ID: 589824 opened as READONLY Starting date and time 2015152:000000 (0:00:00 June 1, 2015) Timestep 010000 (1:00:00 hh:mm:ss) Maximum current record number 504 Checking header data for file: OCEAN\_1 Inconsistent values for VGTYP: 1 versus 7 Inconsistent values for VGTOP: 1.0000E+02 versus 5.0000E+03 Inconsistent value for vertical level shows that your ocean file is rather strange. It should be a single layer, not 29, and it should be time-independent, rather than having 504 data records. I don’t know whether either of those differences are affecting you. I agree it is strange that you are not getting a clearer error in your log file. Have you tried a multiday run of the benchmark case? ---
**Author:** ![ever.barreto](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/ever.barreto/32/778_2.png) [@ever.barreto](https://forum.cmascenter.org/u/ever.barreto)\ **Post date:** [September 17, 2018, 5:32pm UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/13 "2018-09-17T17:32:46Z")
Hi @cgnolte Re: Emissions FIle, ok cool, going to make that change to my run script and change it to 18 levels. Re: Ocean file, yes, [I had to create a ‘zeroed’ ocean file as my domain doesn’t have ocean areas in it, on a separate thread on the former email list](https://lists.unc.edu/read/messages?id=8856365). I’m not 100% sure I did the process correctly as that script referenced to me wasn’t working so I had to do it by hand. I didn’t tried to run the multiday benchmark case, I only ran the single day benchmark case, I’ll look into that and see if I get the same issue. ---
**Author:** ![FengLiu](https://avatars.discourse-cdn.com/v4/letter/f/49beb7/32.png) [@FengLiu](https://forum.cmascenter.org/u/FengLiu)\ **Post date:** [October 2, 2020, 8:53pm UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/14 "2020-10-02T20:53:48Z")
Hi, I have to move back to CMAQv5.2 in order to compare ability of PM-10 modeling between CMAQ v5.2 and CMAQ v5.3 because I had extremely lower PM-10 modeling results using CMAQ 5.3 and 5.3.1. [[Extremely low PM10 by CMAQv5.3](https://forum.cmascenter.org/t/extremely-low-pm10-by-cmaqv5-3/1785)] However, I got the same issue of CCTM stops running without any error message as you described here when I ran CMAQ v5.2. Because CMAQ 5.2 builder does not have DEBUG option as CMAQ 5.3 does, I checked where and why the CCTM halts (seems suspending) by marking the code (related subroutines and modules) step by step according to log files. I finally found that CCTM is running instead of halting but never reaching to the convergence criteria, and then CCTM actually gets into a infinite loop at Line 281, module SEDIMENTATION.F dealing with sedimentation of aerosols: 281 do while ( adts .lt. dtsec ) I changed this line into the following after I referred to the same module in CMAQ 5.3, 281 do while ( adts .lt. dtsec .and. iter .le. 20). ! you can test different criteria (10, 20, …) 282 iter = iter + 1. ! (do not forget to declare this integer variable, iter) One of my cases (with different emissions) was crashed with the updates due to AQCHEM total iterations exceeded 60000 (at Line 926 in aqchem.F). The CCTM ran successfully after I increased ICNTAQ to 100000. I hope this post helps you to figure out why CCTM 5.2 halts at this point. But I highly recommend you to use the latest version of CMAQ. Thanks. ---
**Author:** ![lizadams](https://avatars.discourse-cdn.com/v4/letter/l/49beb7/32.png) [@lizadams](https://forum.cmascenter.org/u/lizadams)\ **Post date:** [October 2, 2020, 9:01pm UTC](https://forum.cmascenter.org/t/cctm-stops-running-on-long-simulation/292/15 "2020-10-02T21:01:50Z")
Thank you for sharing your solution @FengLiu.