CMAQ 3-d simulation with 12US1: running extremely slow with time-step length (HHMMSS): 000400

Dear all;

I am running a 3-d CMAQ simulation for Jan 1 through 3 with 2014 NEI emissions. It is getting very slow such that each time step length takes about 1 hour. The time integration info is attached below.

================================
|>— TIME INTEGRATION —<|
================================

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:000000
   Which is Equivalent to (UTC): 0:00:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...  445.5 seconds
            COUPLE completed...   14.2 seconds
              HADV completed...  123.2 seconds
              ZADV completed...    9.8 seconds
             HDIFF completed...   20.7 seconds
          DECOUPLE completed...    7.4 seconds
           CLDPROC completed...  156.9 seconds
              CHEM completed...   32.0 seconds
              AERO completed...   61.3 seconds
   Master Time Step Processing Completed in  871.2 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:000400
   Which is Equivalent to (UTC): 0:04:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed... 1046.9 seconds
            COUPLE completed...   23.4 seconds
              HADV completed...  115.8 seconds
              ZADV completed...    5.0 seconds
             HDIFF completed...   41.2 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...  193.3 seconds
              CHEM completed...   46.4 seconds
              AERO completed...  102.4 seconds
   Master Time Step Processing Completed in 1574.9 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:000800
   Which is Equivalent to (UTC): 0:08:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed... 1035.6 seconds
            COUPLE completed...    0.7 seconds
              HADV completed...  228.0 seconds
              ZADV completed...    3.5 seconds
             HDIFF completed...   18.9 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   39.2 seconds
              CHEM completed...   24.5 seconds
              AERO completed...  104.6 seconds
   Master Time Step Processing Completed in 1455.6 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:001200
   Which is Equivalent to (UTC): 0:12:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...  727.8 seconds
            COUPLE completed...    0.2 seconds
              HADV completed...  769.2 seconds
              ZADV completed...    5.2 seconds
             HDIFF completed...   26.6 seconds
          DECOUPLE completed...    0.7 seconds
           CLDPROC completed...  128.5 seconds
              CHEM completed...   50.8 seconds
              AERO completed...  137.9 seconds
   Master Time Step Processing Completed in 1847.0 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:001600
   Which is Equivalent to (UTC): 0:16:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed... 1196.9 seconds
            COUPLE completed...    0.8 seconds
              HADV completed...  160.0 seconds
              ZADV completed...    3.6 seconds
             HDIFF completed...   35.1 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   58.3 seconds
              CHEM completed...   24.8 seconds
              AERO completed...  228.8 seconds
   Master Time Step Processing Completed in 1708.9 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:002000
   Which is Equivalent to (UTC): 0:20:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...  554.8 seconds
            COUPLE completed...    0.2 seconds
              HADV completed...  901.4 seconds
              ZADV completed...    3.6 seconds
             HDIFF completed...   20.4 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   81.2 seconds
              CHEM completed...   29.9 seconds
              AERO completed...  138.7 seconds
   Master Time Step Processing Completed in 1730.7 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:002400
   Which is Equivalent to (UTC): 0:24:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed... 1043.6 seconds
            COUPLE completed...    0.8 seconds
              HADV completed...   53.0 seconds
              ZADV completed...    3.5 seconds
             HDIFF completed...   27.2 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   74.9 seconds
              CHEM completed...   23.9 seconds
              AERO completed...  534.8 seconds
   Master Time Step Processing Completed in 1762.1 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:002800
   Which is Equivalent to (UTC): 0:28:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...   12.0 seconds
            COUPLE completed...    0.2 seconds
              HADV completed...  671.9 seconds
              ZADV completed...    4.0 seconds
             HDIFF completed...   22.9 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   65.8 seconds
              CHEM completed...   26.3 seconds
              AERO completed...   17.3 seconds
   Master Time Step Processing Completed in  821.0 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:003200
   Which is Equivalent to (UTC): 0:32:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed... 1197.9 seconds
            COUPLE completed...    0.5 seconds
              HADV completed...   49.3 seconds
              ZADV completed...    3.8 seconds
             HDIFF completed...   21.0 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   89.4 seconds
              CHEM completed...   24.7 seconds
              AERO completed...  718.3 seconds
   Master Time Step Processing Completed in 2105.7 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:003600
   Which is Equivalent to (UTC): 0:36:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...   23.7 seconds
            COUPLE completed...    0.2 seconds
              HADV completed...  520.6 seconds
              ZADV completed...    5.6 seconds
             HDIFF completed...   15.7 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   50.7 seconds
              CHEM completed...   24.3 seconds
              AERO completed...   17.5 seconds
   Master Time Step Processing Completed in  659.0 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:004000
   Which is Equivalent to (UTC): 0:40:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed... 1184.6 seconds
            COUPLE completed...    0.5 seconds
              HADV completed...   40.0 seconds
              ZADV completed...    3.5 seconds
             HDIFF completed...   22.8 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   97.6 seconds
              CHEM completed...   24.5 seconds
              AERO completed...  741.4 seconds
   Master Time Step Processing Completed in 2115.5 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:004400
   Which is Equivalent to (UTC): 0:44:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...   64.5 seconds
            COUPLE completed...    0.2 seconds
              HADV completed...  351.8 seconds
              ZADV completed...    4.1 seconds
             HDIFF completed...   17.8 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   55.8 seconds
              CHEM completed...   25.1 seconds
              AERO completed...   18.6 seconds
   Master Time Step Processing Completed in  538.5 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:004800
   Which is Equivalent to (UTC): 0:48:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed... 1169.4 seconds
            COUPLE completed...    1.3 seconds
              HADV completed...   92.1 seconds
              ZADV completed...    3.5 seconds
             HDIFF completed...   18.0 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   89.0 seconds
              CHEM completed...   25.0 seconds
              AERO completed...  509.7 seconds
   Master Time Step Processing Completed in 1908.6 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:005200
   Which is Equivalent to (UTC): 0:52:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...  275.4 seconds
            COUPLE completed...    2.1 seconds
              HADV completed...  415.2 seconds
              ZADV completed...    4.2 seconds
             HDIFF completed...   21.2 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   73.2 seconds
              CHEM completed...   23.5 seconds
              AERO completed...   17.9 seconds
   Master Time Step Processing Completed in  833.2 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:005600
   Which is Equivalent to (UTC): 0:56:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed... 1413.2 seconds
            COUPLE completed...    6.3 seconds
              HADV completed...   16.4 seconds
              ZADV completed...    3.4 seconds
             HDIFF completed...   17.0 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   98.7 seconds
              CHEM completed...   31.2 seconds
              AERO completed...  653.9 seconds
   Master Time Step Processing Completed in 2240.8 seconds

 Completed Writing Results to Output Files...**** seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:010000
   Which is Equivalent to (UTC): 1:00:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...  960.5 seconds
            COUPLE completed...    2.1 seconds
              HADV completed...  889.1 seconds
              ZADV completed...    6.6 seconds
             HDIFF completed...   29.7 seconds
          DECOUPLE completed...    6.6 seconds
           CLDPROC completed...  237.4 seconds
              CHEM completed...   44.0 seconds
              AERO completed...  976.9 seconds
   Master Time Step Processing Completed in 3153.0 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:010400
   Which is Equivalent to (UTC): 1:04:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed... 1524.5 seconds
            COUPLE completed...    0.3 seconds
              HADV completed...  227.3 seconds
              ZADV completed...    4.6 seconds
             HDIFF completed...   54.4 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   49.1 seconds
              CHEM completed...   25.6 seconds
              AERO completed...  248.0 seconds
   Master Time Step Processing Completed in 2134.3 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:010800
   Which is Equivalent to (UTC): 1:08:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed... 2191.2 seconds
            COUPLE completed...    0.2 seconds
              HADV completed...   85.9 seconds
              ZADV completed...    3.4 seconds
             HDIFF completed...   37.7 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...  109.0 seconds
              CHEM completed...   24.0 seconds
              AERO completed...  672.0 seconds
   Master Time Step Processing Completed in 3124.2 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:011200
   Which is Equivalent to (UTC): 1:12:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...  179.1 seconds
            COUPLE completed...    0.2 seconds
              HADV completed... 1637.8 seconds
              ZADV completed...    4.2 seconds
             HDIFF completed...   32.1 seconds
          DECOUPLE completed...    0.4 seconds
           CLDPROC completed...   83.0 seconds
              CHEM completed...   22.0 seconds
              AERO completed...  560.2 seconds
   Master Time Step Processing Completed in 2519.0 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:011600
   Which is Equivalent to (UTC): 1:16:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed... 1961.1 seconds
            COUPLE completed...    0.2 seconds
              HADV completed...  158.1 seconds
              ZADV completed...    3.4 seconds
             HDIFF completed...   27.8 seconds
          DECOUPLE completed...    1.5 seconds
           CLDPROC completed...   63.1 seconds
              CHEM completed...   21.5 seconds
              AERO completed...  598.8 seconds
   Master Time Step Processing Completed in 2835.8 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:012000
   Which is Equivalent to (UTC): 1:20:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...  967.6 seconds
            COUPLE completed...    0.3 seconds
              HADV completed... 2316.1 seconds
              ZADV completed...    9.2 seconds
             HDIFF completed...   35.6 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...  142.8 seconds
              CHEM completed...   28.1 seconds
              AERO completed...  802.4 seconds
   Master Time Step Processing Completed in 4302.8 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:012400
   Which is Equivalent to (UTC): 1:24:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...  485.6 seconds
            COUPLE completed...    0.4 seconds
              HADV completed... 2254.8 seconds
              ZADV completed...    3.4 seconds
             HDIFF completed...   40.6 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   89.4 seconds
              CHEM completed...   23.5 seconds
              AERO completed... 1163.2 seconds
   Master Time Step Processing Completed in 4061.4 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:012800
   Which is Equivalent to (UTC): 1:28:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...  948.0 seconds
            COUPLE completed...    0.3 seconds
              HADV completed...  900.8 seconds
              ZADV completed...    3.4 seconds
             HDIFF completed...   22.0 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   99.4 seconds
              CHEM completed...   19.7 seconds
              AERO completed...  560.1 seconds
   Master Time Step Processing Completed in 2554.3 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:013200
   Which is Equivalent to (UTC): 1:32:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...  683.0 seconds
            COUPLE completed...    0.2 seconds
              HADV completed...  889.1 seconds
              ZADV completed...    3.5 seconds
             HDIFF completed...   40.7 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   84.0 seconds
              CHEM completed...   20.1 seconds
              AERO completed...  573.3 seconds
   Master Time Step Processing Completed in 2294.4 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:013600
   Which is Equivalent to (UTC): 1:36:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...  570.3 seconds
            COUPLE completed...    0.7 seconds
              HADV completed... 1443.1 seconds
              ZADV completed...    3.4 seconds
             HDIFF completed...   26.7 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   84.2 seconds
              CHEM completed...   19.1 seconds
              AERO completed...  637.2 seconds
   Master Time Step Processing Completed in 2785.3 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:014000
   Which is Equivalent to (UTC): 1:40:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...  477.2 seconds
            COUPLE completed...    0.2 seconds
              HADV completed... 1968.7 seconds
              ZADV completed...    3.6 seconds
             HDIFF completed...   51.5 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...   82.9 seconds
              CHEM completed...   22.5 seconds
              AERO completed...  652.4 seconds
   Master Time Step Processing Completed in 3259.5 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:014400
   Which is Equivalent to (UTC): 1:44:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...  696.0 seconds
            COUPLE completed...    0.3 seconds
              HADV completed... 1990.9 seconds
              ZADV completed...    3.5 seconds
             HDIFF completed...   31.6 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...  103.4 seconds
              CHEM completed...   19.1 seconds
              AERO completed...  670.7 seconds
   Master Time Step Processing Completed in 3516.9 seconds

 Processing Day/Time [YYYYDDD:HHMMSS]: 2014001:014800
   Which is Equivalent to (UTC): 1:48:00  Wednesday,  Jan. 1, 2014
   Time-Step Length (HHMMSS): 000400
             VDIFF completed...  613.8 seconds
            COUPLE completed...    0.2 seconds
              HADV completed... 2029.4 seconds
              ZADV completed...    3.5 seconds
             HDIFF completed...   31.3 seconds
          DECOUPLE completed...    0.5 seconds
           CLDPROC completed...  115.0 seconds
              CHEM completed...   21.0 seconds
              AERO completed...  700.0 seconds
   Master Time Step Processing Completed in 3514.8 seconds

It does not look right and normal to me. Do you happen to have some idea or previous experience with this issue?

Thank you very much!
Yijia

Hi Yijia,
I believe we determined that this was a memory issue that was caused when you requested too many processors for your job, which divided up the memory per processor to a level that caused the run to continually swap out memory.
Fix was to specify the correct number of processors in the qsub job command on your system.
Liz

2 Likes