CCTM suspended suddenly

CMAQ
1

Dear all,

I run the CMAQv5.3.3 with 3km resolution. It is always suspended suddenly at a random time with no error or note. And I have tried many times.
At first, the run_cctm.csh set
setenv CTM_MAXSYNC 720
setenv CTM_MINSYNC 60
Then met the suspended, I modified it to
setenv CTM_MAXSYNC 120
setenv CTM_MINSYNC 60
But it is still suspended.
Here is the LOG_000 file.
CTM_LOG_000.3km_20200113.txt (325.2 KB)
It would be appreciated if you could give me any advice.
Thank you so much!

Best,
Chang

2

It may be difficult to determine what is going wrong here.
When you say the model suspends at “random” times, how variable is it? Do the times vary by hours, or is it approximately the same time step?
If it varies a lot, then that sounds to me like a networking or hardware issue. Are you able to run the benchmark test case?

3

I do see the following warnings in your CTM_000* log file.

    ATTENTION: Attempt to include subgrid cloud effects: photolysis rates           
       include ACM CLOUD effects. It would be prudent to examine diagnostic            
       cloud fractions.
 

          YOU SHOULD VERIFY that the cloud microphysics scheme used
          in the Meteorological Model did not include graupel.  If
          it did, then you need to reprocess the meteorological data
          through MCIP and pass QG to file  MET_CRO_3D  to avoid
          errors in the photolysis simulation.

     Processing will continue with QG set to ZERO.  <<--<<


          YOU SHOULD VERIFY that the cloud microphysics scheme used
          in the Meteorological Model did not include ice/snow.  If
          it did, then you need to reprocess the meteorological data
          through MCIP and pass QI to file  MET_CRO_3D  to avoid
          errors in the photolysis simulation.

     Processing will continue with QI set to ZERO.  <<---<<

Perhaps the WRF inputs that you are using do not provide enough information for the resolved cloud model and this is causing an issue?
The 12km grid resolution that we provide in the benchmark case uses the sub-grid cloud model, and may not encounter or reproduce the error that is occurring here.

4

I checked two log files from two different cctm running.
The one suspended time is
Processing Day/Time [YYYYDDD:HHMMSS]: 2020011:235300.
The other one suspended time is
Processing Day/Time [YYYYDDD:HHMMSS]: 2020013:174600.
The started running time is 12 pm.
That is why I said cctm is suspended at a random time.
Sometimes the cctm could run success (not benchmark case, is our setting simulation), and sometimes suspended. It seems not stable.
For the hardware, do I need to increase the CPU cores?

5

Thanks for your information.
Sorry, I am not familiar with the detailed wrf data.
Thus, do you mean that the wrfout files I used are not good for the mcip because of the sub-grid cloud data? That is the reason why the cctm running is not stable?
I’d better use the other wrf input data (like fnl, gfs) to get a new wrfout, to run cmaq, right?

6

The warning message about missing subgrid cloud information (because a convective scheme was not used) is just a warning. Since you are running at fine (4km) resolution, the lack of a convective scheme is appropriate and does not indicate a problem.

Have you checked all of the “ancillary” log files, which begin with CTM_LOG*? If not, then cd to the directory where the log files exist and give this command:
grep -i abort CTM_LOG*

If that does not turn up anything, then try searching for ‘error’, and if that does not work, use the tail command to look at the last 10-30 lines of each log file.

1 Like