HADV takes too much time

Hi all again,
I doing some tests for 2 different domains with CMAQ by changing the IBCON data. These are the configurations adopted:

  1. profile mode with default of CMAQ: using the file avprofile_cb6r3m_ae7_kmtbr_hemi2016_v53beta2_m3dry_col051_row068.csv
  2. IBCON provided by GEOS-Chem with resolution of 4x5 degrees
  3. IBCON provided by GEOS-Chem with resolution of 2x2.5 degrees

The simulations with configurations 2 e 3 are fine. They concluded with successful with 3-4 days using 2 nodes (total of 80 tasks).
However the first configuration (with default file of CMAQ) is taking too much time. Is someone knows why that is happening?
The only difference that I am finding in the logs are the HADV time, that i think it is causing the issue.
image

The parameters related to time-step length are equal for all configurations.
setenv CTM_MAXSYNC 720 #> max sync time step (sec) [ default: 720 ]
setenv CTM_MINSYNC 60 #> min sync time step (sec) [ default: 60 ]
setenv SIGMA_SYNC_TOP 0.7 #> top sigma level thru which sync step determined [ default: 0.7 ]
setenv CTM_ADV_CFL 0.75 #> max CFL [ default: 0.75]

Thank you in advance.

Yasmin,
This is a bit strange. Maybe do an ncdump -h on the three BCON files and compare the structures?

Also, please specify the version of CMAQ you are using… is this v5.3.1?

The screen shot you posted indicates a one-minute time step. That is the minimum you are specifying via CTM_MINSYNC. Are the other runs using a different domain and meteorological data sets? If so, the amounts of time required may be very different.

Yes. CMAQ v5.3.1

When I mentioned that the simulations were with different domains, I meant that, when comparing one with their respectives, all the simulations that adopted the IBCON profile are facing the same issue.
Meteorological data are from the same dataset.

If I understand it correctly, only the PROFILE version is not working. My guess is that the file is corrupted or incomplete. Have you QA’d the file? Given that one works and one doesn’t, it would probably highlight a problem if you compared the magnitudes you might find something. If you used aqmbc, then I am guessing you have access to PseudoNetCDF on your system. The command below can be run from the command line and will print the min, mean, and max for each variable for two datasets for easy comparison.

python -ic "
import PseudoNetCDF as pnc

bconpathg = 'BCON_GEOS.nc'
bconpathp = 'BCON_v531_RGMV_profile_20191031'
bconfg = pnc.pncopen(bconpathg, format='ioapi')
bconfp = pnc.pncopen(bconpathp, format='ioapi')
varkeys = getattr(bconfp, 'VAR-LIST').split()
for varkey in varkeys:
    varp = bconfp.variables[varkey]
    valsp = varp[:]
    print(f'PROFILE,{varkey:>16s},{valsp.min():.2e},{valsp.mean():.2e},{valsp.max():.2e},{varp.units}')
    if varkey in bconfg.variables:
        varg = bconfg.variables[varkey]
        valsg = varg[:]
        print(f'GC_BCON,{varkey:>16s},{valsg.min():.2e},{valsg.mean():.2e},{valsg.max():.2e},{varg.units}')
    else:
        print('GCBCON,,,,,')
"

If you don’t have PseudoNetCF, you could run ioapi’s m3stat and comb through the reports.

INFILE=BCON_v531_RMGV_profile_20191031 REPORT=PROFILE_REPORT m3stat INFILE REPORT << EOF




EOF
INFILE=BCON_GEOS.nc REPORT=GEOS_REPORT m3stat INFILE REPORT << EOF





EOF

Either way, you’re looking for a weird value in the PROFILE.

Thank you Barron. Yes, you’re right. The issue appears when I setup the IBCON for profile mode.
I did what you suggested, and the mean values didnt look like weird despite some had different orders of magnitude (the greatest was TOLU, profile = 2.27E-20 and GC = 5.44E-05, but most values had an order of magnitude difference around -1 to -3). Anyway, I commented out some variables of the default file in order to check if the issue is associated with the number of variables since they are different.
By default, I had 248 variables in profile file. By adopting 144 variables, the time-step length of HADV dropped to 16.9 secs. With 109 variables, the time-step length of HADV dropped to 9.7 secs. Thank you all for helping again.

The big question is which species led to the slow down. If you can isolate it, that would be useful for future users.

1 Like

Or else it’s lousy code for “modern” (i.e., post-1990, cache based) processor architectures…

In the AQMBC program that you developed, I used the definition files (gc_airmolden.expr, gc12_to_ae7.expr and gc12_to_cb6r3.expr). In the profile file, I left the species that appeared in the IBCON generated from GC, which means that if the variable appeared in both file, it would used in the default mode. Thus, those variables that only appeared in the profile file I removed out.

Hi Ykaore,

Just let you know that we have developed a solution which is under testing. Please stay tuned.

Cheers,
David

Hi Ykaore,

Please contact me at wong.david-c@epa.gov and I will email you a new piece of code which will be in the next release (don't know the exact release date).

Cheers,
David

1 Like