Hi, There is a mrggrid program that is used to merge the gridded emissions files together.
See smoke4.7/scripts/emf/smk_sectormerge_emf.csh in ftp://newftp.epa.gov/air/emismod/2016/v1/smoke_2016v1_platform_core_04feb2020.zip
Thanks - I updated the the /2016fh_16j/scripts/directory_definitions.csh script and also updated+ran the /2016fh_16j/scripts/merge/Sector_merge_12US1_2016fh_16j.csh script. My guess is the program is looking for (10day) spinup files of type *.1.tmp.ncf starting from the date 12/22/2015 and not finding them, giving this standard output+error:
***testing*** SRGPRO_PATH set by SRGPRO input: /proj//MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/ge_dat/gridding/surrogates/CONUS12_2014_30apr2019/
Linux2_x86_64ifort
NOTE: Default MONTH setting to 'jan'
/bin/ls: No match.
/bin/ls: No match.
/bin/ls: No match.
/bin/ls: No match.
SCRIPT NOTE: Setting months with no partial months
SCRIPT NOTE: Getting representative merge dates path from MRGDATE_FILES
SCRIPT NOTE: Using spinup dates for month 12 of previous year during merge
SCRIPT NOTE: Excluding sector onroad_ca from mrggrid
SCRIPT NOTE: Excluding sector onroad_nonconus from mrggrid
SCRIPT NOTE: Excluding sector afdust from mrggrid
SCRIPT NOTE: Excluding sector afdust_ak from mrggrid
SCRIPT NOTE: Excluding sector afdust_ak_adj from mrggrid
SCRIPT NOTE: Excluding sector cmv_c1c2_12 from mrggrid
SCRIPT NOTE: Excluding sector cmv_c3_12 from mrggrid
SCRIPT NOTE: Excluding sector othafdust from mrggrid
SCRIPT NOTE: Excluding sector othptdust from mrggrid
SCRIPT NOTE: Excluding sector othpt from mrggrid
SCRIPT NOTE: Excluding sector ptagfire from mrggrid
SCRIPT NOTE: Excluding sector ptagfire3D from mrggrid
SCRIPT NOTE: Excluding sector ptegu from mrggrid
SCRIPT NOTE: Excluding sector ptfire from mrggrid
SCRIPT NOTE: Excluding sector ptfire3D from mrggrid
SCRIPT NOTE: Excluding sector ptfire_nonconus from mrggrid
SCRIPT NOTE: Excluding sector ptfire_nonconus3D from mrggrid
SCRIPT NOTE: Excluding sector ptfire_othna from mrggrid
SCRIPT NOTE: Excluding sector ptfire_othna3D from mrggrid
## Running merge script for 12/22/2015, using data files:
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_onroad_20161222_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_onroad_ca_adj_20161222_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_afdust_adj_20161222_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_ag_20161222_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_beis_20151222_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_nonpt_20161206_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_nonroad_20161206_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_np_oilgas_20161206_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_onroad_can_20161206_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_onroad_mex_20161206_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_othafdust_adj_20161222_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_othptdust_adj_20161222_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_othar_20161206_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_ptnonipm_20161206_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_airports_20161206_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_pt_oilgas_20161206_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_rail_20161206_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_rwc_20161222_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: One or more input files expected by script does not exist.
Should I just copy over 2016-12-22 to 2016-12-31 2D sector emissions files, change their years to 2015, rename them in the format emis_mole_rwc_20161222_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf and place them at the location 2016fh_16j/premerged/mrggrid?
When creating the merged spinup for a 2016 case, the December 2016 emissions are used for all sectors except biogenics, which use December 2015 emissions. The scripts automatically handle this and there is no need to create copies of the December 2016 sector emissions with December 2015 dates. The stdout indicates that the script is looking for the December 2016 emissions (e.g. emis_mole_onroad_can_**20161206**_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf, which should be a link to the file in the original location). So, there must be some other problem.
I would check the following:
- Do December 2016 sector emissions exist in this directory?
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/ - Does the premerged/mrggrid directory exist? This is supposed to be created automatically but in some circumstances is not. If it does not exist, make the directory and try rerunning.
- If premerged/mrggrid does already exist, is it populated with a bunch of broken links, i.e. the .1.tmp.ncf files? If yes, the files and locations that the broken links attempt to point to show where, and with which names, the scripts expect the model-ready emissions to be located. One possibility is that the filenames of the sector emissions are not exactly what the script is expecting.
We hope these hints will help you get it working…
Thanks - that helps - my 2D sector emission files were under input/ and not under premerged/ so I fixed that and ran again, and most Dec spinup emissions files are detected except for four sectors (afdust_adj, beis, pt_oilgas, rwc):
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_afdust_adj_20161222_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_beis_20151222_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_pt_oilgas_20161206_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: data file expected but not found:
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/premerged/mrggrid/emis_mole_rwc_20161222_36US3_cmaq_cb6_2016fh_16j.1.tmp.ncf
SCRIPT ERROR: One or more input files expected by script does not exist.
I had deliberately removed ‘rwc’ sector files because CMAQv5.3.1 expects ‘rwc’ emissions separately. I removed ‘beis’ because I will process it inline in CMAQ. Since I am also calculating dust inline in CMAQ, I won’t feed dust emissions files, and in the files I downloaded, the pt_oilgas is not 2D gridded emissions but rather stacked emissions.
I probably need to turn off these four sectors by myself somewhere in the scripts, but I am not sure where that would be?
Actually, I experimented with removing the four unwanted sectors in the file sectorlist_2016fh_02aug2019_v0 and ran Sector_merge_12US1_2016fh_16j.csh again, and I am now seeing:
Running part 4, for 20151222...
/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/smoke4.7/Linux2_x86_64ifort/mrggrid: error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory
0.000u 0.001s 0:00.08 0.0% 0+0k 11136+0io 2pf+0wIt appears that the mrggrid executable you have is not compatible with your linux server.
It requires a fortran compiler and to be recompiled to create an executable for your system.
Have any of the other programs you tried run OK?
One other suggestion besides recompiling: You may need to install Intel’s OpenMP library. If it is already on your system, try adding it to your path.
Thanks! I haven’t run other precompiled commands under smoke4.7/Linux2_x86_64ifort/, but when I checked a precompiled command under /ioapi/, it seemed to work (e.g. ./greg2jul 20150103
correctly yielded 2015003). I am transferring 2D emissions files to another cluster with Intel OpenMP module in it, so will let you know how the mrggrid program runs in that cluster.
When I add the intel OpenMP module on the new cluster, it seems to successfully launch the mrggrid command and then the m3stat command with some messages/ERROR:
Program version:
$Id:: m3stat.f 121 2015-01-20 22:24:38Z coats $
now checking log file /proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/intermed/mrggrid/m3stat_logs/m3stat_mrggrid_20151222_36US3_cmaq_cb6_2016fh_16j.log
Running domaintotals for 20151222
Traceback (most recent call last):
File "/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/smoke4.7/scripts/emisqa/emisqa.py", line 7, in <module>
import qamods
File "/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/smoke4.7/scripts/emisqa/qamods/__init__.py", line 8, in <module>
from qamods.helpers import *
File "/proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/smoke4.7/scripts/emisqa/qamods/helpers.py", line 2, in <module>
from builtins import str
ImportError: No module named builtins
ERROR: running domain totals for 20151222
log analyzer
Getting message data (might take some time)....
grep: /proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/intermed/mrggrid/logs: Is a directory
grep: /proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/intermed/mrggrid/logs: Is a directory
No warning or error messages. Exiting
log analyzer
Getting message data (might take some time)....
grep: /proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/intermed/mrggrid/logs: Is a directory
grep: /proj/MYLIB_PROJ/epa_ftp_platform/smk_2016v1_platform_core_7may2020/2016fh_16j/intermed/mrggrid/logs: Is a directory
No warning or error messages. Exiting
Not sure what the two errors above mean:
(1) ImportError: No module named builtins
(2) ERROR: running domain totals for 20151222)
There was a strange output file 2016fh_16j/smoke_out/2016fh_16j/36US3/cmaq_cb6/emis_mole_all_20151222_36US3_nobeis_norwc_2016fh_16j.ncf and ncdump on the file showed it had TFLAG for 2015356:000000 to 2015357:000000 with 010000 timesteps
I wouldn’t worry about the log analyzer messages. Did the merged files come out?
Maybe someone else can help as this is beyond my expertise.
From what you describe, emis_mole_all_20151222_36US3_nobeis_norwc_2016fh_16j.ncf, the timesteps sound like the way they should be.
The domain totals is not working due to a python installation issue (missing modules), but that is not a required part of the process.
I was able to get the mrggrid program running after commenting out the loop where calculation of domain total is done in the smk_merge script. Thanks!
OK – great. If you eventually decide you want the domain totals to work, try running with python 3 instead of python 2.
But it’s not needed just to get the merged emissions.
I did run with python 3, but there was some issue I cannot fully remember - (1) either some python module was not there, or (2) the domain total was not supposed to run on spinup but it tried to and crashed.
Hi Liz, I could not find ptegu sector inline emissions for 36US3 domain under CMAS Data Warehouse > SMOKE > Output > 2016Platform > 2016fh_16j > premerged, although the tar files exist for the 12US1 in the same place. Did I miss something there?
Hi, ptegu is processed as all elevated sources. The provided inln point files should work for both 36US3 and 12US applications.
That’s right - thanks!
I was also wondering if the beld/b3grd/bioseason and surface input files in CMAQ for the 36US3 domain could be obtained by just regridding the 12US1 versions?
Hi, The 36US3 versions of biogenic input files are included in this file:
ftp://newftp.epa.gov/air/emismod/2016/v1/ancillary_data/ge_dat_for_2016v1_beis_27sep2019.zip
Since I am studying the effects of climate change on air quality, I am opting to rerun sectors rwc, ag, onroad in the 2016v1 modeling platform to suit my meteorology. I was thinking of running beis and dust inline in CMAQ, but had a confusion about dust. The inline CMAQ windblown dust module only calculates the wind-entrained dust (which of course depends on meteorology) but does not include the mechanical abrasion (vehicle brakes) part of dust, right? If I understand correctly, when I just use inline CMAQ dust module, I will be missing the mechanically produced dust, so is it just better to run afdust sector in the 2016v1 modeling platform and then turn off the dust module in CMAQ?