hi,friends
i run the edgar from the testcase.my problems:
1.something defined in the assign file not exist in the folder,like REPCONFIG MET_CRO_2D MET_DOT_3D,do we not need them?
2.in case of lacking memory,i changed:(1)ncols 18 ,nrows 18 in GRIDDESC ,SRGDESC,do i need to change GRID_CRO?(2)leave one file in arinv.edgar.lst,(3)change assign as:
setenv G_STDATE 2005709 # Julian start date
setenv G_STTIME 000000 # start time (HHMMSS)
setenv G_TSTEP 60000 # time step (HHMMSS)
setenv G_RUNLEN 250000 # run length (HHMMSS)
setenv ESDATE 20050709 # Start date of emis time-based files/dirs
setenv MSDATE 20050709 # Start date of met time-based files
setenv NDAYS 1 # Duration in days of each emissions file
setenv MDAYS 1 # Duration in days of met time-based files
setenv YEAR 2005 # Base year for year-specific files
what i did,is anything wrong? is that helpful to reduce the memory need or not? anything else like Temporal,gridmask, speciation,cut inventory half ?
error i got in smkinv:
File “ARINV” opened for input on unit: 97
/usr/local/smoke/data/inventory/nei2005/edgar/arinv.edgar.lst
Value for WKDAY_NORMALIZE: N returning FALSE
NOTE: Setting inventory to use full-week normalizer for weekly profiles
*** ERROR ABORT in subroutine SMKINVEN:CHECKMEM
Failure allocating memory for “CEXTORL”: STATUS= 5014
i searched the forum found its about memory,is it? unlimted didnt work.
Thanks for your time.
Sorry for the delay on our response but can you give me an update whether you still have this issue or not?
Hi.
I have recieved the same error while run EDGAR in smoke.
***ERROR ABORT in subroutine SMKINVEN:CHECKMEM
Failure allocating memory for “CEXTORL”: STATUS= 5014
I have split those EDGAR inventories by sectors. And I added limit stacksize unlimited in my .cshrc .
Here is my log filesmkinven.edgar.BRW.log.txt (5.2 KB)
BTW, can SMOKE use the monthly data provided by EDGAR ? Should I just write the file path in the arinv.edgar.lst, or do I need to change somthing in ASSIGN file as well ?
Thanks a lot.
What compiler and versions? These status-numbers are compiler dependent.
Assuming gfortran, that probably means you are trying to allocate an array too large for your current executable’s memory model – see https://cjcoats.github.io/ioapi/AVAIL.html#medium
It may work to re-compile smkinven with the medium memory model (BIN=Linux2_x86_64gfort_medium, Linux2_x86_64ifort_medium, or Linux2_x86_64pg_medium) and see if that works. And try this on a machine with lots of RAM and memoryuse unlimited (not just stacksize…)
Before recompiling, try to run it with more RAM memory since it requires lots of memory due to its size of files.
Also, there could be an issue with your arinv.edgar.lst file, and that’s close to where the log stopped. If adding memory doesn’t work please provide the file for review.
Thanks for your reply! @cjcoats @bbaek @eyth.alison
Here are the compiler I use and the versions of the softwares:
gnu/4.8.5
mvapich2/2.3
netcdf/4.3.0
hdf5/1.8.20
zlib/1.2.7
I think the HPC I use needs me to write a PBS job script to read large files and run the program.
However, when I reduced the number of input files, a new problem appeared:
Value for PROMPTFLAG: N returning FALSE Returning default value TRUE for query: "Continue with program?" Value for GRDMAT_ADJUST not defined;returning default: FALSE Value for GRDMAT_LINKDEFS not defined;returning default: FALSE Value for USE_VARIABLE_GRID not defined;returning default: FALSE Value for IMPORT_GRDNETCDF_YN: Y returning TRUE Value for SMK_SOURCE: 'A' Value for SMK_SOURCE: 'A' Value for PROMPTFLAG: N returning FALSEFile “AREA” opened for input on unit: 98
/nuist/scratch/pb02005030/lism/v47_smoke/data/inventory/BRW/area.map.edgar.BRW.txtValue for AREA: '/nuist/scratch/pb02005030/lism/v47_smoke/data/inventory/BRW/area.map.edgar.BRW.txt' Value for AREA: '/nuist/scratch/pb02005030/lism/v47_smoke/data/inventory/BRW/area.map.edgar.BRW.txt' *** ERROR ABORT in subroutine RDINVMAP Corrupted map formatted file - missing lines
Thank you for your time !
Hi, regarding the new error:
We are interested in seeing the Smkinven log, the ARINV .lst file used by Smkinven, and the area.map.edgar.BRW.txt file.
This error is from Grdmat, but the AREA file causing the error is created by Smkinven, so it’s possible something went wrong in Smkinven…
Did you also try running Smkinven with the additional memory?
Hi, thank you for your reply!
It seems that I can’t enlarge the RAM, but I tried to recompile it with -mcmodel=medium in the src/Makefile and still got the same error…
I found that after reducing the files, it appeared such words in foreground (shown below). Maybe this is the real reason why I got the error about AREA file.
And these are my log files
grdmat.edgar.BRW.Pol_71N_BRW.log.txt (3.4 KB) smkinven.edgar.BRW.log.txt (7.5 KB) and list file arinv.edgar.lst.txt (1.8 KB)
I can’t upload the area.map.edgar.BRW beacause it is empty…
Thanks for your time.
Hi.
I think there was a mistake in my previous arinv.lst. After I corrected it, there were no more error from SMKINVEN and GRDMAT.
But this showed up in smkmerge:
"AGMAT" opened as OLD:READ-ONLY File name >"/nuist/scratch/pb02005030/lism/v47_smoke/data/run_ODE/static/agmat.edgar.Pol_71N_BRW.ODE.ncf" File type SMATRX3 Execution ID "????????????????" Grid name "2019_BRW_CROSS" Dimensions: 39420 rows, 72405 cols, 1 lays, 1 vbles NetCDF ID: 196608 opened as READONLY Time-independent data. ASMAT :/nuist/scratch/pb02005030/lism/v47_smoke/data/run_ODE/static/asmat_l.edgar.cmaq_cb05_soa.ODE.ncf >>--->> WARNING in subroutine OPENSET File not available. Could not open file set "ASMAT". ***ERROR ABORT in subroutine OPENMRGIN Could not open file set "ASMAT".
Here is my new log file. grdmat.edgar.BRW.Pol_71N_BRW.log.txt (13.6 KB) smkinven.edgar.BRW.log.txt (27.9 KB) smkmerge.edgar.BRW.20190328.Pol_71N_BRW.log.txt (10.1 KB)
BTW, there are many warnings in my smkinven.edgar.BRW.log …
This is some good news that you have made progress. We note that the Smkmerge error is because the speciation matrix is missing. Assuming the end goal is speciated, hourly, model-ready emissions, you also need to run Spcmat and Temporal. There is a way to bypass those programs if they don’t want speciation or temporalization, via the MRG_SPCMAT_YN and MRG_TEMPORAL_YN settings: https://www.cmascenter.org/smoke/documentation/4.5/html/ch06s14s03.html
Note that EPA has a package posted for running SMOKE with hemispheric emissions including the EDGAR/HTAP inventories. Some of the scripts provided with this package may be helpful for you: ftp://newftp.epa.gov/Air/emismod/2011/hemispheric/
The 2016 hemispheric platform has a TSD posted, but note the data: https://www.epa.gov/air-emissions-modeling/2016-hemispheric-emissions-modeling-platform
Hi, Thank you for your reply!
I tried to set the MRG_SPCMAT_YN and MRG_TEMPORAL_YN to N, and leaved the RUN_SPCMAT to Y. But I got an error from Spcmat (No other subroutine reported any error this time. ) :spcmat.edgar.BRW.cmaq_cb05_soa.log.txt (11.1 KB)
“TMP_POL_FILE” opened as OLD:READ-ONLY
File name “/nuist/scratch/pb02005030/lism/v47_smoke/data/inventory/BRW/edgar.ODE_dat/PMC.ncf”
File type GRDDED3
Execution ID “???”
Grid name “”
Dimensions: 2081258 rows, 1 cols, 1 lays, 7 vbles
NetCDF ID: 131072 opened as READONLY
Time-independent data.
Closing file TMP_POL_FILE
Reading source data from inventory file…
Generating unique lists from inventory data…
Reading speciation cross-reference file…
Number of inital xrefs: 0
Number of COMBO xrefs: 0
Number of FRACS xrefs: 0
ERROR: No valid speciation cross-reference entries!*** ERROR ABORT in subroutine RDSREF Problem reading speciation cross-reference file.
Is it because something wrong with my source classification codes (SCC) in ARINV? I would like to know what sectors are included in SMOKE.
I have looked up the TSD you sent me, and I saw this:
Is that all the sectors listed?
In addition, do I need to change the GSREF and ATREF file?
Thank you so much for your time.
Based on the error message from your Spcmat log file, the following speciation cross-reference input file (GSREF) is missing during its run. Please check the file listed below is existed or not in your SMOKE location.
GSREF: /nuist/scratch/pb02005030/lism/v47_smoke/data/ge_dat/EDGAR/gsref.edgar.txt
Hi
I checked it in my SMOKE dir.
In the spcmat.edgar.BRW.cmaq_cb05_soa.log.txt (11.1 KB) , there is no such error with any other chemical species (CO, NH3, NOx, SO2, PM2.5, PM10, VOC) except PMC.
You need to check what kind of SCCs were used in your Smkinven run. Checkout the intermediate output file (ASCC) from Smkinven (Check Smkinven log file to locate this ASCC output file). This ASCC file contains the unique list of SCCs used in your Smkinven run. You need to make sure you use that SCCs in your GSREF input file to assign a proper specaition profile. In your GSREF file, “TRANSPORT” is one of SCCs used in GSREF input file. Check out the format of GSREF from SMOKE user’s guide.
Hi, thank you for your reply!
I finally run the smk_BRW_edgar.csh successfully! But while I tried to merge the files in output/cmaq_cb05_soa, a new error came out…
"AGTS_L" opened as OLD:READ-ONLY File name "/nuist/scratch/pb02005030/lism/v47_smoke/data/run_ODE/output/cmaq_cb05_soa/agts_l.area.20190328.9.Pol_71N_BRW.ODE.ncf" File type GRDDED3 Execution ID "????????????????" Grid name "2019_BRW_CROSS" Dimensions: 180 rows, 219 cols, 1 lays, 8 vbles NetCDF ID: 65536 opened as READONLY Time-independent data. Value for ADJ_FACS not defined; returning defaultval ': ' ' Value for ADJ_FACS not defined; returning defaultval ': ' ' NOTE : No adjustment factors were applied because there is no ADJ_FACS environment variable defined Value for TAG_SPECIES not defined; returning defaultval ': ' ' Value for TAG_SPECIES not defined; returning defaultval ': ' ' NOTE : No tagging species were available because there is no TAG_SPECIES environment variable defined NOTE: Grid settings initialized using FILE001 in gridded file. ERROR: At least two input files contain data for time periods that do not overlap. No output file can be created. *** ERROR ABORT in subroutine SETOUTDATE Problem with input file time periods
Or is it mean that I do not have to merge them?
Thank you so much for your help all these days!
Have you taken a look at any of our scripts for the 2011 platform that used HTAP data? That may give you some cludes.
Sounds like you are trying to run Mrggrid to merge multiple sectors output files. If the dates from those files are different, you need to set this flag called MRG_DIFF_DAYS to Y so then it can merge multiples gridded hourly emissions output files with different dates.
But please make sure whether merging those different dates make sense and intentional or not.
If you are interested in the complete set of HTAP/EDGAR gridded inventory processing, I would recommend you to follow the suggestion from @eyth.alison.
You need a special GSREF and GSPRO that can work with the EDGAR emissions. You can try pulling some files from 2011 hemispheric package on FTP since that is the most recent one we’ve posted: ftp://newftp.epa.gov/air/emismod/2011/hemispheric/ge_dat_for_2011_hemi_speciation.zip