EXTRACT emissions from EDGAR_HTAPv3 with limited geographical domain (South America)

Jano · January 8, 2024, 3:26am

Hi there,
I was wondering if I can process with SMOKE my little domain (São Paulo) from EDGAR_HTAPv3 emission files that covers only a big geographical domain (e.g. South America). I was tried to run but unfortunately I got the log message related to smkinven:

     Value for WKDAY_NORMALIZE:  N returning FALSE
     NOTE: Setting inventory to use full-week normalizer for weekly profiles
     Bad NDIMS for  "Transport" in "TMPFILE"
     Could not read "Transport" from gridded inventory file
     Bad NDIMS for  "Transport" in "TMPFILE"
     Could not read "Transport" from gridded inventory file
     Bad NDIMS for  "Transport" in "TMPFILE"
     Could not read "Transport" from gridded inventory file
     Bad NDIMS for  "Transport" in "TMPFILE"
     Could not read "Transport" from gridded inventory file
     Bad NDIMS for  "Transport" in "TMPFILE"
     Could not read "Transport" from gridded inventory file
     Bad NDIMS for  "Transport" in "TMPFILE"
     Could not read "Transport" from gridded inventory file
     Bad NDIMS for  "Transport" in "TMPFILE"
     Could not read "Transport" from gridded inventory file
     Bad NDIMS for  "Transport" in "TMPFILE"
     Could not read "Transport" from gridded inventory file
     Bad NDIMS for  "Transport" in "TMPFILE"
     Could not read "Transport" from gridded inventory file

I don’t know if it is mandatory to use all dimensions of EDGAR-HTAPv3, I mean 1800 rows, 3600 cols (all the world and not only South American). Do you have any suggestions?

Best regards,
Alejandro

eyth.alison · January 10, 2024, 12:30am

It may be that the gridded inventory list input (EMISLST) is not set up correctly.

The error seems to indicate that SMOKE is not parsing the gridded inventories properly, as if there is an incorrect path, or the variable name in the EMISLST file does not match what is in the gridded inventory.

The EMISLST input for the road transport sector should look something like this:

#LIST GRID

#SCC, Pollutant, Variable_Name, Month, File_location_name

“TRANSPORT”,“PEC”,“Transport_Road”,6,“/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-06_BC.nc”

“TRANSPORT”,“CO”,“Transport_Road”,6,“/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-06_CO.nc”

“TRANSPORT”,“NH3”,“Transport_Road”,6,“/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-06_NH3.nc”

“TRANSPORT”,“VOC”,“Transport_Road”,6,“/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-06_NMVOC.nc”

“TRANSPORT”,“NOX”,“Transport_Road”,6,“/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-06_NOx.nc”

“TRANSPORT”,“POC”,“Transport_Road”,6,“/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-06_OC.nc”

“TRANSPORT”,“PM10”,“Transport_Road”,6,“/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-06_PM10.nc”

“TRANSPORT”,“PM2_5”,“Transport_Road”,6,“/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-06_PM2.5.nc”

“TRANSPORT”,“PM2_5_OTH”,“Transport_Road”,6,“/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-06_PM2.5_OTHER.nc”

“TRANSPORT”,“SO2”,“Transport_Road”,6,“/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-06_SO2.nc”

Jano · January 10, 2024, 1:07am

Hi Eyth. Thank you for your answer and support.

I understand that the problem was in an additional dimension in my EDGAR-HTAPv3 files (time, see below), because when I ran the SMOKE yesterday with two dimensions (lat, lon) and for all the Globe (1800, 3600) , I got “results” but with a wrong location of emissions.

netcdf edgar_HTAPv3_2018_PM2.5 {
dimensions:
        time = 12 ;
        lat = 230 ;
        lon = 350 ;
variables:
        float Transport(time, lat, lon) ;
                Transport:_FillValue = 9.96921e+36f ;
                Transport:description = "Road_Transport" ;
                Transport:grid_mapping = "spatial_ref" ;
                Transport:long_name = "Road Transport" ;
                Transport:units = "kg m-2 s-1" ;
        float Industry(time, lat, lon) ;
                Industry:_FillValue = 9.96921e+36f ;
                Industry:description = "Industry" ;
                Industry:grid_mapping = "spatial_ref" ;
                Industry:long_name = "Industry" ;
                Industry:units = "kg m-2 s-1" ;

On the other hand, I was wondering if I can use a small domain of EDGAR-HTAP that covers only South America as an input to SMOKE or I must create a GRIDMASK_EDGAR.ncf that covers South America as well as geocodes files?

As I was saying, a problem that I’m experiencing is emission outputs from SMOKE. I saw that emission locations are not consistent with my domain (Southeastern Brazil, center in São Paulo). I don’t know if the GRIDDESC for São Paulo domain that was processed with MCIPv5.4 has errors or maybe the IOAPI_ISPH=20 in my ASSIGN file. I attached my files if you want to see my two scripts (ASSIGN and sms_edgar_TRANSPORT_SEBRA.csh).
ASSIGNS.EDGAR.TRANSPORT.SEBRA.csh (17.9 KB)

' '                                                                             
'LAM_28S55W'                                                                    
  2       -60.000       -30.000       -55.000       -55.000       -45.000          
' '                                                                             
'2018_SEBRA'                                                                                                     
'LAM_28S55W'    -40499.000   1813357.500      9000.000      9000.000  247  177    1
' '

smk_edgar_TRANSPORT_SEBRA.csh (6.7 KB)

Best regards,
Alejandro

Jano · January 10, 2024, 12:03pm

Hi Eyth,
SMOKE is an interesting tool, but I thought that it could process pre-gridded emissions based on lat lon coordinates like the “anthro_emis” tool for WRF-Chem. Yesterday, I ran the SMOKE model with the EDGAR-HTAPv2 (the base year 2010), and I got consistent locations of the emissions according to my domain, and that is because the map of EDGAR-HTAPv2 starts with a time zone 0 in Africa that is different compared with the EDGAR-HTAPv3. So, the problem was not the GRIDDESC.

This is the EDGAR-HTAPv2:

This is the EDGAR-HTAPv3:

BTW, is it possible that you could share us the geocodes and the GRIDMASK_EDGAR.ncf adapted to the EDGAR-HTAPv3 order?

Best regards,
Alejandro

eyth.alison · January 12, 2024, 4:24pm

Thank you for pointing this out. We will soon be starting some work with HTAPv3 but we have not yet.

We will be providing some information on how to process HTAPv3 data in the next couple of months.

If I learn anything new before then, I will let you know.

eyth.alison · January 12, 2024, 5:14pm

Also, we note the variables are organized a bit differently in the HTAPv3 gridded inventories. Instead of a single variable called “Transport” there is now “Transport_Road”, “Transport_BWTW”, “Transport_Other”, etc. You will need to update the EMISLST input accordingly. Please let us know if you are able to get it to work.

Jano · January 12, 2024, 7:20pm

Alison, thank you for your answer and support. I understand that previously to list in the EMISLST, I must change variable names in each netCDF with no more than 16 character. I can do that with nco and cdo operators using a script. The only thing that I couldn’t do is to create a new GRIDMASK_EDGAR_htapv3.ncf. I’ve read about it how can I do. I will return with you in case I will create one to process EDGA-HTAPv3.

Best regards,
Alejandro

Jano · February 26, 2024, 2:36pm

Hi Alison, we resolved this issue using the cdo operators. So, we converted from longitudes (-180,180) from EDGAR-HTAPv3 to 0,360 longitudes. The line code in cdo is:

cdo sellonlatbox,0,360,-90,90 $input_file temp.nc

Another solution is with Python. I considered in the script slicing the EDGAR-HTAPv3 into two files:

sector = # add sector name with less than 10 character
range1 = slice(0,180)
range2 = slice(180,0)

v3_1 = v3.sel(lon=range1)
v3_2 = v3.sel(lon=range2)
v3_2['lon'] = v3_2.lon.values + 360
v3_new = xr.concat([v3_1[sector],v3_2[sector], dim='lon')

In SMOKE I run successfully with the year emission files. However, If I want to process monthly emission files in the SMOKE for the temporal step, it will stop with errors. I noted an interesting message in log files that could be related to the running issue:

SMKINVEN_MONTH not defined returning default 0

I understand that I have to set up in the SMK run the SMKINVEN_MONTH = 5 that represents my month (May), is that correct?

Best regards,
Alejandro

eyth.alison · February 27, 2024, 1:49am

SMKINVEN_MONTH should be set to 5 if your EDGAR-HTAPv3 files are monthly.

We aren’t sure about the starting point for your scripts. If you are using scripts from one of our modeling platforms, then the following applies:

We have these two helper scripts which are called by the top-level SMOKE script (along with “3D” versions of each of these for sectors with plume rise):

smk_ar_gridded_annual_emf.csh

smk_ar_gridded_monthly_emf.csh

When using the monthly_emf.csh script, SMKINVEN_MONTH is set to the appropriate month automatically. So if you are using scripts based on one of our EMPs, they should switch from the annual_emf.csh script to the monthly_emf.csh script (near the bottom of the top-level SMOKE script that they run directly) and that will take care of SMKINVEN_MONTH.

Heomin · December 7, 2024, 11:18pm

If I set SMKINVEN_MONTH, will SMOKE be able to directly process monthly data with dimensions (12, 1800, 3600)?

eyth.alison · December 9, 2024, 3:14pm

The raw HTAPv3 gridded files are structured so that all months are in a single yearly file with 12 timesteps. However when processing this data in SMOKE we typically split the months into separate files, and then list each monthly file separately in the Gridded List Inventory input (EMISLST), with the 4^th column pertaining to the month number. See the below example. We hope that helps.

#LIST GRID

#SCC, Pollutant, Variable_Name, Month, File_location_name

“ENERGY”,“PEC”,“Energy”,1,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-01_BC.nc”

“ENERGY”,“PEC”,“Energy”,2,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-02_BC.nc”

“ENERGY”,“PEC”,“Energy”,3,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-03_BC.nc”

“ENERGY”,“PEC”,“Energy”,4,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-04_BC.nc”

“ENERGY”,“PEC”,“Energy”,5,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-05_BC.nc”

“ENERGY”,“PEC”,“Energy”,6,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-06_BC.nc”

“ENERGY”,“PEC”,“Energy”,7,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-07_BC.nc”

“ENERGY”,“PEC”,“Energy”,8,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-08_BC.nc”

“ENERGY”,“PEC”,“Energy”,9,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-09_BC.nc”

“ENERGY”,“PEC”,“Energy”,10,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-10_BC.nc”

“ENERGY”,“PEC”,“Energy”,11,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-11_BC.nc”

“ENERGY”,“PEC”,“Energy”,12,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-12_BC.nc”

“ENERGY”,“CO”,“Energy”,1,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-01_CO.nc”

“ENERGY”,“CO”,“Energy”,2,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-02_CO.nc”

“ENERGY”,“CO”,“Energy”,3,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-03_CO.nc”

“ENERGY”,“CO”,“Energy”,4,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-04_CO.nc”

“ENERGY”,“CO”,“Energy”,5,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-05_CO.nc”

“ENERGY”,“CO”,“Energy”,6,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-06_CO.nc”

“ENERGY”,“CO”,“Energy”,7,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-07_CO.nc”

“ENERGY”,“CO”,“Energy”,8,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-08_CO.nc”

“ENERGY”,“CO”,“Energy”,9,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-09_CO.nc”

“ENERGY”,“CO”,“Energy”,10,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-10_CO.nc”

“ENERGY”,“CO”,“Energy”,11,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-11_CO.nc”

“ENERGY”,“CO”,“Energy”,12,“/work/MOD3DEV/ktalgo/WR713/work/inputs/HTAPv3/data/fluxes/monthly/edgar_HTAPv3_2018-12_CO.nc”

Heomin · December 20, 2024, 6:38am

Hi Jano

I have a question. The original EDGAR file has data with dimensions (1800, 3600). If I crop this data to match the modeling domain (e.g., (350, 550)), can it still be used to run SMOKE?

Jano · December 24, 2024, 1:54am

Hi Heomin,
I tried cropping the data, and the SMOKE couldn’t recognize it. So, the SMOKE only works with global dimensions (1800, 3600).

Best regards,
Jano

yan2024 · December 30, 2024, 3:40pm

Hi Jano,

Thank you for your guide and I followed what you advised by cdo to process HTAPv3 files but I still get some errors. I doubt if it is the reason for the different attribute of variables. Could you please see my latest thread and kindly give me some suggestions?

eyth.alison · December 30, 2024, 5:34pm

Sorry, but we won’t be able to offer more support on this at this time – maybe someone else on the forum has suggestions. Sometime later this winter, we plan to post a package with all needed information to run HTAPv3 through SMOKE, but it likely won’t be for a couple of months.

Jano · January 13, 2025, 9:38pm

Hi Yan,
Sorry for my delay. I have a useful Python script for reducing source names and correcting east coordinates. Those converted files work when I process them in SMOKE v5.0. Please see the file attached.

HTAP_processing.py (6.3 KB)

Best regards,
Jano

yan2024 · January 14, 2025, 2:06am

Hi Jano,

Thank you for your kindly help, I used what you advised with cdo to resolve that problem. I found that it is the issue of netcdf format of HTAP-v3 files. So I transferred them to netcdf3 and it worked within smoke successfully. However, I encountered errors when I runned CMAQ 5.4 after processing htap-v3 files with CB6-AE7 MECHS gspro/gsref. I am wondering if you could provide suggesstions if you had runned successfully by CMAQ with htap-v3 inventory. Could you help me check my error logs that are in my latest thread?

Best Regards,
Yan

Heomin · January 16, 2025, 1:33pm

Hi Jano,

Thank you for the code you provided. However, I have a question.

HTAPv3 files are provided on a monthly basis (12,1800,3600).

In the ARINV file, you set all months to 0. Could you clarify if you merged all the monthly HTAP files into an annual dataset, or if you extracted only the values corresponding to each month and saved them in the LST file?

Best Regards,
Heomin

Jano · January 20, 2025, 7:02pm

Hi Heomin,
I downloaded HTAPv3 files on an annual basis (1800, 3600). After that, I also considered setting 0 in the ARINV file. During the temporal profile, I specify the sector variation that could come from monthly HTPAv3 files or other criteria (sources behavior).

Best regards,
Jano

Jano · January 20, 2025, 7:11pm

Hi Yan,
I also experienced those bad errors under SMOKE running. I mentioned that because gspro/gsref are related with that program. I fixed the error when I deleted output files and premerged files. It is important review if all species name are correct and the path files.

I usually run HTAPv-3 in SMOKE by sector or I sum some sectors to create a new merge sector called “Others”. I processed by each sector for example Industry, Road exhaust, Road resuspension, and Others. After that, I run merge to create files for all time series based on the temporal profile chosen.

Finally, depending on the period, I create other emission files related to sea spray, biogenic, biomass burning.

Best regards,
Jano

Topic		Replies	Views
SMOKE using HTAPv3 emission SMOKE	12	89	December 30, 2024
Smoke v4.6 inputfiles, GRIDMASK file and Layalloc Emissions Inventory smoke	3	993	January 9, 2020
Questions for SMOKE EDGAR-HTAP (flux,emission) SMOKE	3	34	January 9, 2025
GRIDMASK file for smoke SMOKE smoke	4	941	July 14, 2020
SMOKE v4.8 process MEIC inventory, using EDGAR program Emissions Inventory	8	1899	November 18, 2021

EXTRACT emissions from EDGAR_HTAPv3 with limited geographical domain (South America)

Related topics