Spatial Surrogate codes 6831, 6832, and 6833

Spatial Allocator
1

Hi,

I’m running the Spatial Allocator program to make the spatial surrogates needed for SMOKE for a domain other than what EPA has been prepared. I have two problems:

  1. Although I need spatial surrogate codes 6831, 6832, and 6833, I couldn’t find the “surrogate specification” for the above-mentioned codes in the Spatial Allocator software.
  2. I found some spatial surrogates included in the EPA NEI platform, which have been gapfilled from ERG data. How can I gapfill those surrogate files using ERG data?

Your help is much appreciated.

Regards,
Ali

2

Here are the specifications for 6831/6832/6833, including the gapfilling hierarchy for these 2017 platform surrogates. Note that the shapefile names vary from year to year. These are new for 2017 plaftorm and only impact produced water emissions. I’m not sure if those exist in 2016 platform (I can’t recall if you are trying to do 2016 or 2017-based work)

“USA”,“Produced water at CBM wells”,6831,tl2019Counties_LCC_WRF,GEOID,PRODUCED_WATER_CBM_CONUS_2016,ACTIVITY,“Well count - CBM wells”,“Well count - all wells”,“NLCD Open + Low”,“New for 2017 platform”

“USA”,“Produced water at gas wells”,6832,tl2019Counties_LCC_WRF,GEOID,PRODUCED_WATER_GAS_CONUS_2016,ACTIVITY,“Well count - gas wells”,“Well count - all wells”,“NLCD Open + Low”,“New for 2017 platform”

“USA”,“Produced water at oil wells”,6833,tl2019Counties_LCC_WRF,GEOID,PRODUCED_WATER_OIL_CONUS_2016,ACTIVITY,“Well count - oil wells”,“Well count - all wells”,“NLCD Open + Low”,“New for 2017 platform”

3

Thanks.
I’m working on NEI2017. I downloaded the “pg_srgtools” scripts for NEI2017 from google drive (https://drive.google.com/drive/u/0/folders/1tQViUa6b_GAVxgK4hG60ZuWDlSzS-omw). The problem is that I couldn’t find the “/util/” folder within the provided pg_srgtools 2017 platform. Since the “load_shapefile_reproject_multi.csh” and “load_shapefile.csh” are different for 2014 and 2017, would you please send me the “/util/” folder including those script for shapefile2017?

4

Here are two files that might help if they get attached properly.

load_shapefile.2017_csh.txt (9.42 KB)

load_shapefile_reproject_multi.2017_csh.txt (11 KB)

5

Thank you so much.
I could not find “ntad_2017_county_pol” shapefile. Would you mind letting me know where can I find it?

6

At the current time, the ntad_2017_county_pol is not packaged with 2017. It is available as part of the 2016 surrogates as NTAD_2014.

The NTAD_2017_County_Pol table was directly created as a table in the PostgreSQL database based on the NTAD_2014_County_Pol shapefile, which had been loaded as part of the 2016 platform.

The current SurrogateToolsDB GitHub page is based on the 2016 platform, so the util/load_shapefile_reproject_multi.csh script can be used to load the NTAD_2014_County_Pol shapefile. Uncomment “source load_shapefile.csh” at line 57.

The archive pg_srgcreate_2020.tar.gz has the updated scripts for the 2017 platform. In pg_srgcreate/pg_srgtools/CONUS/util/load_shapefile_reproject_multi.csh, uncomment line 92. When load_shapefile.csh runs, it won’t try to read a shapefile from disk, instead it will create a new table from the ntad_2014_county_pol table, with updated FIPS codes for 2017.

There were two FIPS changes from 2014/2016 platforms to 2017:

46113 Shannon, SD à 46102 Oglala Lakota, SD

02270 Wade Hampton, AK à 02158 Kusilvak, AK

You don’t necessarily need to take the above approach – if you have a Shapefile editor you could instead make the two substitutions to the NTAD 2014 Shapefile.

Or you can use the 2014 Shapefile as is and just substitute the FIPS in the output of the surrogate tool.

7

I couldn’t find “tl2019Counties_LCC_WRF” and “PRODUCED_WATER_CBM_CONUS_2016” shapefiles and their shapefile catalog characteristics. Would you please post them here or let me know where I can find them?

8

Check here:

ftp://newftp.epa.gov/Air/emismod/2016/alpha/spatial_surrogates/shapefiles/

or here:

https://drive.google.com/drive/folders/1idGoi6I3GvKFCcf87O_8zMirM7Gtd0_E

9

Thank you for your response. I could find the “ProducedWater_CBM” shapefile using the links you provided. However, I still cannot find the “tl2019Counties_LCC_WRF” shapefile.

10

The tl2019Counties is available from https://www.census.gov/geographies/mapping-files/time-series/geo/tiger-line-file.2019.html

Although that version is not projected to LCC_WRF. This shouldn’t matter if you properly describe the file’s map projection to the surrogate generation tool – either when you load it to Postgres for the PostgreSQL version, or properly set the PROJ.4 string in the Shapefile catalog file.

11

If you haven’t found this file yet based on the previous info, I’ve uploaded the Shapefile to here:

https://gaftp.epa.gov/air/emismod/2016/alpha/spatial_surrogates/shapefiles/tl2019Counties_LCC_WRF.zip

12

It would be helpful, thank you!

13

Hi @eyth.alison

I have a question regarding the new Spatial Allocator tool. I’m running the Postgres SA to make spatial surrogates for an arbitrary domain. The issue is that surrogates 321, 340, and 350 take a huge time to be made (for domain grid size 394*322, it took more than two weeks!). Is it normal? Is it possible to run the tool with multiple processors?
My system’s CPU configuration is Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz.

14

On our computer it takes about two days to run each of those surrogates. You may want to investigate your Postgres configuration. Did you try to run them concurrently or at the same time? (I’m not sure off-hand if the tool kicks off more than one surrogate at a time).

Do you have multiple processors available? I can ask around about how to access additional processors.

15

Once I used the former Spatial Allocator tool, it took 2 days, but now it takes much longer and I have no idea why!
I’m running SA to make surrogates one by one. And, yes, I can use multiple processors if applicable.

16

Hi, It is not a problem to use the old tool if it is working on your computer.

However, here are some postgres configuration updates we made to give PG access to more RAM and some of these may help you:

shared_buffers = 1280MB # min 128kB - the appropriate value depends on your system configuration

# (change requires restart)

#huge_pages = try # on, off, or try

# (change requires restart)

#temp_buffers = 8MB # min 800kB

#max_prepared_transactions = 0 # zero disables the feature

# (change requires restart)

# Caution: it is not advisable to set max_prepared_transactions nonzero unless

# you actively intend to use prepared transactions.

work_mem = 24MB # min 64kB

maintenance_work_mem = 256MB # min 1MB

#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem

#max_stack_depth = 2MB # min 100kB

#shared_memory_type = mmap # the default is the first option

# supported by the operating system:

# mmap

# sysv

We’d be interested to know if you find changing some of these helps improve performance.

We will be doing some tests on our systems over the next couple of weeks.

17

@eyth.alison
I couldn’t find the “gpw-v4-popcount2015-mexico” shapefile on google drive. Could you please upload it?

18

Unfortunately we cannot find this Shapefile. At some point it will be recreated, but probably not until later in the fall.

This is the data source if you want to try to make one yourself:

https://sedac.ciesin.columbia.edu/data/set/gpw-v4-population-count-rev11

20

Hi Alison, I also found that I need these files, do you happen to have these files uploaded to google drive?

21

Apologies but we do not have this Shaepefile – we looked for it. As I mention in the other email it could be recreated.