Question about using spatial surrogates

copycky28 · September 18, 2023, 10:08pm

I am using scripts from https://github.com/CMASCenter/Spatial-Allocator/tree/master/pg_srgtools and try to run USA_* files for my own grids. I got problems in using the script for test run in step 4 “Load the shapefiles into tables in the database”. The error is: /Spatial-Allocator//src/libs/gdal-2.0.2/local/bin/ogr2ogr: error while loading shared libraries: libxerces-c-3.1.so: cannot open shared object file: No such file or directory.
Is there anyone know how to solve this issue?

Ryan · September 18, 2023, 10:31pm

I guess this error is attributed to the exe of ogr2ogr which is pre-compiled. In general, you may need to install all libs and re-compile the spatial allocator for use (at least for my case it works after re-compile).

copycky28 · September 18, 2023, 10:45pm

Thanks for your response, Ryan! I changed all libraries to my own and got another error:ERROR: could not form projection (LWPROJ) from ‘srid=4269’ to ‘srid=900921’
Have you ever got this kind of error? Do you know how to deal with it?

eyth.alison · September 19, 2023, 2:20am

What platform are you wanting to recreate the surrogates for and for what region / modeling domain?

james.beidler · September 19, 2023, 12:46pm

900921 is the reference for the Lambert Conformal Conic projection commonly used for SMOKE and CMAQ modeling. Try running step (3) in the README on the pg_srgtools github to add this srid to the database:

github.com

CMASCenter/Spatial-Allocator/blob/master/pg_srgtools/README.md

# How to Run the Postgres Surrogate Tool
Last updated: 7/29/2020

Note that the Postgres Surrogate Tool currently only supports creating surrogates for regular grids. For E-Grid or census track (polygon) surrogates, please use the C/C++ and Java driver version (srgtools).

0. Prerequisites
   - Install the Postgres database server and the PostGIS extension. Make sure the Postgres server is running.
   - Create a new database for the surrogates work. This document uses a database named "surrogates" by default. The following commands can be executed from the `psql` command line, and will create a database named "surrogates", connect to that database, and add the PostGIS extension.
   ```
   CREATE DATABASE surrogates;
   \c surrogates
   CREATE EXTENSION postgis;
   ```
   - For PostGIS version 3 or newer, add the separate PostGIS raster extension.
   ```
   CREATE EXTENSION postgis_raster;
   ```

   - For the surrogate tool to run, you will need to have a database user with all privileges on the new database. The following `psql` commands create a new user named "pgsurg" and assign the appropriate privileges.
   ```

This file has been truncated. show original

copycky28 · September 19, 2023, 3:32pm

I am using surrogate tool on the linux system. I am runing for a projection other than SMOKE pre-existed USA_* files so I might need to run this to generate the input for SMOKE.

copycky28 · September 19, 2023, 3:44pm

Thanks for your suggestion. I have run the step 3 and it shows “INSERT 0 1” on my screen, once I run step 4, the error became:
proj_create: cannot build geodeticCRS 4269: SQLite error on SELECT name, ellipsoid_auth_name, ellipsoid_code, prime_meridian_auth_name, prime_meridian_code, area_of_use_auth_name, area_of_use_code, publication_date, deprecated FROM geodetic_datum WHERE auth_name = ? AND code = ?: no such column: publication_date

Do you know how to solve this issue?

copycky28 · September 19, 2023, 5:34pm

I have a quick question. If I want to generate USA_FILL* files for my own grid using spatial surrogates script, what should I change in the script?
I list the file that should changed here to make sure I don’t miss anything:

proj,lat_1,lat_2,lat_0,lon_0 and others in create_900921.sql
xorig, yorig, xcellsize, ycellsize, cols and rows in “generate_modeling_grid.sh”

I have my own GRIDDESC file from MCIP run and I am trying to match those USA_FILL_* file with the same grid setting in GRIDDESC so that I can use them in SMOKE simulation

james.beidler · September 19, 2023, 5:56pm

I have not seen that issue but it looks like a library mismatch. Can you please share which versions of libproj and GDAL you are using?

copycky28 · September 19, 2023, 7:33pm

Thanks for your response. I reinstalled all software and libraries and re-ran all the steps, now the test run is done successfully, thanks for your help. I am curious how should I use the same script to generate the USA_* files for my own grids? I have grids that has different projection than the default inputs so I have to re-generate them for further running SMOKE.

eyth.alison · September 19, 2023, 8:24pm

It would be helpful to know which EPA platform you are trying to reproduce surrogates for – 2016, 2017, some other?

Also are you using the original Spatial Allocator or the SurrogateToolsDB?

FYI, there are examples for the 2017 platform on the Surrogate Tool github:

https://github.com/CEMPD/SurrogateToolsDB

“run_srgtool_fill.csh” and “surrogate_specification_pg.2017.csv” are the most relevant to gapfilling.

copycky28 · September 19, 2023, 8:32pm

Thanks for your response, Alison! I am generating the files for 2017 platform, I have successfully generate the USA_* files with matching header with my own GRIDDESC and I will test if it works on SMOKE. Thanks again for your help.

copycky28 · September 22, 2023, 10:54pm

Thanks for response. I am using 2017 platform. But I notice that the compressed input shape file “PG_SurrogateTool_scripts_files.2017v1.31Jul2020.tar.gz” not includes all needed input when I trying to use load_shapefile_reproject_multi.2017.csh to load all data to generate all USA_* files. Do you you know where can I get completed dataset?

eyth.alison · September 23, 2023, 2:21am

Which files appear to be missing from the zip file? I believe some were added as separate zipped files to the same GoogleDrive area as the large file.

copycky28 · September 24, 2023, 9:16pm

Thanks for your response Alison. I am trying to generate all needed files for SMOKE simulation and once I
“runload_shapefile_reproject_multi.2017.csh” there are a lot files seems missing. I got some example error message here:
Unable to open datasource `/spatial_allocator/SurrogateToolsDB-1.3/data/emiss_shp2017/US/tl2019Counties_LCC_WRF.shp’ with the following drivers.

Unable to open datasource `/spatial_allocator/SurrogateToolsDB-1.3/data/emiss_shp2017/US/acs2016_5yr_bg.shp’ with the following drivers.

Unable to open datasource `/spatial_allocator/SurrogateToolsDB-1.3/data/emiss_shp2017/NCES/public_schools_2018_2019.shp’ with the following drivers.

Do you know where I can download the full inputs for generate all USA_* files for SMOKE?

cseppan · September 25, 2023, 3:45pm

From the Google Drive location linked in the section “Download additional shapefiles”, click on the folder named “NEI 2017/2016v2 Platforms”. You’ll see archives for two of the missing shapefiles:

public_schools_18_19_shp.zip
tl2019Counties_LCC_WRF.zip

For the acs2016_5yr_bg shapefile, there was an error in the utility script load_shapefile_reproject_multi.2017.csh where the input directory didn’t get reset properly. I’ve updated the GitHub repository with the fixed version:

https://raw.githubusercontent.com/CEMPD/SurrogateToolsDB/master/util/load_shapefile_reproject_multi.2017.csh

copycky28 · September 25, 2023, 4:15pm

Thanks for your responses. There are some other errors that show up when running the updated load_shapefile_reproject_multi.2017.csh.
When loading fema_bsf_2002bnd, the error is:
ERROR 1: PROJ: proj_as_proj_string: Unsupported conversion method: Lambert_Conformal_Conic
ERROR: canceling autovacuum task
CONTEXT: while scanning block 13324 of relation “public.fema_bsf_2002bnd”
automatic vacuum of table “surrogates.public.fema_bsf_2002bnd”
NOTICE: index “fema_bsf_2002bnd_geom_900921_idx” does not exist, skipping
ERROR: ST_Transform: Input geometry has unknown (0) SRID
STATEMENT: UPDATE public.fema_bsf_2002bnd SET geom_900921 = ST_Multi(ST_Transform(wkb_geometry, 900921));
ERROR: ST_Transform: Input geometry has unknown (0) SRID

Do you know how to deal with it?

Best,
K

cseppan · September 25, 2023, 5:29pm

I have run into this specific problem when loading the FEMA shapefile with certain versions of the GDAL library. An edit to the projection file for the shapefile seems to take care of it. I’ve documented the specific change in this GitHub issue:

github.com/CEMPD/SurrogateToolsDB

error loading FEMA shapefile with GDAL 3.0.4

opened 05:26PM - 25 Sep 23 UTC

cseppan

With the original fema_bsf_2002bnd.prj file, running ogr2ogr fails: > ogr2ogr… -f "PostgreSQL" "PG:dbname=surrogates user=pgsurg host=localhost" data/emiss_shp2017/FEMA/fema_bsf_2002bnd.shp -lco PRECISION=NO -nlt PROMOTE_TO_MULTI -nln public.fema_test -overwrite ERROR 1: PROJ: proj_as_proj_string: Unsupported conversion method: Lambert_Conformal_Conic > ogr2ogr --version GDAL 3.0.4, released 2020/01/28 Changing the DATUM name in the fema_bsf_2002bnd.prj file from "Sphere" to "D_Sphere" seems to fix the problem: PROJCS["Lambert_Conformal_Conic",GEOGCS["WRF_Sphere",DATUM["**Sphere**",SPHEROID[... PROJCS["Lambert_Conformal_Conic",GEOGCS["WRF_Sphere",DATUM["**D_Sphere**",SPHEROID[...

copycky28 · September 25, 2023, 6:15pm

Thanks for your help, it works! There are other errors come up when loading other tables:
1.
SPUD_OIL_CONUS_2016
FAILURE: Unable to open datasource `/SurrogateToolsDB 1.3/data/emiss_shp2017/OilGas_2016/SPUD_OIL_CONUS_2016.shp’ with the following drivers.

TOTAL_GAS_PRODUCTION_CONUS_2016
FAILURE:
Unable to open datasource `/SurrogateToolsDB-1.3/data/emiss_shp2017/OilGas_2016/TOTAL_GAS_PRODUCTION_CONUS_2016.shp’

I noticed there is no folder named OilGas_2016 and the files in this folder. Do you know where can I download it?

copycky28 · September 25, 2023, 6:27pm

Never mind, I found it in the NEI 2017/2016v2 Platforms - Google Drive

Topic		Replies	Views
Problem in finding data for spatial surrogate SMOKE	6	243	August 11, 2023
Error in pg_srgtools Spatial Allocator	4	484	December 10, 2020
Failed in using Surrogate Tool of Spatial-Allocator Spatial Allocator	12	1105	May 2, 2019
Error in Spatial-Allocator Spatial Allocator	8	1149	August 12, 2021
Error running Spatial allocator Spatial Allocator	2	101	February 14, 2024

Question about using spatial surrogates

Related Topics