generateBeld4Data/geo_functions crash with gdal_rasterize

Spatial Allocator
1

I’m running generateBeld4Data through the FEST-C interface. It appears all the input files are read in correctly, but it’s failing in geo_functions, even after I updated it per Raster Tool Error: error while loading shared libraries: libgeos-3.4.2.so - #9 by pkasper
I found a similar error posted ( EPIC Site Information Generation Tool - #3 by ll70 ), but I don’t have any errors related to opening or writing files (log file attached), and don’t think there are any memory issues on my machine (several TB of space available)

Am I wrong that the files were read in correctly? or is this a gdal issue, or a SA installation issue?

I have SA version 4.4 installed on a linux machine.

Thank you!

**tail end of log file:
Computing new raster information…
Raster resolution is: 30.00
Compute new raster info from a shapefile: tmp1654136920_prj.shp
Projected shapefile extent min_xy: (-2824482, 155248) max_xy: (1140506, 5456242)
Modified extent min_xy: (-2824485.000000, 155235.000000) max_xy: (1140525.000000, 5456265.000000)
xcells = 132167 ycells = 176701
Proj4 projection string is: +proj=aea +lat_1=29.5 +lat_2=45.5 +lat_0=23 +lon_0=-96 +x_0=0 +y_0=0 +ellps=WGS84 +units=m +no_defs

Rasterizing projected Shapefile…
Rasterized grid Shapefile is stored in file: tmp1356972685_img.bil
/usr/local/Spatial-Allocator-4.4/src/libs/gdal-2.0.2/local/bin/gdal_translate -ot UInt32 -a_ullr -2824485.000000 5456265.000000 1140525.000000 155235.000000 -scale 0 255 0 0 -outsize 132167 176701 -of EHdr /data/harkey/Spatial-Allocator/data/sat/NLCD2011/new_temp_landcover_2011.img tmp1356972685_img.bil
Successful in creating 0 value domain image for rasterizing.
/usr/local/Spatial-Allocator-4.4/src/libs/gdal-2.0.2/local/bin/gdal_rasterize -a GRIDID -l tmp1654136920_prj tmp1654136920_prj.shp tmp1356972685_img.bil
Error in system call: /usr/local/Spatial-Allocator-4.4/src/libs/gdal-2.0.2/local/bin/gdal_rasterize -a GRIDID -l tmp1654136920_prj tmp1654136920_prj.shp tmp1356972685_img.bil
261.899u 70.482s 5:36.62 98.7% 0+0k 138832+183037112io 97pf+0w
==== Error in Beld4 data generation.

generateBeld4Data_12US1_20260714113423.txt (66.6 KB)

2

Can you tell if the intermediate shapefile “tmp1654136920_prj.shp” was retained? If so please try to run the gdal_rasterize command below outside of the SA to see if you get a more descriptive error.

/usr/local/Spatial-Allocator-4.4/src/libs/gdal-2.0.2/local/bin/gdal_rasterize -a GRIDID -l tmp1654136920_prj tmp1654136920_prj.shp tmp1356972685_img.bil

Is your target grid for BELD4 the 12US1?

3

Yes, the target grid is the 12US1 grid (for this run, I also want to run things for two nested grids), and yes, the temporary shapefile is there. Running the command doesn’t give much information, but it looks like I do get a new/updated tmp1356972685_img.bil file from it:

$ /usr/local/Spatial-Allocator-4.4/src/libs/gdal-2.0.2/local/bin/gdal_rasterize -a GRIDID -l tmp1654136920_prj tmp1654136920_prj.shp tmp1356972685_img.bil
Warning : the output raster dataset and the input vector layer do not have the same SRS.
Results might be incorrect (no on-the-fly reprojection of input data).
0…10…20…30…40…50…60…70…80…90…100 - done.

4

That looks successful, so this may be an error handling issue in the SA. @lizadams may have more insight into the issue.

1km CONUS grid BELD4 tiles are available on the FTP site. These may work for your nested scenarios.

5

thank you, and more insight would greatly be appreciated.
For full context, this error stems from running the first step of FEST-C on the 12US1 grid, with the goals of also running it on nested 4km and 1km grids. I’m not sure if 2014 BELD4 will work for FEST-C (seems like the most recent year it can work with is 2011), but I’ll see if the 1km tiles align with my 1km domain just in case.

6

I am sharing information from google AI:

1. The Output File is Simply Too Large (File Size Limit / RAM Exhaustion)

Look at the -outsize dimensions calculated by Spatial Allocator: 132,167 columns by 176,701 rows.

What the Resource Metrics Tell Us
The resource usage numbers (261.899u 70.482s 5:36.62) reveal exactly what is happening. The process ran for over 5 and a half minutes, consumed massive CPU power, performed a staggering amount of Input/Output operations (183,037,112 blocks written), and then crashed.

This confirms that the error is caused by Disk Space Exhaustion or File Size Violations while generating the .bil raster file.

  • Massive Disk Writes: Your system attempted to write over 183 million blocks of data to the disk.
  • The Crash Point: The tool successfully allocated the empty 0-value image framework, but as gdal_rasterize attempted to unpack and burn the huge GRIDID attributes line-by-line across all 23.3 billion pixels, it filled up the storage volume completely and died mid-write.

The /tmp directory on Linux is often capped at a few gigabytes.

  • The suggested Fix: Create a temporary folder on a hard drive partition that has at least 150 GB to 200 GB of free space. Export it in your terminal session before running the script:

    bash

    mkdir -p /path/to/large_disk/sa_tmp
export TMPDIR=/path/to/large_disk/sa_tmp

    tcsh

    mkdir -p /path/to/large_disk/sa_tmp
setenv TMPDIR /path/to/large_disk/sa_tmp

Please let us know if this helps to resolve your issue.