AMET - AQ project segmentation fault only with CSN data

AMET
#1

Hi AMET developers and users,

Below is the issue I am trying to solve with AMETv1.4.

Segmentation Fault with AMET v1.4 aqProject_post_only.csh, happening with CSN data ONLY while database of other networks were populated successfully. This Error message happened with both intel and pgi compiler.

Below if the Error I copied from the log file.
You comments are highly appreciated.

…
Value for IN_TABLE: ‘/uufs/chpc.utah.edu/common/home/u6002243/AMET_v1.4_pgi/AMET/obs/AQ/2011/CSN_data_2011_VIEWS.csv’
Value for IN_TABLE: ‘/uufs/chpc.utah.edu/common/home/u6002243/AMET_v1.4_pgi/AMET/obs/AQ/2011/CSN_data_2011_VIEWS.csv’
Value for SITE_FILE: ‘/uufs/chpc.utah.edu/common/home/u6002243/AMET_v1.4_pgi/AMET/obs/AQ/site_metadata_files/AQS_full_site_list.csv’
Value for SITE_FILE: ‘/uufs/chpc.utah.edu/common/home/u6002243/AMET_v1.4_pgi/AMET/obs/AQ/site_metadata_files/AQS_full_site_list.csv’
Value for OUT_TABLE: ‘/uufs/chpc.utah.edu/common/home/u6002243/AMET_v1.4_pgi/AMET/output/aqARMS17_Feb/sitex_output/CSN_aqARMS17_Feb.csv’
Value for OUT_TABLE: ‘/uufs/chpc.utah.edu/common/home/u6002243/AMET_v1.4_pgi/AMET/output/aqARMS17_Feb/sitex_output/CSN_aqARMS17_Feb.csv’
Value for START_DATE: 2011032
Value for START_TIME: 0
Value for END_DATE: 2011059
Value for END_TIME: 230000
Value for TIME_SHIFT: 0
Value for APPLY_DLS: N returning FALSE
Value for PRECIP: ‘RT’
Value for PRECIP: ‘RT’
Value for MISSING: ‘-999’
Value for MISSING: ‘-999’
Segmentation fault
Finished running site compare for network CSN

Loading required package: RMySQL
Loading required package: DBI

Project_ID: aqARMS17_Feb
Network: CSN
Sitex File: /uufs/chpc.utah.edu/common/home/u6002243/AMET_v1.4_pgi/AMET/output/aqARMS17_Feb/sitex_output/CSN_aqARMS17_Feb.csv
MySQL Server: mysql.chpc.utah.edu
Error in file(file, “rt”) : cannot open the connection
Calls: read.csv -> read.table -> file
In addition: Warning message:
In file(file, “rt”) :
cannot open file ‘/uufs/chpc.utah.edu/common/home/u6002243/AMET_v1.4_pgi/AMET/output/aqARMS17_Feb/sitex_output/CSN_aqARMS17_Feb.csv’: No such file or directory
Execution halted
Finshed populating database for network CSN

#2

Hi Trang,

The segmentation fault is indicative of an error with running site compare. It’s not immediately obvious to me what’s causing specifically only the CSN processing to fail. Let me get back to you with some suggestions of things for you to try to figure out what’s going on. I need to consult with another developer here first though.

Wyat

#3

Thanks for your response, Wyat. I am using the most up-to-date AMET Version 1.4. Just downloaded it in December. And I am doing evaluation for year 2011. I tried the data populated run with three different CSN input files:

CSN_data_2011.cvs
CSN_data_2011_VIEWS.csv
and CSN_data_2011_VIEWS_new.csv

The first two produced seg fault error. The third one produced error with mismatched variable name in file header. But I am not sure if CSN_data_2011_VIEWS_new.csv is the right input for CSN observation, though. I can’t find any document noting which file should be best to use for CSN network, especially for year 2011. Do you have any idea on what purpose CSN_data_2011_VIEWS_new.csv serve? Thanks!

Trang

#4

Yeah, the CSN is a bit a of mess due to the varying sources I’ve used to obtain the data. But for 2011, the CSN_data_2011_VIEWS.csv file is the correct one to use.

One cause of segmentation faults in the past has been an unsorted site file. However, the CSN uses the AQS site file, which was used successfully for processing AQS, so I don’t think that’s the issue.

We may need you to compile site compare with debug options so we can see exactly where the segmentation fault is occurring, then we can backtrack the problem from there.

I’ll get back to you on that.

Wyat

Wyat

#5

Also, the CSN_data_2011_VIEWS.csv was a development file. It really should not have been included in the distributed obs data for AMET. That was an oversight on my part. Sorry that it caused some confusion.

Wyat

#6

Yes, the regular AQS database was paired with modeled_data just fine. I got all network, obs-mod pairs looked reasonable, except CSN seg fault error. Maybe this is only be the case for year 2011?

#7

CSN_data_2011_VIEWS.gz.csv (1.6 MB) Hi Trang,

Perhaps the issue is that the CSN file itself is not sorted. Please try using the attached CSN_data_2011_VIEWS.csv file and see if that solves the issue. Note that the file I’ve attached is actually a gzip file. So, after you download it you’ll need to rename from .gz.csv to .csv.gz and then unzip it.

Once you do that try using the new file and let me know the result. Thanks.

Wyat

#8

Yay the program passed that Error and I got CSN data paired with modeled data now. Thank you very much Wyat!

#9

Wyat,

By the way, in the $AMETBASE/scripts_db/input_files/AQ_species_list.input I found:

if (year > 2010) {

CSN_OC_SPEC <- “88370_val-blank”: I don’t see any “88370_val-blank” for CSN obs_data in 2011, I only see “88370_val”, so I made edit as below. Am I right doing this?

CSN_OC_SPEC <- “88370_val” ## edited by Trang Tran. check CSN_data_2011_VIEWS.csv
CSN_EC_SPEC <- “88380_val”
CSN_PM_FRM_SPEC <- “88502_val”

#10

Hi Trang,

Glad that fix worked. I’ll be doing a full update of the AMET obs files sometime soon to correct for issues such as this. Thanks for confirmed the fix.

You don’t actually have to make that change to column name. The 88370_val-blank is actually a combination of two columns, namely 88370_val and blank. The 88370_val-blank is just the equation that gets filled in use the R paste command. The script should work with CSN_OC_SPEC set to 88370_val-blank. If it does not, let me know. Thanks.

Wyat

#11

Thank you for your confirmation, Wyatt. I changed that option back to your original setup for not missing any possible data.
“88370_val-blank”
The script worked!

Thanks,
Trang