I’m using data from 2022v1 Emissions Modeling Platform. When I try to run the ptegu category script “Annual_ptegu_summer_daily_12US1_2022hc_cb6_22m.csh”, I get the following errors:
NOTE: Default MONTH setting to ‘jan’
/bin/ls: No match.
/bin/ls: No match.
/bin/ls: No match.
/bin/ls: No match.
SCRIPT NOTE: Setting months with no partial months
SCRIPT NOTE: Scanning PTREF for duplicate records
SCRIPT NOTE: No duplicates found in /home/modeling/2022v1EMP/2022hc_cb6_22m/ge_dat/temporal/ptegu_noncem_ptref_summer_2022hc_26jun2024_v0 testing SRGPRO_PATH set by SRGPRO input: /home/modeling/2022v1EMP/2022hc_cb6_22m/ge_dat/gridding/surrogates/CONUS12_2020NEI_25may2023/2022_oilgas/
Linux2_x86_64ifort testing SRGPRO_PATH set by SRGPRO input: /home/modeling/2022v1EMP/2022hc_cb6_22m/ge_dat/gridding/surrogates/CONUS12_2020NEI_25may2023/2022_oilgas/
Linux2_x86_64ifort
Not running daily inventory for this sector (DAY_SPECIFIC_YN not set to Y).
Creating dataset /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/ptegu/pthour_ptegu_may_2022hc_cb6_22m.lst
using script combine_data.csh
Processing environment variables EMISHOUR_MULTI_A
nameprefix = /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/HOUR_UNIT_2022_01_2022_2022cems_20240615_1-30_hourly.csv
month = 05
monname = may
/home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/HOUR_UNIT_2022_01_2022_2022cems_20240615_1-30_hourly.csv
05
6
/bin/ls: No match.
/bin/ls: No match.
SCRIPT ERROR: Some file patterns not matched for style Hourly_egu
According to MULTIFILE_NAMEBREAK, the file prefixes are:
/home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/HOUR_UNIT_2022_01_2022_2022cems_20240615_1-30_hourly.csv
Check file names and rerun
ERROR: Could not run combine_data.csh for hourly list file in Annual_area.
The cem input directory has the following files (I renamed the files and added files for last day of the month):
To me, it seems that the issue is caused by the combine_data_v6.csh scripts. I do not understand why the script came up with this nameprefix = /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/HOUR_UNIT_2022_01_2022_2022cems_20240615_1-30_hourly.csv instead of nameprefix = /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/HOUR_UNIT.
The “Annual_ptegu_summer_daily_12US1_2022hc_cb6_22m.csh” script has the following input for cem data:
setenv EMISHOUR_MULTI_A “${CASEINPUTS}/cem/HOUR_UNIT_2022_01_2022_2022cems_20240615_1-30_hourly.csv”.
I also have the same errors when running the cmv_c1c2 and _cmv_c3 scripts.
You shouldn’t have to rename and split the hourly data as they did – everything should work as we provided it with the original filenames as they were zipped. It’s not like the old CEMs format, which it seems like you were trying to reproduce.
Did you try running the scripts with no changes to the filenames? If so, and if it didn’t work, what errors did you get?
Yes, I did try to run it with no changes to the file name. This is the error I got:
testing SRGPRO_PATH set by SRGPRO input: /home/modeling/2022v1EMP/2022hc_cb6_22m/ge_dat/gridding/surrogates/CONUS12_2020NEI_25may2023/2022_oilgas/
Linux2_x86_64ifort
Not running daily inventory for this sector (DAY_SPECIFIC_YN not set to Y).
Creating dataset /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/ptegu/pthour_ptegu_may_2022hc_cb6_22m.lst
using script combine_data.csh
Processing environment variables EMISHOUR_MULTI_A
nameprefix = /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv
month = 05
monname = may
/home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv
05
6
/bin/ls: No match.
/bin/ls: No match.
SCRIPT ERROR: Some file patterns not matched for style Hourly_egu
According to MULTIFILE_NAMEBREAK, the file prefixes are:
/home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv
Check file names and rerun
ERROR: Could not run combine_data.csh for hourly list file in Annual_area
Please note that I was able to run other point categories (airports and ptnonipm). There was an error that had to do with m3stat. However, I was able to resolve it based on your recommendation on a previous posting. I needed to replace csh with tcsh in the m3stat script.
I am not running spinup days:
setenv RUN_MONTHS “5 6 7 8 9”
setenv SPINUP_DURATION “0”
setenv SPINUP_MONTH_END “Y”
And the script seems to be able to locate the cem input files.
Here is the output after the suggested change to the combine_data_v6.csh script.
testing SRGPRO_PATH set by SRGPRO input: /home/modeling/2022v1EMP/2022hc_cb6_22m/ge_dat/gridding/surrogates/CONUS12_2020NEI_25may2023/2022_oilgas/
Linux2_x86_64ifort
NOTE: Default MONTH setting to ‘jan’
/bin/ls: No match.
/bin/ls: No match.
/bin/ls: No match.
/bin/ls: No match.
SCRIPT NOTE: Setting months with no partial months
SCRIPT NOTE: Scanning PTREF for duplicate records
SCRIPT NOTE: No duplicates found in /home/modeling/2022v1EMP/2022hc_cb6_22m/ge_dat/temporal/ptegu_noncem_ptref_summer_2022hc_26jun2024_v0 testing SRGPRO_PATH set by SRGPRO input: /home/modeling/2022v1EMP/2022hc_cb6_22m/ge_dat/gridding/surrogates/CONUS12_2020NEI_25may2023/2022_oilgas/
Linux2_x86_64ifort testing SRGPRO_PATH set by SRGPRO input: /home/modeling/2022v1EMP/2022hc_cb6_22m/ge_dat/gridding/surrogates/CONUS12_2020NEI_25may2023/2022_oilgas/
Linux2_x86_64ifort
Not running daily inventory for this sector (DAY_SPECIFIC_YN not set to Y).
switch ( 5 )
set scriptname = combine_data.csh
set date = date +%m/%d/%Y
date +%m/%d/%Y
set eflag = N
set monname = ( jan feb mar apr may jun jul aug sep oct nov dec )
set lastday = ( 31 28 31 30 31 30 31 31 30 31 30 31 )
set valid_month = N
set style = 0
if ( 5 > 3 ) then
set month = 05
foreach m ( 01 02 03 04 05 06 07 08 09 10 11 12 )
if ( 01 == 05 ) then
end
if ( 02 == 05 ) then
end
if ( 03 == 05 ) then
end
if ( 04 == 05 ) then
end
if ( 05 == 05 ) then
set valid_month = Y
endif
end
if ( 06 == 05 ) then
end
if ( 07 == 05 ) then
end
if ( 08 == 05 ) then
end
if ( 09 == 05 ) then
end
if ( 10 == 05 ) then
end
if ( 11 == 05 ) then
end
if ( 12 == 05 ) then
end
if ( Y == N ) then
set prevmonnum = 0
if ( echo $month | cut -c1 == 0 ) then
cut -c1
echo 05
set pmonth = echo $month | cut -c2
cut -c2
echo 05
else
@ prevmonnum = 5 - 1
@ nextmonnum = 5 + 1
if ( 5 > 4 ) then
set style = Hourly_egu
switch ( Hourly_egu )
if ( 1 ) then
if ( 2022 % 4 == 0 ) then
set thisyear = 2022
set prevyear = 2022
@ prevyear = 2022 - 1
else
breaksw
endif
else
if ( 5 > 2 ) then
switch ( list )
case concat:
set type = list
breaksw
endif
set prefix = EMISHOUR
set outfile = /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/ptegu/pthour_ptegu_may_2022hc_cb6_22m.lst
echo Creating dataset /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/ptegu/pthour_ptegu_may_2022hc_cb6_22m.lst
Creating dataset /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/ptegu/pthour_ptegu_may_2022hc_cb6_22m.lst
echo using script combine_data.csh
using script combine_data.csh
set sort_list_evs = N
if ( 1 ) then
set sort_list_evs = Y
endif
if ( Y == Y ) then
set envlist = ( env | grep $prefix | sort | cut -d\= -f1 )
env
grep EMISHOUR
cut -d= -f1
sort
set filelist = ( env | grep $prefix | sort | cut -d\= -f2 )
env
grep EMISHOUR
cut -d= -f2
sort
else
set multifile_break = 0
if ( 1 ) then
if ( 5 > 0 ) then
set multifile_break = 5
endif
endif
if ( EMISHOUR_MULTI_A == ) then
set exitstat = 0
foreach f ( /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv )
if ( ! -e /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv ) then
end
if ( 0 > 0 ) then
if ( list == concat ) then
if ( -e /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/ptegu/pthour_ptegu_may_2022hc_cb6_22m.lst ) then
echo Processing environment variables EMISHOUR_MULTI_A
Processing environment variables EMISHOUR_MULTI_A
set n = 0
set firstime = Y
set nmax = 90
set use_ev_array = ( N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N )
set filename =
foreach ev ( EMISHOUR_MULTI_A )
@ n = 0 + 1
set use_ev = Y
set ev_month = echo $ev | cut -d_ -f2 | cut -c1-2
echo EMISHOUR_MULTI_A
cut -c1-2
cut -d_ -f2
set ev_month_yn = N
set multi = N
echo EMISHOUR_MULTI_A
grep -q MULTI
if ( 0 == 0 ) then
set multi = Y
endif
foreach m ( 01 02 03 04 05 06 07 08 09 10 11 12 )
if ( 01 == MU ) then
end
if ( 02 == MU ) then
end
if ( 03 == MU ) then
end
if ( 04 == MU ) then
end
if ( 05 == MU ) then
end
if ( 06 == MU ) then
end
if ( 07 == MU ) then
end
if ( 08 == MU ) then
end
if ( 09 == MU ) then
end
if ( 10 == MU ) then
end
if ( 11 == MU ) then
end
if ( 12 == MU ) then
end
if ( 05 == N && N == Y ) then
if ( Y == Y && N == Y ) then
set eflag = N
if ( Y == Y ) then
if ( 5 > 0 && Y == Y ) then
if ( Hourly_egu == Hourly ) then
if ( Hourly_egu == Hourly_CMV || Hourly_egu == Hourly_CMV19 || Hourly_egu == Hourly_egu ) then
set namecheck = ls $filelist[$n] | awk 'match($0, /.*_hourly.csv$/) {print($0)}'|wc -m
ls /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv
wc -m
awk match(0, /.*_hourly.csv/) {print($0)}
if ( 115 == 0 ) then
set i = basename $filelist[$n] | awk 'match($0, /.*?_(0[1-9]{1}|1[0-2]{1}|[1-9]{1})_/) {print substr($0,RSTART,RLENGTH-1)}' | tr -dc '_' | awk '{ print length; }'
basename /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv
awk { print length; }
awk match($0, /.*?(0[1-9]{1}|1[0-2]{1}|[1-9]{1})/) {print substr($0,RSTART,RLENGTH-1)}
tr -dc _
set nameprefix = ( dirname $filelist[$n]/basename $filelist[$n] | cut -d_ -f1-$i )
dirname /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv
cut -d_ -f1-
basename /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv
else
echo nameprefix = /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv
nameprefix = /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv
echo month = 05
month = 05
echo monname = may
monname = may
switch ( Hourly_egu )
echo /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv
/home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv
echo 05
05
set nextmonnum0 = 6
echo 6
6
if ( 6 < 10 ) set nextmonnum0 = 06
set nextmonnum0 = 06
if ( 6 < 13 ) then
set filename = ( /bin/ls ${nameprefix}_${nextmonnum0}_*_hourly.csv )
/bin/ls: No match.
if ( 1 > 0 ) set exitstat = 1
set exitstat = 1
echo
else
@ monthnum = 05 + 0
set monthnum0 = 5
if ( 5 < 10 ) set monthnum0 = 05
set monthnum0 = 05
set filename_x = ( /bin/ls ${nameprefix}_${monthnum0}_*_hourly.csv )
/bin/ls: No match.
if ( 1 > 0 ) set exitstat = 1
set exitstat = 1
echo
set filename = ( )
if ( 0 > 0 ) set exitstat = 1
breaksw
else
if ( 1 > 0 ) then
echo SCRIPT ERROR: Some file patterns not matched for style Hourly_egu
SCRIPT ERROR: Some file patterns not matched for style Hourly_egu
echo According to MULTIFILE_NAMEBREAK, the file prefixes are:
According to MULTIFILE_NAMEBREAK, the file prefixes are:
echo /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv
/home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv
echo Check file names and rerun
Check file names and rerun
exit ( 1 )
ERROR: Could not run combine_data.csh for hourly list file in Annual_area
This may be related to your Unix environment and possibly awk.
Can you please run the following commands interactively from a C-shell terminal and send the output for each one?
First, do setenv FILELIST /home/modeling/2022v1EMP/2022hc_cb6_22m/2022hc_cb6_22m/inputs/cem/pthour_01_2022_2022_2022cems_20240615_hourly.csv
Then execute each of these, and report back with the output from each command:
But when I replaced the single quotes from ‘ ’ to ’ ’ using my keyboard, I got no outputs.
Here is the awk information in our system:
update-alternatives --display awk
awk - auto mode
link best version is /usr/bin/mawk
link currently points to /usr/bin/mawk
link awk is /usr/bin/awk
slave awk.1.gz is /usr/share/man/man1/awk.1.gz
slave nawk is /usr/bin/nawk
slave nawk.1.gz is /usr/share/man/man1/nawk.1.gz
/usr/bin/mawk - priority 5
slave awk.1.gz: /usr/share/man/man1/mawk.1.gz
slave nawk: /usr/bin/mawk
slave nawk.1.gz: /usr/share/man/man1/mawk.1.gz
awk -W version
mawk 1.3.4 20200120
Copyright 2008-2019,2020, Thomas E. Dickey
Copyright 1991-1996,2014, Michael D. Brennan
random-funcs: srandom/random
regex-funcs: internal
compiled limits:
sprintf buffer 8192
maximum-integer 2147483647
UPDATE
Based on the above, our system uses mawk. I went ahead and installed gawk, reran the script, and the ptegu script is now working!
Alison, thank you for pointing me out to the awk as a possible culprit.