COMBINE utility run problems _cannot open or read SPECIES_DEF #CMAQ#CAMx

[Iqbal]

Hello,
I have installed and compiled ioapi v3.2 and netcdf v-4.7.4 by using pgf90 compiler, using these I got .nc files from CAMx output.
Now, I am trying to run ‘combine’ utility to combine files. I have compiled the combine.exe successfully. I have made several changes in the ‘makefile’ while compiling the ‘combine.exe’ and the changes are : 1. changed appropriate directories of IOAPI_INC, IOAPI_LIB, NETCDF_LIB. 2. I modified -lnetcdf to -lentcdff -lnetcdf in LIBRARIES as my netcdf is v4.7.4 (attached makefile Makefile.txt (822 Bytes) . I have also attached combine.f file combine.F.txt (7.5 KB) .

The I tried to run a job file by using attached job file ( TRIAL_COMBINE_elements.csh.txt (1016 Bytes) .

But I am getting an error (error file attached:combine_error.txt (3.2 KB) :
Missing environment variable EXECUTION_ID
** I Started
Value for GENSPEC not defined;returning default: FALSE
ERROR Cannot open or read SPECIES_DEF file:/root/Documents/CAMxPostProcessSoft/spec_def/spec_def.camxv7.elements
finish 20160716

1.I double checked the directory of spec_def_camxv7.elements. (attached screenshot : )
2. I tried different ways, such as moving the directory to another place and subsequently changed the job file directory etc.
3. I checked file permission (attached ls.log :ls.txt (3.6 KB) ).
4. I checked whether file is corrupted or not (attached file : spec_def.camxv7.elements.txt (1.1 KB) . And used another spec_def file (spec_def.camx.avrg.txt (772 Bytes) , subsequently changed the file name in the job script.
Note: in the filename ‘.txt’ is added here just to upload ‘spec_def.camxv7.elements’ file in the CMAS Forum.

Please help

[lizadams]

I believe the error indicates that the species_definition file can’t be read correctly.
ERROR Cannot read SPECIES_DEF

In the species definition file, the input file names are indicated by the number in the brackets.

According to your spec_def file, you are trying to read in the variables PM25_TOT and PMC_TOT to create a new variable PM10_TOT from the first file denoted by [0], and the remaining variables such s Ozone are being redefined by being read in from the second file, denoted by [1].

/new species    ,units    ,expression
O3         ,ppbv    ,O3[1]*1000


PM10_TOT   ,ug/m3   ,PM25_TOT[0]+PMC_TOT[0]

Can you confirm the identity of your input files as specified in your run script?

grep INFILE1 run_combine.csh
grep INFILE2 run_combine.csh

Once you confirm the identify of those input files, please check and see what variables they contain

ncdump -h filename

This should output the header of the input file, so you can determine what variables are available in the files.

[hogrefe.christian]

@lizadams , the file references in the square brackets are defined slightly differently. [1] refers to INFILE1, [2] would refer to INFILE2, etc. [0] is a special case referring to a variable defined previously in the SpecDef file, i.e. any variable already defined in the file can be used in subsequent expressions by referring to it with [0] after the variable name.

@Iqbal, I’ve been staring at your run script, log file, and SpecDef file and nothing jumps out at me. I’m probably overlooking something basic, but I just can’t figure out what it is.

We do know that subroutine readSpec in module_specdef.F returns a non-zero exit status, causing combine.F to exit with the message “ERROR Cannot open or read SPECIES_DEF file:” you’re reporting.

To find out where readSpec fails when trying to open or read your SpecDef file, you could add write statements such as Write(*,'(a)') 'After file open' or Write(*,'(''record field:'',a)') trim(record) after OPEN and READ statements in the readSpec subroutine of module_specdef.f, recompile the code, and rerun the executable to see exactly where the file open or read error occurs in readSpec.

[hogrefe.christian]

When you run the script, do you run as ‘root’ or under a different user name?
In addition to checking the permissions for the spec_def.camxv7.elements file and the spec_def directory, did you also check the required permissions for each of the higher level directories in the path (/root/Documents/CAMxPostProcessSoft)?

[hogrefe.christian]

Could you please provide more information on which version of combine you are using and when and where you obtained it? The combine.F.txt file you posted seems to differ from the current version.

[Iqbal]

Hello @lizadams
It is still giving me same problem. Please see the attached (ncdum -h ouput :ncdump_header_log.txt (22.8 KB) ; grep INFILE1 command (

; and the jobfile used for this ( attached : TRIAL2_COMBINE_PM25.txt (1016 Bytes) . Also, in this job file my spec_def file is spec_def.camx.avrg.txt (772 Bytes)

Thank you for the help

[lizadams]

It looks like there is an extra blank line after the PM10_TOT definition in the spec_def.camxv7.elements file. I don’t know if that would cause an issue, but it is perhaps safer to remove it.

[Iqbal]

Hello @hogrefe.christian
Thank you so much for your responses. Here is what I did as you suggested:

1. I added Write(*,’(’‘record field:’’,a)’) trim(record) in module_specdef.f file (attached : module_specdef.txt (15.9 KB) , recompile and re-ran the job file . But no output shows and error persists and error message is ‘segmentation fault core dumped’ ( attached error.log file :ErrorLogmodified_SpecDef.txt (18.3 KB)
2. In the ErrorLog_modified_SpecDef.txt (10.3 KB) file , I can see "
   “INFILE1” opened as OLD:READ-ONLY
   File name “/run/media/root/55A07B0360BDF544/camx2ioapi_v6/output/ARDEQ_camxv7_cb6r4DMS.20160715.2016v1.avrg.grd02.nc”

So, I checked the file permission again where I don’t see any problem (attached screenshot :

3. In the ErrorLog file ErrorLogmodified_SpecDef.txt (18.3 KB)
   API Started
   Value for GENSPEC not defined;returning default: FALSE
   rcord field:\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00
   Does it conerning?

4. I checked higher level directories path permission too (attached : ls_lR.txt (3.8 KB)

5. I got this src code from one of our project partners. I attached the different ‘.f’ file here (combine.F.txt (7.5 KB) Makefile.txt (822 Bytes) module_evaluator.F.txt (25.6 KB) utils.F.txt (4.6 KB)

Thank you so much

[hogrefe.christian]

Can you please also post module_specdef.F?

Sorry, never mind, I see you posted it further above, let me have a look.

[hogrefe.christian]

The write statement after OPEN should have been Write(*,'(a)') 'After file open'. Because nothing was read into ‘record’ yet, and because you used the write statement that included a reference to ‘record’, your write statement ended up looking the way you posted under 3.

Can you try changing line 89 in your modified file, i.e. the line after the OPEN statement, to

Write(*,'(''after file open, istatus:'',i2)') istatus

and run again?

[Iqbal]

Hi @hogrefe.christian
I tried to compile that after modifying module_specdf.F at line 89 (attached : module_specdef.F.txt (15.9 KB)

But this error occurs

[hogrefe.christian]

Sorry, my bad. The variable name in the write statement should be status, not istatus

[Iqbal]

Hi @hogrefe.christian
I successfully complied and ran it again, the error log is here: terminal_log.txt (6.3 KB)
The error message:
“INFILE1” opened as OLD:READ-ONLY
File name “/run/media/root/55A07B0360BDF544/camx2ioapi_v6/output/ARDEQ_camxv7_cb6r4DMS.20160716.2016v1.avrg.grd02.nc”
File type GRDDED3
Execution ID “???”
Grid name “???”
Dimensions: 248 rows, 398 cols, 1 lays, 96 vbles
NetCDF ID: 65536 opened as READONLY
Starting date and time 2016198:000000 (0:00:00 July 16, 2016)
Timestep 010000 (1:00:00 hh:mm:ss)
Maximum current record number 24
Segmentation fault (core dumped)
finish 20160716

[hogrefe.christian]

It’s strange that after adding these write statements to the readSpec subroutine, the code now proceeds further. It seems the SpecDef file is being read in its entirety and the code now proceeds to open the model output file before encountering a segmentation fault. Did you make any modifications to the SpecDef file compared to what you posted earlier?

You could try adding the following debug flags to your Makefile
-O0 -g -Mbounds -Mchkptr -traceback -Ktrap=fp,
i.e. change what you currently have

FC_FLAGS = -c -tp px -Mextend $(INCLUDES) -I$(IOAPI_LIB)

to

FC_FLAGS = -c -tp px -Mextend -O0 -g -Mbounds -Mchkptr -traceback -Ktrap=fp $(INCLUDES) -I$(IOAPI_LIB)

Then remove the .o and .mod files in your BLD directory and compile and run again.

[Iqbal]

Hello @hogrefe.christian
I added the code as you suggested and ran it again, but problems still happening . Please see the error.log terminal_log.txt (6.3 KB)
and the modified makefile Makefile.txt (878 Bytes)
I did not change spec_def file in all the changes I have been doing as you suggested . Here is the file i am using spec_def.camx.avrg.txt (772 Bytes)

Thanks

Edit: Correct error file (terminal log file is : terminal_log3.txt (3.3 KB) , where you can see the same problem

[hogrefe.christian]

When you recompiled the code with the Makefile updated to include debug flags, you still kept the modified version of module_specdef.F that included the write statements, correct?

The terminal_log.txt file you shared in post number 13 and also at the beginning of post number 15 differs from the terminal_log3.txt file you shared at the end of post number 15. The former reports successfully reading the SpecDef file and encountering a segmentation fault after opening the model output file while the latter file goes back to the error with reading the SpecDef file you reported in your earlier posts.

Adding the debug flags was meant to track down the cause of the segmentation fault you reported in post 13.

[lizadams]

Can you run the program m3stat on your input file as follows:

setenv INFILE /run/media/root/55A07B0360BDF544/camx2ioapi_v6/output/ARDEQ_camxv7_cb6r4DMS.20160715.2016v1.avrg.grd02.nc
m3stat
Then post the output file: REPORT

[Iqbal]

Hi @lizadams
Please see the report here: m3statReport.txt (490.3 KB)

I couldn’t find anything to concern.

Am I missing anything?

Thanks

[Iqbal]

Hi @hogrefe.christian
Okay, I did these again freshly. Here is what I did

1. Created new directories and checked the file permission ( permission.txt (1.8 KB) and checked the file path (

   
   ). the spec_def file is used in all the trials is spec_def.camx.avrg.txt (772 Bytes)

2. I compiled combine.exe by changing only directories of IOAPI_INC,IOAPI_LIB,NETCDF_LIB, and -lnetcdf to -lnetcdff -lnetcdf in LIBRARIES (Makefile.txt (830 Bytes) and kept this file as it is module_specdef.F.txt (15.8 KB) . Then I ran this job script ( combine.pm25.SMAT.txt (1.2 KB) which resulted error original_terminal_error.txt (5.0 KB)

3. Then I modified the module_specdef.F file as you suggested . The only modification was to add Write(*,’(’‘after file open, status:’’,i2)’) status at line 89. No changes were made in makefile. Then after compilation I ran it again. All the files : terminal_error log file, makefile, job file etc. are attached here. trial1_terminal_error.txt (6.5 KB) combine.pm25.SMAT_T1.txt (1.2 KB) Makefile_T1.txt (824 Bytes) module_specdef_T1.F.txt (15.9 KB)

4. In this trial, the module_specdef.F is same as it was in step-3 (i.e. modified in step-3) . In the makefile, I change FC_FLAGS = -c -tp px -Mextend (INCLUDES) -I(IOAPI_LIB)
   to
   FC_FLAGS = -c -tp px -Mextend -O0 -g -Mbounds -Mchkptr -traceback -Ktrap=fp (INCLUDES) -I(IOAPI_LIB)
   Then I recompile and ran it again. The error still occurs. I have attached all the files ( makefile, module_specdef, jobfile, error.log etc.)
   trial1_terminal_error.txt (6.5 KB) Makefile_T2.txt (870 Bytes) combine.pm25.SMAT_T2.txt (1.2 KB) module_specdef_T1.F.txt (15.9 KB)

[hogrefe.christian]

Thanks. Please correct me if I’m wrong, but what I think I see are two separate issues:

1. When you use spec_def.camxv7.elements, the program exits with the “Cannot open or read SPECIES_DEF file” error message

2. When you use spec_def.camx.avrg, the SpecDef file is being read and the I/O API file with CAMx values is being opened but the program terminates without producing output.

For issue 1, this indicates a problem with the formatting of that file, though I cannot see what it is. The ‘write’ statements I had suggested adding to module_specdef.F after the OPEN and READ statements were intended to track down that issue. This would be like your test number 3 in post 19, but adding more write statements after reads to pinpoint the location of the crash, and of course using spec_def.camxv7.elements as SpecDef file

For issue 2, which you first reported in post 8 and again in post 13, the log file doesn’t show any error message, but your screen shows Segmentation fault (core dumped). After you recompiled in debug mode and ran, i.e. when you conducted test number 4 in post 19, what did the screen show (i.e. how does the screen output from that test differ from what you showed in posts 8 and 13)? Adding debug flags was intended to obtain more information on what led to the segmentation fault. As an aside, the error log files you posted for test 3 and test 4 have the same file name, so I just wanted to confirm that they reflect the two separate tests.