ERROR detected in Smkreport / ERROR: Running qa_run for inv

SMOKE
1

Hi there,

I’ve received this error when trying to run smkreport for a new pollutant in the UK (using SMOKE v4.9):

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0 0x155285f48d4f in ???
#1 0x15528607ed86 in ???
#2 0x408281 in ???
#3 0x448054 in ???
#4 0x402e6c in ???
#5 0x155285f3329c in ???
#6 0x402ec9 in ???
at ../sysdeps/x86_64/start.S:120
#7 0xffffffffffffffff in ???
Segmentation fault
0.091u 0.179s 0:02.07 12.5% 0+0k 153940+72io 3081pf+0w
SCRIPT NOTE: timetracker is replacing line 2 of the TIMELOG file
Missing ‘}’.
now checking log file [dir]/smoke/run_2023_cb6_pfas/static/logs/smkreport_pt_pfas_2023_cb6_pfas_inv.log

  • ERROR detected in logfile:
  • [dir]/smoke/run_2023_cb6_pfas/static/logs/smkreport_pt_pfas_2023_cb6_pfas_inv.log

ERROR detected in Smkreport
ERROR: Running qa_run for inv
log analyzer
Getting message data (might take some time)…
grep: [dir]/smoke/run_2023_cb6_pfas/static/logs: Is a directory
grep: [dir]/smoke/run_2023_cb6_pfas/static/logs: Is a directory
Finished getting data
Classifying message types…
Total number of known messages: 12
Total number of unknown messages: 1
Level 3 analysis…
Finished classifying message types
log analyzer
Getting message data (might take some time)…
grep: [dir]/smoke/run_2023_cb6_pfas/static/logs: Is a directory
grep: [dir]/smoke/run_2023_cb6_pfas/static/logs: Is a directory
Finished getting data
Classifying message types…
Total number of known messages: 12
Total number of unknown messages: 1
Level 1 analysis…
Finished classifying message types
Testing for exit priority <= 1
ERROR: At least one message’s priority is less than or equal to the exit priority: 1
ERROR: Level 1 errors or Level 0 warnings found. Check log files, fix inputs, and rerun.

When I look at the level1 log in the log_analyzer folder it includes the following:

priority,known,count,message type,first file
0,False,1,‘WARNING: Applying default time zone of 00 to country, state, and county code 001001002000’,[dir]/smoke/run_2023_cb6_pfas/static/logs/smkinven_3km_NI_2023_cb6_pfas.log
2,True,1,‘WARNING: No kept pollutants found at line .*’,[dir]/smoke/run_2023_cb6_pfas/static/logs/smkinven_3km_NI_2023_cb6_pfas.log
3,True,5,‘Value for SMK_MAXWARNING:’,[dir]/smoke/run_2023_cb6_pfas/static/logs/smkinven_3km_NI_2023_cb6_pfas.log
3,True,1,‘Value for SMK_MAXERROR:’,[dir]/smoke/run_2023_cb6_pfas/static/logs/smkinven_3km_NI_2023_cb6_pfas.log
3,True,4,‘WARNING: output file already exists:’,/[dir]/smoke/run_2023_cb6_pfas/static/logs/smkinven_3km_NI_2023_cb6_pfas.log
3,True,1,‘WARNING: Missing average day emissions for’,[dir]/smoke/run_2023_cb6_pfas/static/logs/smkinven_3km_NI_2023_cb6_pfas.log

I’m not sure whether it’s the SMK_DEFUALT_TZONE entry of 0 that is causing the issue as it’s returning a priority of 0 (I am trying to run for GMT) or whether I need to try installing a different version of smkreport as per a previous post https://forum.cmascenter.org/t/smoke-issue-program-received-signal-sigsegv-segmentation-fault-invalid-memory-reference/3102 (or some other issue).

Looking at the smkreport logfile the program appears to stop part way through. Is there anything obvious here that could be causing an issue?

Thanks very much

2

You mention that you were running smkreport for a new pollutant, was this pollutant already in the pnts_map text file in your intermediate directory? Is the associated inventory intermediate file populated with values?

If you are able to share your run script, inventory table, and repconfig files that will help us identify a potential cause for the seg fault.

3

Hi James, thank you for your response.

I can’t seem to find a pnts_map text file (there are no files in the intermediate directory that I have set in the directory_definitions.csh file).

I’ve attached my onetime run file, inventory table (new pollutants added at the end) and repconfig file.

Thanks very much.

repconfig_point_PFAS.txt (461 Bytes)
Annual_pt_PFAS_onetime_issue.csh (4.1 KB)
invtable_CMAQ_PFAS_v1.txt (125.4 KB)

4

Can you confirm that smkinven and grdmat successfully ran? You should have outputs from both in your intermediate directory. If they did run please check that smkreport is looking for the intermediate files in the same place that they are being written.

According to your post smkinven and smkreport logs were created. When you get an opportunity please share those two logs. That may help me narrow down the exact location that the process is failing. Your repconfig and invtable both look correct.

5

Hi James,

INTERMED and IMD_ROOT are set as $PROJECT_ROOT/$CASE/intermed, however there are no files in that directory. According to the smkinven and grdmat log files, both processes successfully ran (smkinven log attached). I’ve checked the output files listed in those logs and they exist, but [dir]/smoke/reports/2023_cb6_pfas/static/repinven.3km_NI.pfas_POINT.txt is empty.

The smkreport log indicates that the process is stopping abruptly on open of REPORT 1. That file does exist but is empty.

Many thanks
smkreport.txt (5.3 KB)
smkinven.txt (10.3 KB)

6

Hello,

I wondered whether you had any initial ideas as to what the issue may be that I could investigate further?

Very grateful for your help,
Kate

7

Kate,

I was able to reproduce the error by processing a point inventory with one source and pollutant. Based on your smkinven log it looks like you inventory only contains a single source. I haven’t done a full debug yet but the smkreport log points to the asgnbins subroutine.

Could you please try to add an additional source and pollutant to the inventory within your domain and rerun?

8

Hi James,

I’ve tried adding another source and rerunning but I get the same error. I still receive a priority 0 warning in the following file: [dir]/smoke/reports/2023_cb6_pfas/log_analyzer/rep_logs_pt_pfas_2023_cb6_pfas_3km_NI_onetime_level1.csv

priority,known,count,message type,first file
0,False,1,‘WARNING: Applying default time zone of 00 to country, state, and county code 001001002000’,[dir]/smoke/run_2023_cb6_pfas/static/logs/smkinven_3km_NI_2023_cb6_pfas.log

At run time I get this message before it fails:

Level 1 analysis…
Finished classifying message types
Testing for exit priority <= 1
ERROR: At least one message’s priority is less than or equal to the exit priority: 1
ERROR: Level 1 errors or Level 0 warnings found. Check log files, fix inputs, and rerun.

Could this be causing the error? I have set OUTZONE “0” and SMK_DEFAULT_TZONE = “00” in my main run script.

The programme is producing populated PSCC.XX.txt, pnts.map.XXX.txt, pnts_X.ncf, and psrc_XX.txt files in the output directory, and within that, another directory with a .ncf file per pollutant.

There is a repinven.XX.txt file within [dir]/smoke/run_2023_cb6_pfas/static/logs, but it’s empty.

Many thanks

9

This warning shouldn’t be a problem if your emissions source is located in the GMT timezone. It seems like smkinven ran successfully based on the output that you described. The problem is likely still in the asgnbins subroutine of smkreport and related to source aggregation. I reproduced the issue with a dummy inventory but I would like to try your exact inventory.

Could you please share your point inventory and grid definition (you are using GRIDDING aggregation in smkreport)?

10

Hi James,

I’ve attached a copy of the emissions inventory, griddesc and geocode files. I had added a new lambert griddesc at the top of the griddesc*.txt file. I realised I didn’t have a description of the grid further down the file, which I’ve now added (copied from the GRIDDESC file produced via mcip). However I’m getting the same error.

I am using GRIDDING in the smkreport settings.

geocode_level3.txt (89 Bytes)

geocode_level2.txt (72 Bytes)

geocode_level1.txt (51 Bytes)

geocode_level4.txt (108 Bytes)

ptpfas_POINT_share.csv (3.1 KB)

griddesc_lambertonly_18jan2019_v7.txt (12.0 KB)

Thank you

11

Thank you so much for sharing these files. I was able to exactly reproduce the error and create a fix that worked on our system.

In the inventory file (ptpfas_POINT.csv (2.1 KB)) I changed the header to “COUNTRY=NI”, the country_cd to “NI”, the region_cd to “1”, and added a dummy value to the process_id

In the COSTCY file (costcy_for_2017platform_07dec2021_v2.txt (754.5 KB)) I added a country record of “NI” with a country code of 4. I then added corresponding state and county level records for the whole country. I assigned a timezone of “GMT” to the state and county records. To use the COSTCY file set the “USE_EXP_GEOCODE” variable in your script to “N” and define the COSTCY environment variable to the location of the COSTCY file.

Finally I modified the repconfig file (repconfig_point_PFAS.txt (478 Bytes)). Note that I added the group by lines of “BY CELL” and “BY STATE”. The “BY CELL” is optional, although if you are applying the GRIDDING function I’m asuming that you want the output grid cell.

If you are still having issues after deleting the original intermediate files and rerunning from the beginning of SMOKE please let us know.

12

Thank you very much James. All of the logs within the static/logs folder are now showing “Normal completion of program” (elevpoint, grdmat, smkinven, smkreport_X_inv, smkreport_X_invgrid.

I still get the same message at run time:

Level 1 analysis…
Finished classifying message types
Testing for exit priority <= 1
ERROR: At least one message’s priority is less than or equal to the exit priority: 1
ERROR: Level 1 errors or Level 0 warnings found. Check log files, fix inputs, and rerun.

Does this matter given that the logs show successful completion?

Thanks again,

Kate

13

That message can be ignored if all SMOKE program logs show successful completion.