Log_analyzer flags routine Smkinven info messages as priority-1 in SMOKE v5.2.1

SMOKE
1

Hi All,

Environment:
-SMOKE v5.2.1 (Sep 2025 release), compiled with Intel ifx 25.0
-known_messages.txt from an earlier SMOKE release (still shipped with the log_analyzer package), probably?
-Exit priority threshold: <= 1

Issue:
Running Smkreport with QA (log_analyzer) triggers exit 1 even though SMOKE
completes successfully with no ERROR ABORT in any log file. The level-1 report
lists appx. 10 messages, all marked “unknown” (False in the classified column) and defaulted to priority 1 by log_analyzer. Examples from a livestock sector run:

‘Reading INVENTORY TABLE…’
‘Cross-reference entries read:’
‘Actual COUNTIES:’
‘Actual monthly PROFILES:’
‘Actual WEEKLY PROFILES:’
‘Actual HOURLY PROFILES:’
‘Actual TIME ZONES:’
‘Missing SCCs:’
‘Total number of SRCS:’
These are standard informational stdout lines from Smkinven and Smkreport,
not actual errors as far as I know. As I suspect, they aren’t in the shipped known_messages.txt, so log_analyzer classifies them as unknown and defaults to priority 1. That trips the wrapper’s exit threshold (<=1) and halts the pipeline. As a
downstream consequence, Smkmerge never runs and no gridded livestock
output NCF is produced, even though the underlying SMOKE run is clean and
all intermediate files are valid. Disabling smokereport, I am getting the gridded output normally.
Query:
Is there an updated known_messages.txt covering the current SMOKE v5.2.1
info messages, or a recommended way to treat “unknown-but-informational”
lines without lowering the exit threshold globally (which would also mask
real priority-1 warnings from other sectors)? I plan to reduce the priority of the unknown messages as a temporary fix, would that be reasonable? if not the updated file is available.

Thanks in advance for your help and guidance.

-Priom

2

In the original (1991-1993) requirements analysis it was recognized that searching logs for error-messages was known to be unreliable (in part because the system can kill processes before they can “scream for help”); however, the UNIX/Linux systems-programming construct of exit-status provides a robust mechanism for error-detection:

  • SMOKE programs terminate with routine M3EXIT, which sets the error-status from its status-argument: 0 for success, non-zero for failure
  • If the system kills a program, it sets a non-zero error-status; these and their meanings are catalogued in the system’s /usr/include/errorno.h file
  • All execution-errors give a non-zero error-status for failure, zero error-status for success by the above two mechanisms.
  • Master-scripts should not continue running after detection of an error Otherwise one is left in the situation of “Just where did the error occur?”, a question asked repeatedly on this Forum
  • Immediately after the execution of a program or script, a code-chunk such as that immediately below easily finds whether the program or script failed or not:
program
set foo = ${status}
if ( ${foo} != 0 )  
    echo "ERROR ${foo} on program"
    exit( ${foo})
endif
...

Somewhere, the results of this analysis were lost, and its robust error-detection was replaced by the earlier search-the-log known-inadequate system.

[Note that when you have a “master script” run control strategy, you are doing systems programming whether you admit it or not.]

– Carlie J. Coats, Jr., Ph.D
I/O API Author/Maintainer
Original SMOKE author

3

Hi Dr. Coats,

Thank you for your reply and the historical context , it was very helpful to understand the original design intent. The M3EXIT / ${status} approach makes complete sense, and you’re rightly mentioned that the log-scraping layer has caused us exactly the kind of ambiguity you describe (“just where did the error occur?”).

Our plan is to refactor our sector wrapper scripts (nonpt, livestock,cmv*, etc.) to check ${status} immediately after each SMOKE call and fail-fast, exactly as you outlined. We are planning to bypass or retire the log_analyzer step entirely in our system if it is causing issues.

One follow-up question: for the intermediate scripts SMOKE ships (e.g., smk_area_daily.csh, smk_pt_daily.csh), do you recommend we add explicit ${status} checks after each Fortran program call within those, or do the shipped scripts already exit non-zero if any internal call fails? We’ve observed some cases where a sector script appeared to complete cleanly but produced no output wanted to make sure we’re not missing an internal error-swallowing pattern.

Thanks again for the guidance.

Best,
Priom

4

Suppose you have multiple levels of script calls, then it might look
something like this below, where script able calls script “baker” (and
other stuff), which then calls Fortran program foo (and other stuff).

In script able:

...
baker
set istat = ${status}
if ( ${istat} != 0 )  
    echo "ERROR ${istat} on script baker called from script able"
    exit( ${istat})
endif
....

and in script baker:

....
foo
set istat = ${status}
if ( ${istat} != 0 )  
    echo "ERROR ${istat} on program foo called from script baker"
    exit( ${istat})
endif
....

Then if foo fails, you will have a simple, 2-step trace from the top level log down to the place the error occurred: the log for alpha will end with something like

ERROR 2 on script baker called from script able

and the log for baker will end with

ERROR 2 on program foo called from script baker

-- Carlie

5

Thanks so much, Dr. Coats, the layered ${status} pattern is clear and we’ll try to adopt it uniformly.

A follow-up clarification for our situation: our sector scripts inherit the EPA-shipped log_analyzer step (e.g., smk_ar_monthly_emf.csh calls $SCRIPTS/log_analyzer/log_analyzer.py at the end for QA reporting). If I understand correctly, your guidance is to bypass that entirely and rely on ${status} even though it ships in the reference scripts? Apologies, If I did not understand your suggestions correctly.

If so, we’ll retire the log_analyzer invocation from our sector wrappers and switch to the layered ${status} trace throughout.

Best,
Priom