Problems running latest SMOKE test case

SMOKE
2

Hi Lorenzo,

Thanks for reporting the issue and your related email to me. I’ve rerun the SMOKE example package and I could not replicate the issue that you reported here.

Upon having deeper investigation, there are several reasons that might explain the issue that you are seeing:

  1. Buggy timetracker _v2.csh that came with SMOKE Example Case v2:
    In smoke_example_case/smoke5.1/scripts/run/timetracker_v2.csh (lines 139-143), the script deletes an existing TIMELOG entry using /bin/ed via heredoc:
    /bin/ed -s $TIMELOG << EOF
    H
    ${nrow}d
    w
    EOF

On your Rocky Linux 10.1 system, it is likely /bin/ed is not available, and so this code block silently failed. Consequently, the delete command is never executed and the script continues as if the deletion succeeded. The old TIMELOG entry was NOT removed. (This could explain why I could not replicate this issue since /bin/ed does exist on my system).

  1. The solution for this buggy timetracker script is to get a newer version of timetracker_v2.csh that does not use /bin/ed. In fact, the timetracker_v2.csh that Alison provided in the forum post that you linked here is the correct one to use (which looks like came from EMP 2022v2). This timetracker_v2.csh does not use /bin/ed, rather, it uses sed/mv command to remove $TIMELOG file

  2. However, replacing timetracker_v2.csh, as you noted, did not completely resolve your issue because you might still have residual $TIMELOG files created from previous incomplete run. To completely resolve this issue, you should clean up the log directory, or better yet, remove the intermediate/ directory before re-run with the updated timetracker_v2.csh

  3. Why this issue only happened to some sectors and not to others: The crash requires at least 3 active smkreport/timetracker calls in one run. Which sectors have this depends on which REPCONFIG_INV* environment variables are set in the sector-specific run script. Sectors with fewer active REPCONFIG files make fewer qa_run calls and may crash later or not at all.

Hope this helps.

Btw, I recently developed some interactive tool to run SMOKE. Please check it out let me know what you think. Thanks.

Huy