# Program Exception when running SMOKE-TestCase

**URL:** <https://forum.cmascenter.org/t/program-exception-when-running-smoke-testcase/3680>\
**Category:** Run Errors and Warnings\
**Created:** [November 2, 2022, 4:11pm UTC](https://forum.cmascenter.org/t/program-exception-when-running-smoke-testcase/3680 "2022-11-02T16:11:44Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![SolL](https://avatars.discourse-cdn.com/v4/letter/s/9fc348/32.png) [@SolL](https://forum.cmascenter.org/u/SolL)\
**Post date:** [November 2, 2022, 4:11pm UTC](https://forum.cmascenter.org/t/program-exception-when-running-smoke-testcase/3680/1 "2022-11-02T16:11:44Z")

</div>

We succeeded compiling IOAPI - CMAQ - SMOKE wusing gcc compilers on a Deian 10 machine

```auto
Linux hydra 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux

```

using:

```auto
gnu/10.2.1
ioapi/v3.2
netcdf_c-4.8.1_fortran-4.5.3
openmpi-4.1.2

```

We downloaded the scripts and data from here [SMOKE-TestCase](https://github.com/CEMPD/SMOKE-TestCase/)

```auto
git clone -b main https://github.com/CEMPD/SMOKE-TestCase.git SMOKE-TestCase

```

However, when we tried the first sector nonpoint, we got the following error:

```auto
SMOKE-TestCase/scripts/nonpoint/Annual_np_oilgas_12US1_2017gb_17j_TestCase.csh

(...)
SCRIPT NOTE: No duplicates found in ${SMOKEHOME}/ge_dat/temporal/atref_2017platform_np_oilgas_ONLY_2017_27apr2020_nf_v1
Running part 1...

SCRIPT NOTE: Automatically deleting log file.
      ${SMOKEHOME}/SMOKE-TestCase/intermed/np_oilgas/logs/smkinven_np_oilgas_SMOKE-TestCase.log

forrtl: severe (168): Program Exception - illegal instruction
Image PC Routine Line Source
smkinven 0000000000659A0A Unknown Unknown Unknown
libpthread-2.31.s 000014F9438B5140 Unknown Unknown Unknown
smkinven 00000000007F9629 Unknown Unknown Unknown
smkinven 00000000005721A9 nameval_ 128 nameval.c
smkinven 00000000005B37C0 initlog3_ 86 initlog3.F
smkinven 000000000055C7AB init3_ 166 init3.F90
smkinven 000000000045D34B MAIN__ 183 smkinven.f
smkinven 000000000040565E Unknown Unknown Unknown
libc-2.31.so 000014F9436FAD0A __libc_start_main Unknown Unknown
smkinven 0000000000405569 Unknown Unknown Unknown
0.000u 0.005s 0:00.00 0.0% 0+0k 0+0io 0pf+0w
SCRIPT ERROR: timetracker script found multiple existing entries
              for the following primary keys in the file already:
      Sector = np_oilgas
      Job Name = Annual_area
      Program = smkinven
      Run Date = inv
      This should not happen, so script does not know how to
      replace the entries for these keys
ERROR: Problem calling timetracker from smk_run script
ERROR: Running smk_run for one-time steps

```

We are new using CMAQ-SMOKe, any help will be very welcome

Solange and Lluís

---

<div class="post-metadata">

**Author:** ![chef](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@chef](https://forum.cmascenter.org/u/chef)\
**Post date:** [November 2, 2022, 6:29pm UTC](https://forum.cmascenter.org/t/program-exception-when-running-smoke-testcase/3680/2 "2022-11-02T18:29:48Z")

</div>

Hello,

It is not clear to me if you have precisely followed ALL the steps listed in the github page:

> **[GitHub - CEMPD/SMOKE-TestCase: Scripts for the SMOKE Test Case](https://github.com/CEMPD/SMOKE-TestCase/)**
>
> Scripts for the SMOKE Test Case. Contribute to CEMPD/SMOKE-TestCase development by creating an account on GitHub.

Specifically did you download the data from googledrive, have them at the right directory as pointed there, made all changes to runscripts?

If so, please post your log from the command at step 4.

Also, I assume that your other thread is on the same issue/question? Lets keep one for future reference (you can always update a question/thread in this forum, no need to keep creating new).

---

<div class="post-metadata">

**Author:** ![SolL](https://avatars.discourse-cdn.com/v4/letter/s/9fc348/32.png) [@SolL](https://forum.cmascenter.org/u/SolL)\
**Post date:** [November 2, 2022, 6:54pm UTC](https://forum.cmascenter.org/t/program-exception-when-running-smoke-testcase/3680/3 "2022-11-02T18:54:33Z")

</div>

Thank you for your answer,

We open the other post, because we thought is another topic of importance to left it clear.

Yes, we understand we followed all the steps and we have the files from the drive at the right spot.

We attach the  
[run\_nonpoint.txt](https://forum.cmascenter.org/uploads/short-url/1165ZnSn7NdRc73y0ynS8mA0NX9.txt) (4.6 KB)  
log

---

<div class="post-metadata">

**Author:** ![chef](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@chef](https://forum.cmascenter.org/u/chef)\
**Post date:** [November 2, 2022, 8:20pm UTC](https://forum.cmascenter.org/t/program-exception-when-running-smoke-testcase/3680/4 "2022-11-02T20:20:03Z")

</div>

Thanks for the log file.  
I think that the underlying issue may be the same in both threads and related to the way the source command is invoked in your system.

Specifically, in my logs, I do not see the 8th line of your logs:  
source: shell built-in command.

How do you try to run these scripts? Are you using slurm/sbatch command?  
Do you have both bash and tcsh in your system? Which is the default?

---

<div class="post-metadata">

**Author:** ![eyth.alison](https://avatars.discourse-cdn.com/v4/letter/e/7ab992/32.png) [@eyth.alison](https://forum.cmascenter.org/u/eyth.alison)\
**Post date:** [November 3, 2022, 12:25pm UTC](https://forum.cmascenter.org/t/program-exception-when-running-smoke-testcase/3680/5 "2022-11-03T12:25:00Z")

</div>

Noting that you need to be running in C-shell for the source command to work.

At the top of the scripts, there may be something like #!/bin/csh

If csh doesn’t exist in /bin on your system, there could be problems. Or, if the paths are set improperly to the file to be sourced that would be a problem

---

<div class="post-metadata">

**Author:** ![cjcoats](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@cjcoats](https://forum.cmascenter.org/u/cjcoats)\
**Post date:** [November 3, 2022, 12:32pm UTC](https://forum.cmascenter.org/t/program-exception-when-running-smoke-testcase/3680/6 "2022-11-03T12:32:28Z")

</div>

A minor caveat here: RedHat is trying to change the directory-standard so that shells are in directory _/usr/bin_ instead of _/bin_ – e.g., _/usr/bin/tcsh_. Also note that (the full path for) _tcsh_ must be in the _/etc/shells_ allowed-shells file…

---

<div class="post-metadata">

**Author:** ![SolL](https://avatars.discourse-cdn.com/v4/letter/s/9fc348/32.png) [@SolL](https://forum.cmascenter.org/u/SolL)\
**Post date:** [November 9, 2022, 1:59pm UTC](https://forum.cmascenter.org/t/program-exception-when-running-smoke-testcase/3680/7 "2022-11-09T13:59:13Z")

</div>

Thank you for your suggestions.

At this stage, we are not using any queue system to perform our tests. Therefore we are running directly on the terminal.

Our O.S. in th HPC that we are using (remember no queue system is being used):  
`Linux hydra 5.10.0-16-amd64 #1 SMP Debian 5.10.127-1 (2022-06-30) x86_64 GNU/Linux`

Our IT staff linked `/bin/tsch` as `/bin/csh`

We created a very tiny `tcsh` script to test our system: called `test.tsch`

```auto
#!/bin/csh
   set var = '1234'

```

In our HPC by default uses `bash`. Running the test:

```auto
~$ source ./test.tsch 
~$ echo $var

~$ csh
~% source test.tsch 
~% echo $var
1234

```

Do you think, that there maybe any other even more simple test (than the SMOKE-TestCase) that we could perform?

Thanks

---

<div class="post-metadata">

**Author:** ![cjcoats](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@cjcoats](https://forum.cmascenter.org/u/cjcoats)\
**Post date:** [November 9, 2022, 2:20pm UTC](https://forum.cmascenter.org/t/program-exception-when-running-smoke-testcase/3680/8 "2022-11-09T14:20:27Z")

</div>

You need to understand the difference between _set_ and _setenv_:

- _set_ establishes a variable within the shell-executable, invisible outside the shell-executable itself.
- _setenv_ establishes a variable at the operating-system level, visible to all executables launched afterwards.  
You need variables visible within the CMAQ-executable (etc.), hence you must use _setenv_.

---

<div class="post-metadata">

**Author:** ![SolL](https://avatars.discourse-cdn.com/v4/letter/s/9fc348/32.png) [@SolL](https://forum.cmascenter.org/u/SolL)\
**Post date:** [November 9, 2022, 2:56pm UTC](https://forum.cmascenter.org/t/program-exception-when-running-smoke-testcase/3680/9 "2022-11-09T14:56:41Z")

</div>

Thank you for the answer, certainly make sense what you say.

but we did not write scripts, we are directly using the scripts as they come for the SMOKE-TestScase. We tested as you mention, we modified the `test.tsch`:

```auto
#!/bin/csh
   echo "Previous var: "${var}" varRoman: "${varRoman}
   set var = '1234'
   setenv varRoman 'MCCXXXIV'

```

And using it, it works as expected:

```auto
$ source ./test.tsch 
Previous var: varRoman: 
-bash: setenv: command not found
~$ csh
~% source ./test.tsch
var: Undefined variable.
~% echo ${var}
var: Undefined variable.
~% vim ./test #(remove the first echo)
hydra:~% source ./test.tsch
hydra:~% echo $var
1234
hydra:~% echo $varRoman
MCCXXXIV
hydra:~% vim ./test.tsch # (add again the fist echo)
hydra:~% source ./test.tsch
Previous var: 1234 varRoman: MCCXXXIV

```

As @chef mentions, in our log we do have the line:  
`source: shell built-in command.`

But, it does not appear in the direct simple tests.

Thanks

---

<div class="post-metadata">

**Author:** ![SolL](https://avatars.discourse-cdn.com/v4/letter/s/9fc348/32.png) [@SolL](https://forum.cmascenter.org/u/SolL)\
**Post date:** [December 14, 2022, 12:39pm UTC](https://forum.cmascenter.org/t/program-exception-when-running-smoke-testcase/3680/10 "2022-12-14T12:39:17Z")

</div>

Dear all,

Thank you for all your responses.

To summarize, we downloaded the SMOKE-TestCase and we can not make it run in our HPC.

We got some news.

We discovered, that in our machine me were missing some system utilities:

- GNU arbitrary precision calculator language [bc](https://packages.debian.org/bullseye/bc)
- classic UNIX line editor [ed](https://packages.debian.org/bullseye/ed)

Also, we forgot to mention, that we already had to introduce some tiny modifications into the code and / scripts to make it work:

- linea #1 of `smoke4.7/scripts/log_analyzer/log_analyzer.py`, `smoke4.7/scripts/run/path_parser.py`, `smoke4.7/scripts/annual_report/annual_report_v2.py`, we introduced:

```auto
#!/usr/bin/env python3

```

- When compiling IOAPI with gcc, we need also to modify a line of code: ioapi/v3.2-20200828/gnu/ioapi/fixed\_src/PARMS3.EXT, in line #104:

```auto
! INTEGER, PARAMETER :: M3TYPES( NM3TYPES ) = &
! & (/ M3INT, M3REAL, M3DBLE, M3INT8 /)
        INTEGER, PARAMETER :: M3TYPES( NM3TYPES ) = (/ M3INT, M3REAL, M3DBLE, M3INT8 /)

```

- Due to the use of gcc compilation, we also need to change `smoke4.7/scripts/platform` by default it is prepared for intel, has to be changed:

```auto
#set comp_abbrv = ifort 
set comp_abbrv = gcc

```

We got an error message with the point sector of the test case when is using the `smkreport.f90`

```auto
(...)
malloc(): corrupted top size
(...)

```

So, we recompile SMOKE in debug mode (we couldn’t compile IOAPI in debug mode, some compiling errors related to flags [`-Wtabs` related] were encountered), and re-running again. In doing that, we encountered the point where the error happens (now also happening with nonpoint, aside of the point sectors):

- nonpoint:

```auto
(...)
At line 1902 of file /home/solange.luque/MODELOS/SMOKE_dbg/gcc/src/emqa/wrrephdr.f
Fortran runtime error: Index '20' of dimension 1 of array 'outdnam' above upper bound of 19

Error termination. Backtrace:
#0 0x14e579400bd0 in ???
#1 0x14e579401685 in ???
#2 0x14e579401c76 in ???
#3 0x55e8e8e454dc in wrrephdr_
        at /home/solange.luque/MODELOS/SMOKE_dbg/gcc/src/emqa/wrrephdr.f:1902
#4 0x55e8e8e3a2e9 in smkreport
        at /home/solange.luque/MODELOS/SMOKE_dbg/gcc/src/emqa/smkreport.f:406
#5 0x55e8e8df5978 in main
        at /home/solange.luque/MODELOS/SMOKE_dbg/gcc/src/emqa/smkreport.f:47
     SMOKE ---------------
     Copyright (c)2004 Environmental Modeling for Policy Development
(...)

```

- point:

```auto
(...)
At line 560 of file /home/solange.luque/MODELOS/SMOKE_dbg/gcc/src/emqa/bldrepidx.f
Fortran runtime error: Index '20' of dimension 1 of array 'outdnam' above upper bound of 19

Error termination. Backtrace:
#0 0x14f1d4b2bbd0 in ???
#1 0x14f1d4b2c685 in ???
#2 0x14f1d4b2cc76 in ???
#3 0x55d128c00dac in bldrepidx_
        at /home/solange.luque/MODELOS/SMOKE_dbg/gcc/src/emqa/bldrepidx.f:560
#4 0x55d128c2e775 in smkreport
        at /home/solange.luque/MODELOS/SMOKE_dbg/gcc/src/emqa/smkreport.f:179
#5 0x55d128bea978 in main
        at /home/solange.luque/MODELOS/SMOKE_dbg/gcc/src/emqa/smkreport.f:47
     SMOKE ---------------
(...)

```

Digging into the code, the lines with the problem are related with an index named `J` of `OUTDNAM( J,RCNT ) ` (from line #1902 of `src/emqa/wrrephdr.f`)  
and index `J` of `OUTDNAM( J,N ) = SBUF` (from line #560 of `src/emqa/bldrepidx.f`).

We tried to understand the definition of `J` and the size of `OUTDAM`, but we couldn’t fine any clue.

Any help will be very welcome.

Solange