Error running CMAQv5.4_2018_12NE3_Benchmark_2Day_Input

CMAQ Benchmark Issues and Errors
1

Hi everyone. I am trying to run the WRF-CMAQ benchmark mentioned in the title to test my configuration, that is:

  • NetCDF-C 4.8.1, NetCDF-Fortran 4.5.4 (together in a single folder) and I/O API 3.2, compiled following this tutorial;

  • WRF 4.5.2 coupled with CMAQ 5.5, compiled following this tutorial.

I’m also using gcc, g++ and gfortran 9.5.0, as well as OpenMPI 5.0.7, running on Ubuntu 24.04 LTS.

I was able to compile all the libraries and the coupled model succesfully, obtaining the executables. I’m using the attached script to run the benchmark
run_cctm_Bench_2018_12NE3.WRFCMAQ.csh (51.4 KB).

The WRF model starts running together with CMAQ just fine, until it reaches the third simulation hour: the output for hour 3 is written, but the simulation crashes with a “Segmentation Fault - Invalid Memory reference” error. I suspect there is something wrong with the emission files (I am currently using the ones suggested by the tutorial): CMAQ probably tries to read them at hour 3 but fails, for some reason. All the paths to the libraries and the folders seem to be set correctly (in the script), as well as the necessary environment variables.

I also attached the first rsl.error rsl.error.0000.txt (670 KB) and rsl.out
rsl.out.0000.txt (377.3 KB) files , as well as the benchmark output cmaq_log.txt (3.1 KB).

Any help is greatly appreciated.

2

Can you compile everything with -g -O0 -fbacktrace and re-run?

This should give the routine and line-number at which the SEGFAULT occurs (you need the -g -O0 in addition to the -fbacktrace because optimization can “play games” with the line-numbers at which the problem happens while the compiler is doing such things as loop-optimization).

3

If you are using csh or tcsh, please also try this:

limit stacksize unlimited

If you are using sh or bash, use this command:

ulimit -s unlimited

This may not solve your problem but the default stack size is often quite small and may result in segmentation faults due to insufficient stack memory.

4

Thank you @cjcoats and @lizadams for your answers. I recompiled following your suggestions, but the benchmark keeps crushing. To be thorough, I’m attaching the relative rsl.out rsl.out.0001.txt (7.4 KB) and rsl.errorrsl.error.0001.txt (4.3 KB) files (only the ones that seem to show some more information). However, I found that this same benchmark runs smoothly if recompiling CMAQ 5.5 using WRF 4.5.1, instead of any newer version: this behaviour may be related to what is explained in this post from the official repository. I actually didn’t try using the workaround suggested there, but may be doing so in the near future.

5

Given the lack of traceback, it is clear that you failed to run such a debug/traceback executable.

  • Make sure you compiled the entire system for debug/traceback.

  • Check the scripts very carefully to ensure that they did not over-ride the choice of executable that you wanted.

6

Your issue very likely is related to the use of WRF > v4.5.1 to build the twoway model. If sticking with WRFv4.5.1 (which you confirmed is running for you) is not an option for you, you will need to follow the steps described in our documentation to update your copy of WRFv4.5.2 with the two specific pieces of WRFv4.5.1 code (WRFv451/share/wrf_tsin.F and WRFv451/frame/module_domain_type.F).