# Program received signal SIGABRT: Process abort signal **URL:** \ **Category:** CMAQ\ **Created:** [May 5, 2024, 7:53pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913 "2024-05-05T19:53:26Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![azurebullet](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@azurebullet](https://forum.cmascenter.org/u/azurebullet)\ **Post date:** [May 5, 2024, 7:53pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/1 "2024-05-05T19:53:26Z")
Hello, I am trying to do a simulation in a small domain with CONUS data. I did my own wrf simulation and use MCIP, BCON and ICON to prepare the emission files to do the simulation. I faced the issue when I began to start CCTM. You can check the log files [run.cctm.txt](https://forum.cmascenter.org/uploads/short-url/fzcZ2wa2cTCCvk0hnyjAyl7hDKx.txt) (51.1 KB) [CTM\_LOG\_000.v54\_cb6r5\_ae7\_aq\_WR413\_MYR\_2019mid\_20190802.txt](https://forum.cmascenter.org/uploads/short-url/5TPdzT2xVr8hHzRdhSUQvGS5UPD.txt) (30.3 KB) [CTM\_LOG\_001.v54\_cb6r5\_ae7\_aq\_WR413\_MYR\_2019mid\_20190802.txt](https://forum.cmascenter.org/uploads/short-url/euhKIrxEeGpvVinGSnnJzuSNMeE.txt) (30.3 KB) [CTM\_LOG\_002.v54\_cb6r5\_ae7\_aq\_WR413\_MYR\_2019mid\_20190802.txt](https://forum.cmascenter.org/uploads/short-url/u2oyYnPqatE1QwzqZ51eMSAH6Vg.txt) (30.3 KB) [CTM\_LOG\_003.v54\_cb6r5\_ae7\_aq\_WR413\_MYR\_2019mid\_20190802.txt](https://forum.cmascenter.org/uploads/short-url/8j0fCAtWP99naIHVGUFIJhF5cKi.txt) (30.3 KB) ---
**Author:** ![hogrefe.christian](https://avatars.discourse-cdn.com/v4/letter/h/74df32/32.png) [@hogrefe.christian](https://forum.cmascenter.org/u/hogrefe.christian)\ **Post date:** [May 6, 2024, 11:35am UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/2 "2024-05-06T11:35:06Z")
You mentioned that you ran MCIP to process your WRF simulation with 44 layers and then ran BCON and ICON to prepare boundary and initial conditions for CCTM using this 44 layer structure. However, the CTM\_LOG files you posted show that your run actually tries to use the 35-layer file EQUATES restart file CCTM\_CGRID\_v532\_cb6r3\_ae7\_aq\_WR413\_MYR\_STAGE\_2019\_12US1\_20190801.nc as initial conditions, i.e. file INIT\_CONC\_1. This will cause an array mismatch error when trying to read the initial conditions that seems to be consistent with the traceback messages you posted. Assuming that the initial and boundary condition files you prepared with BCON and ICON indeed have the correct 44 layer structure, make sure that your run script points to those files rather than the corresponding EQUATES files that used a 35 layer structure that is incompatible with your WRF setup. ---
**Author:** ![azurebullet](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@azurebullet](https://forum.cmascenter.org/u/azurebullet)\ **Post date:** [May 6, 2024, 1:20pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/3 "2024-05-06T13:20:59Z")
Thanks for your suggestion. I can understand what you mean. To solve the problem, can I rerun my wrf to prepare the 35 layers icon and bcon file for the simulation? ---
**Author:** ![hogrefe.christian](https://avatars.discourse-cdn.com/v4/letter/h/74df32/32.png) [@hogrefe.christian](https://forum.cmascenter.org/u/hogrefe.christian)\ **Post date:** [May 6, 2024, 1:39pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/4 "2024-05-06T13:39:21Z")
If you want to use 44 layers for your simulations, there is no need to rerun WRF, you just need to make sure that you use your 44 layer initial and boundary condition files (which in your first post you said you prepared) as input to your CCTM simulation since you cannot use the existing 35 layer EQUATES initial and boundary conditions (even if you were to change to a 35 layer setup, you’d still have to generate your own boundary conditions since your domain is smaller than the EQUATES domain, but you could use the EQUATES CGRID files as initial conditions). You didn’t provide details on how you prepared these 44 layer initial and boundary conditions files mentioned in your first post, but if you deem them to be appropriate for your project, I see no reason why it would be necessary to change your WRF setup and run WRF again. ---
**Author:** ![azurebullet](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@azurebullet](https://forum.cmascenter.org/u/azurebullet)\ **Post date:** [May 6, 2024, 2:10pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/5 "2024-05-06T14:10:42Z")
Thanks for your suggestion. I will try to try to use 44-layer files for my simulation. Another question is that what does “EQUATES CGRID” mean. Is that the concentration file with 35-layes structure? My understanding is that the CGRID file is not required to modify for my domain as initial conditions but as boundary condition I still need to prepare by myself. Is that correct? ---
**Author:** ![hogrefe.christian](https://avatars.discourse-cdn.com/v4/letter/h/74df32/32.png) [@hogrefe.christian](https://forum.cmascenter.org/u/hogrefe.christian)\ **Post date:** [May 6, 2024, 2:41pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/6 "2024-05-06T14:41:07Z")
Your log file shows that you are using input files from the [EQUATES project](https://www.epa.gov/cmaq/equates) (e.g. CCTM\_CGRID\_v532\_cb6r3\_ae7\_aq\_WR413\_MYR\_STAGE\_2019\_12US1\_20190801.nc and emis\_mole\_all\_20190802\_12US1\_nobeis\_norwc\_WR413\_MYR\_2019.nc), probably downloaded from the CMAS data warehouse where we make that data available. The CGRID file (CCTM\_CGRID\_v532\_cb6r3\_ae7\_aq\_WR413\_MYR\_STAGE\_2019\_12US1\_20190801.nc) is the file your run script specifies as initial conditions, based on your log file: ``` "INIT_CONC_1" opened as OLD:READ-ONLY File name "/scratch/mnpfubl/Build_CMAQ/CMAQ_project/data/2019_08/icbc/CCTM_CGRID_v532_cb6r3_ae7_aq_WR413_MYR_STAGE_2019_12US1_20190801.nc" File type GRDDED3 Execution ID "CMAQ_CCTMv532_kappel_20220521_150314_338864756" Grid name "12US1" Dimensions: 299 rows, 459 cols, 35 lays, 224 vbles NetCDF ID: 589824 opened as READONLY Starting date and time 2019214:000000 (0:00:00 Aug. 2, 2019) Timestep 010000 (1:00:00 hh:mm:ss) ``` Yes, the CGRID file contains species concentrations for all species and model grid cells and is written at the very end of a CCTM simulation. It can therefore be used as initial conditions for the next time period following the previous CCTM simulation, instead of using an initial conditions file prepared by ICON to initialize the model. CCTM can automatically horizontally [window gridded files](https://github.com/USEPA/CMAQ/blob/main/DOCS/Users_Guide/CMAQ_UG_ch04_model_inputs.md#431-windowing-capability) like CGRID or emissions (in your case, EQUATES input files with 299 rows and 459 columns) to a smaller domain (like your subdomain with 125 rows and 125 columns), as long as they share the same projection and the subdomain is fully contained in the larger domain. However, windowing does not apply to the vertical dimension. The vertical dimension for your CCTM simulation is defined by your MCIP files, and the initial and boundary condition files need to match that vertical structure. ---
**Author:** ![azurebullet](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@azurebullet](https://forum.cmascenter.org/u/azurebullet)\ **Post date:** [May 6, 2024, 2:56pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/7 "2024-05-06T14:56:54Z")
Thanks for your reply. I realized that I have prepared my ICON file, but it was not used in my script. I have modified them and run again. However, I still get the similar errors. Here are the log files for this time. [CTM\_LOG\_000.v54\_cb6r5\_ae7\_aq\_WR413\_MYR\_2019mid\_20190802.txt](https://forum.cmascenter.org/uploads/short-url/iNXa8egz9WLUv1r0jr4UKuE3NUc.txt) (30.3 KB) [run.cctm.log.txt](https://forum.cmascenter.org/uploads/short-url/86Ju0YHhRZEid7WSR7hx37tgYbB.txt) (51.1 KB) ---
**Author:** ![hogrefe.christian](https://avatars.discourse-cdn.com/v4/letter/h/74df32/32.png) [@hogrefe.christian](https://forum.cmascenter.org/u/hogrefe.christian)\ **Post date:** [May 6, 2024, 3:18pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/8 "2024-05-06T15:18:06Z")
Thanks for confirming what I mentioned in my first post, i.e. that your run was not using your own 44 layer initial condition file you thought you were using and instead used the downloaded 35 layer CGRID file that is incompatible with your 44 layer MCIP files. While I had hoped that that mismatch might have explained the crash, that doesn’t appear to be the case, since in your new run you are clearly using your own 44 layer initial condition file. I’m not sure what to suggest next - the traceback message points to some memory issue, but I don’t know what the cause could be. I noticed you are only using 4 processors. Are you able to use more, and if so, do you encounter the same error? Did you try to run the benchmark case, and were you successful in doing so? ---
**Author:** ![azurebullet](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@azurebullet](https://forum.cmascenter.org/u/azurebullet)\ **Post date:** [May 6, 2024, 3:51pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/9 "2024-05-06T15:51:42Z")
I can give more cores. I will give it a try and I hope it could be fine. Thanks for your help ---
**Author:** ![azurebullet](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@azurebullet](https://forum.cmascenter.org/u/azurebullet)\ **Post date:** [May 6, 2024, 6:40pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/10 "2024-05-06T18:40:09Z")
I have successfully ran benchmark case. I use 32 cores to give it a try. The error is still the same. I am not sure what I should do now. Do I need to rebuild my library or do something else? ---
**Author:** ![hogrefe.christian](https://avatars.discourse-cdn.com/v4/letter/h/74df32/32.png) [@hogrefe.christian](https://forum.cmascenter.org/u/hogrefe.christian)\ **Post date:** [May 6, 2024, 7:00pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/11 "2024-05-06T19:00:03Z")
Thanks for performing this test and reporting back. If you haven’t compiled CCTM in debug mode, I would give that a try next. To do that, go to your BLD directory, type “make clean”, followed by “make DEBUG = TRUE”. Then execute the run script again and see if there are any error messages pointing to specific lines in the CCTM code, rather than the traceback messages pointing to shared object libraries. ---
**Author:** ![azurebullet](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@azurebullet](https://forum.cmascenter.org/u/azurebullet)\ **Post date:** [May 6, 2024, 7:21pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/12 "2024-05-06T19:21:01Z")
I have a question. I used the bld\_cctm.csh to build it. If I want to compile with debug mode, should I modify some thing in the scripts?(Never mind, I have found one option in the file ---
**Author:** ![hogrefe.christian](https://avatars.discourse-cdn.com/v4/letter/h/74df32/32.png) [@hogrefe.christian](https://forum.cmascenter.org/u/hogrefe.christian)\ **Post date:** [May 6, 2024, 7:31pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/13 "2024-05-06T19:31:24Z")
Yes, you can use the bldit\_cctm.csh script to compile in debug mode, you just have to uncomment ‘#set Debug\_CCTM’. But if you already have an existing BLD directory and want to rebuild the executable in debug mode there, you can use the approach listed in my previous post. ---
**Author:** ![azurebullet](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@azurebullet](https://forum.cmascenter.org/u/azurebullet)\ **Post date:** [May 6, 2024, 7:48pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/14 "2024-05-06T19:48:21Z")
Hi, Christian. I have compiled a debug version CCTM. The problem is that where I can find other error messages. I can only find same log files as before. ---
**Author:** ![hogrefe.christian](https://avatars.discourse-cdn.com/v4/letter/h/74df32/32.png) [@hogrefe.christian](https://forum.cmascenter.org/u/hogrefe.christian)\ **Post date:** [May 6, 2024, 8:00pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/15 "2024-05-06T20:00:40Z")
The hope was that by compiling in debug mode and rerunning the script, the error message in the main log file (run.cctm.txt) would change to something more concrete, and/or that the processor-specific log files (CTM\_LOG\_…) would show something after opening the E2C\_SOIL file. ---
**Author:** ![azurebullet](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@azurebullet](https://forum.cmascenter.org/u/azurebullet)\ **Post date:** [May 6, 2024, 8:54pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/16 "2024-05-06T20:54:21Z")
I have rerun with the debug mode CCTM. I cannot find any changes. Should I use bsub(a tool to submit job in our university super computer like slurm) maybe? [CTM\_LOG\_000.v54\_cb6r5\_ae7\_aq\_WR413\_MYR\_2019mid\_20190802.txt](https://forum.cmascenter.org/uploads/short-url/8ghEElWleIXErvSK3kq4WkbJrey.txt) (30.3 KB) [run.cctm.log.txt](https://forum.cmascenter.org/uploads/short-url/9cfqMwHnZhUn4ESPspabYbRQyFm.txt) (89.7 KB) ---
**Author:** ![hogrefe.christian](https://avatars.discourse-cdn.com/v4/letter/h/74df32/32.png) [@hogrefe.christian](https://forum.cmascenter.org/u/hogrefe.christian)\ **Post date:** [May 6, 2024, 9:06pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/17 "2024-05-06T21:06:04Z")
You could try, yes, but if you successfully ran the benchmark case on the same system which you used for your tests so far, I’m not very hopeful. I don’t have any other suggestions, though, so you might as well try. Maybe others can think of what else to try. ---
**Author:** ![hogrefe.christian](https://avatars.discourse-cdn.com/v4/letter/h/74df32/32.png) [@hogrefe.christian](https://forum.cmascenter.org/u/hogrefe.christian)\ **Post date:** [May 6, 2024, 11:36pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/18 "2024-05-06T23:36:52Z")
One additional test you could try is setting the `CTM_OCEAN_CHEM` and maybe also the `CTM_ABFLUX` run script flags to F, just to see if opening their required input files windowed for your smaller domain is causing any issues. ---
**Author:** ![azurebullet](https://avatars.discourse-cdn.com/v4/letter/a/f1d935/32.png) [@azurebullet](https://forum.cmascenter.org/u/azurebullet)\ **Post date:** [May 7, 2024, 6:31pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/19 "2024-05-07T18:31:13Z")
Hi, Christian. I have tried to disable CTM\_OCEAN\_CHEM and CTM\_ABFLUX. The script can run through the input file process. However I got another error. I will also attached my log files here. I try to modify the cores number and the memory usage. It will show different error. [CTM\_LOG\_001.v54\_cb6r5\_ae7\_aq\_WR413\_MYR\_2019mid\_20190802.txt](https://forum.cmascenter.org/uploads/short-url/E7l0g9Ko44urEdj7z4MoCqC5DF.txt) (43.4 KB) [run.cctm.log.txt](https://forum.cmascenter.org/uploads/short-url/uyIg6TPCwdC1Ss4oy6ZqBlYVtDY.txt) (126.4 KB) [CTM\_LOG\_000.v54\_cb6r5\_ae7\_aq\_WR413\_MYR\_2019mid\_20190802.txt](https://forum.cmascenter.org/uploads/short-url/gF9CT2cCU6fJ3AKHlMiMJGyMF8C.txt) (56.2 KB) ---
**Author:** ![fsidi](https://avatars.discourse-cdn.com/v4/letter/f/7feea3/32.png) [@fsidi](https://forum.cmascenter.org/u/fsidi)\ **Post date:** [May 7, 2024, 7:05pm UTC](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913/20 "2024-05-07T19:05:54Z")
@azurebullet, After talking to @hogrefe.christian about your issue could you upload a few additional things: 1. An [ncdump](https://www.unidata.ucar.edu/software/netcdf/workshops/2011/utilities/Ncdump.html) of your METCRO3D (something like `ncdump -h /scratch/mnpfubl/Build_CMAQ/CMAQ_project/data/2019_08/met/METCRO3D_190802d1.nc >> metcro3d_ncdump.log`) 2. A [cat](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/cat.html) of your GRIDDESC file (something like `cat /scratch/mnpfubl/Build_CMAQ/CMAQ_project/data/2019_08/met/GRIDDESC >> GRIDDESC.log`) 3. Lastly, a cat of your Makefile (something like `cat /scratch/mnpfubl/Build_CMAQ/CMAQ_project/CCTM/scripts/BLD_CCTM_v54_gcc_debug/Makefile >> makefile.log`) Looking at your most recent log files suggests that the model cannot determine an appropriate time step for the science processes (vertical diffusion, advection, etc) to synchronize. [Next page](https://forum.cmascenter.org/t/program-received-signal-sigabrt-process-abort-signal/4913.md?page=2)