CMAQv5.3 2-day benchmark index error

jdeast2 · September 25, 2019, 8:08pm

Hi all,

I am running the 2-day benchmark for CMAQv5.3. The 1st day successfully runs, but the 2nd day fails almost immediately with an indexing error “NCVGT: : NetCDF: Index exceeds dimension bound”

The last statement in all the CTM log files is:

 "E2C_CHEM_YEST" opened as OLD:READ-ONLY   
 File name "/share/fgarcia4/jdeast2/cmaq-5.3-intel2017-BENCH/CMAQ_REPO/data/2016_12SE1//land/epic_festc1.4_20180516/2016_US1_time20160701_bench.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "SE53BENCH"
 Dimensions: 80 rows, 100 cols, 42 lays, 41 vbles
 NetCDF ID:   2555904  opened as READONLY            
 Starting date and time  2016183:000000 (0:00:00   July 1, 2016)
 Timestep                          240000 (24:00:00 hh:mm:ss)
 Maximum current record number         1
 Checking header data for file: E2C_CHEM_YEST
     Inconsistent values for NLAYS: 42 versus 35

This seems as if the E2C_CHEM_YEST file has mismatching dimensions, but the file for day 1 has the same dimensions and there is no error in day 1. The inconsistent NLAYS does not seem to cause the model to fail when it happens in other files.

Any help solving this error is appreciated! My run script is attached. There are only minor changes to paths from the run script packaged with the benchmark data.

Thanks,
James
run_cctm_Bench_2016_12SE1.csh (31.8 KB)

zyj.jess · September 26, 2019, 1:38pm

Hi, James;

I encountered with this NCVGT error many times. The scenarios I was in when I saw this error message was the dimensions of my inputs file were not matching each other. Especially, I had this error when my initial condition or boundary condition had a different dimension such as time step. It could be a hint for you to check the time step of this soil file or other input files. As far as I know, once the dimension issue is fixed, you should have no problem at this point.

Hope this helps!
Best;
Yijia

lizadams · September 26, 2019, 4:25pm

Please share a copy of your CTM_LOG_000* file.
It should be located in the following directory:
data/output_CCTM_v53_gcc_Bench_2016_12SE1/LOGS

jdeast2 · September 26, 2019, 5:40pm

Thank you! In this case, the dimensions of the input boundary and initial conditions files match (80x100x35). Besides the inconsistent NLAYS, all input files appear to have 80x100 except DOT met files and point emissions

jdeast2 · September 26, 2019, 5:43pm

Here is the CTM_LOG_000 file:

Thank you!

CTM_LOG_000.v53_intel2017_Bench_2016_12SE1_20160702.txt (49.6 KB)

lizadams · September 26, 2019, 6:18pm

Can you use the following command to search for an ERROR statement in all of the log files:
grep -i ERROR CTM_LOG_*

Is there an error message in any of the other CTM_LOG files?

The warning that you are seeing would not cause the model to fail.

jdeast2 · September 30, 2019, 12:08pm

Searching with that command shows no error messages in any of the log files.

Additionally, here’s the output created by the system job scheduler that contains the actual error and some other output not contained in the LOG files.

Should any changes to the benchmark run script provided with the CMAQ distribution have to be made to run the simulation? I had to make a few changes.

out.cmq53.761790.txt (238.9 KB)

lizadams · September 30, 2019, 1:21pm

Hi James,

You should not have a problem running the benchmark version with the scripts provided.

However, it looks like you are using an older version of the I/O API library.
Please try downloading and using the latest version. Here is the information on the version that I am using:

ioapi-3.2: $Id: init3.F90 120 2019-06-21 14:18:20Z coats $
 Version with PARMS3.EXT/PARAMETER::MXVARS3= 2048
 netCDF version 4.7.0 of Sep 16 2019 15:07:57 $

I followed these instructions to download and install.

github.com

USEPA/CMAQ/blob/master/DOCS/Users_Guide/Tutorials/CMAQ_UG_tutorial_build_library_intel.md

## Install netCDF-C

1. Download netCDF-C from the following website https://www.unidata.ucar.edu/downloads/netcdf/index.jsp

```
wget ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-c-4.7.0.tar.gz
```

2. Untar the file 

```
tar -xzvf netcdf-c-4.7.0.tar.gz
```

3. Change directories into the package
```
cd netcdf-c-4.7.0
```

3. Verify that no modules are currently loaded using module commands.

This file has been truncated. show original

jdeast2 · September 30, 2019, 7:16pm

Hi Liz,

I’m using IOAPI-3.2, which matches yours. Where do you see that I have an old version? Perhaps something is linked incorrectly for me.

I’ve tried several things but still encountering the same error:

NCVGT: : NetCDF: Index exceeds dimension bound

Is it possible that it is related to time steps in the simulation?

Thanks,
James

lizadams · October 1, 2019, 6:04pm

Hi James,

You are correct that we have the same Models-3 I/O API version number, but the dates are different.
In your out.cmq53.761790.txt file is the following output:

   ioapi-3.2: $Id: init3.F90 98 2018-04-05 14:35:07Z coats $
     netCDF version 4.6.1 of Nov 12 2018 11:27:53 $

jdeast2 · October 1, 2019, 6:06pm

Hi Liz,

I installed the most recent version of ioapi available and both days ran successfully.

The version used now matches yours:

ioapi-3.2: Id: init3.F90 120 2019-06-21 14:18:20Z coats

Thank you for your help!

thayna.gomes · December 4, 2019, 1:07am

Dear @lizadams

I’m also having problem running CMAQ v5.3. However, I’m did not find any message that may point to the error.
Do you think that if we update I/O API library it would run succesfully?

Please find attached run script and log files.

If you find any possible mistake, please let me know. I would appreciate the support.

Thank you in advance!

maplewind · December 4, 2019, 2:35am

Hi Liz,
Is it necessary to run CMAQ v5.3 with using IOAPI-3.2.
When I ran CMAQ v5.3 with IOAPI-3.1 ,there are no distinct errors in log files.

Thanks！

thayna.gomes · December 4, 2019, 7:47pm

What about version of I/O API-3.2?
I’m running with:
ioapi-3.2: Id: init3.F90 3 2017-04-15 20:19:16Z coats
netCDF version 4.4.1.1 of Aug 7 2017 18:04:11 $

I’m having trouble as commented in my post.

Thank you!

cjcoats · December 4, 2019, 8:35pm

It is conceivable that you can get rid of this by getting rid of all the executable, object, module, and library files, and then re-building from scratch.

The October 16 I/O API release increased the maximum-files parameter MXFILE3 from 64 to 256 at the request of EPA. This increase should have been transparent to any application which does not include/use I/O API-internal/private data structures in STATE3.EXT – doing so is forbidden, but CMAQ’s “pario” uses these anyway. This may be the cause of the problem.

lizadams · December 4, 2019, 9:45pm

I am not familiar with the error that is reported in your log_run_cctm_2010_RMGV.txt
"forrtl: severe (27): too many records in I/O statement, unit -5, file Internal Formatted "

Can you change the run script to set the CLOBBER_DATA to true and try running again?

#> Keep or Delete Existing Output Files
set CLOBBER_DATA = TRUE

thayna.gomes · December 5, 2019, 11:57am

Dear @lizadams

Thank you for the suggestion, but I am still having the same problem.
Do you any other suggestion?
I really appreciate the support!

cmaq_log_file.txt (22.3 KB) run_cctm_2010_RMGV.csh (33.7 KB) CTM_LOG_000.txt (42.6 KB)

cjcoats · December 5, 2019, 1:37pm

Re-compile the code with “-traceback” and that will at least indicate the source-file and line at which the error occurs. I would not be surprised if the error happens due to a bad format during the construction of an error-message for the “real” error; if so, then fixing that format might then help to get the real error-message out to the log, and we can go from there (two-stage process, sorry).