SMOKEv4.8 SMKMERGE ERROR: Gridding matrix dimension is inconsistent with records count! Delete gridding matrix and recreate it

SMOKE
1

Hi, I was processing MEIC inventory. Programs, including smkinven, grdmat, spcmat, and temporal, worked well. But an error occurred during smkmerge, “ERROR: Gridding matrix dimension is inconsistent with records count!
Delete gridding matrix and recreate it.”. Any thoughts on this error? Thanks.
The complete log file is shown as follows:

                                                                     
     This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
     I/O Applications Programming Interface, [I/O API] which is      
     built on top of the netCDF I/O library (Copyright 1993, 1996    
     University Corporation for Atmospheric Research/Unidata         
     Program) and the PVM parallel-programming library (from         
     Oak Ridge National Laboratory).                                 
     Copyright (C) 1992-2002 MCNC,                                   
     (C) 1992-2018 Carlie J. Coats, Jr.,                             
     (C) 2003-2012 Baron Advanced Meteorological Systems, LLC, and   
     (C) 2014-2019 UNC Institute for the Environment.                
     Released under the GNU LGPL  License, version 2.1.  See URL     
                                                                     
         https://www.gnu.org/licenses/old-licenses/lgpl-2.1.html     
                                                                     
     for conditions of use.                                          
                                                                     
     ioapi-3.2: $Id: init3.F90 136 2019-10-16 13:57:49Z coats $
     Version with PARMS3.EXT/PARAMETER::MXVARS3= 2048
     netCDF version 4.4.1.1 of Apr 13 2017 12:41:52 $
      
      
     Missing environment variable EXECUTION_ID
     SMOKE ---------------
     Copyright (c)2004 Environmental Modeling for Policy Development
     All rights reserved

     Program SMKMERGE, Version SMOKEv4.8_Oct2020
     Online documentation 
         http://www.cep.unc.edu/empd/products/smoke                 
      
     Program SMKMERGE to merge the inventory or hourly emission
     files with gridding matrices and with optionally any
     combination of speciation matrices, multiplicative control
     matrices, additive control matrices, or reactivity control
     matrices. The program can operate on one to four source 
     categories (area, biogenic, mobile, or point sources), or any
     combination of these.  Gridded and/or state reports and/or
     county reports can be written from this program. If a layer-
     fractions file is input, then the total emissions output file
     is three-dimensional.
      
      
     You will need to enter the logical names for the input and
     output files (and to have set them prior to program start,
     using "setenv <logicalname> <pathname>").
      
     You may use END_OF-FILE (control-D) to quit the program
     during logical-name entry. Default responses are given in
     brackets [LIKE THIS] and can be accepted by hitting the
     <RETURN> key.
      
     Value for PROMPTFLAG:  N returning FALSE
     Returning default value TRUE for query:
     "Continue with program?"
     Value for MRG_SOURCE:  'A'
     Value for MRG_SOURCE:  'A'
     Value for MRG_CTLMAT_MULT not defined; returning defaultval ':  ' '
     Value for MRG_CTLMAT_MULT not defined; returning defaultval ':  ' '
     Value for MRG_CTLMAT_REAC not defined; returning defaultval ':  ' '
     Value for MRG_CTLMAT_REAC not defined; returning defaultval ':  ' '
     Value for MRG_BYDAY not defined; returning defaultval ':  ' '
     Value for MRG_BYDAY not defined; returning defaultval ':  ' '
     Value for MRG_TEMPORAL_YN:  Y returning TRUE
     Value for MRG_SPCMAT_YN:  Y returning TRUE
     Value for MRG_METCHK_YN not defined;returning default:   TRUE
     Value for MRG_MARKETPEN_YN not defined;returning default:   TRUE
     Value for MRG_GRDOUT_YN:  Y returning TRUE
     Value for MRG_REPSTA_YN:  Y returning TRUE
     Value for MRG_REPCNY_YN:  Y returning TRUE
     Value for MRG_REPINV_YN not defined;returning default:   TRUE
     Value for MRG_REPSPC_YN not defined;returning default:   TRUE
     Value for MRG_REPCTL_YN not defined;returning default:   FALSE
     Value for USE_VARIABLE_GRID not defined;returning default:   FALSE
     Value for SMK_SRCGROUP_OUTPUT_YN not defined;returning default:   FALSE
     Value for SMK_AVEDAY_YN:  N returning FALSE
     Value for GRIDDESC:  '/home/atmos/MODEL/SMOKE-ifort/data/ge_dat/meic/GRIDDESC_MYD02'
     Value for GRIDDESC:  '/home/atmos/MODEL/SMOKE-ifort/data/ge_dat/meic/GRIDDESC_MYD02'

     File "GRIDDESC" opened for input on unit:  98
     /home/atmos/MODEL/SMOKE-ifort/data/ge_dat/meic/GRIDDESC_MYD02

     Value for IOAPI_GRIDNAME_1:  'MYd02'
     Value for IOAPI_GRIDNAME_1:  'MYd02'

     File "GRIDDESC" opened for input on unit:  98
     /home/atmos/MODEL/SMOKE-ifort/data/ge_dat/meic/GRIDDESC_MYD02

     NOTE: Grid settings initialized using general in  
          grid description file.
     Value for PROMPTFLAG:  N returning FALSE

     File "AREA" opened for input on unit:  98
     /home/atmos/MODEL/SMOKE-ifort/data/inventory/MYD02/area.map.meic.MYD02.txt

     Value for AREA:  '/home/atmos/MODEL/SMOKE-ifort/data/inventory/MYD02/area.map.meic.MYD02.txt'
     Value for AREA:  '/home/atmos/MODEL/SMOKE-ifort/data/inventory/MYD02/area.map.meic.MYD02.txt'

     File "ASRC" opened for input on unit:  97
     /home/atmos/MODEL/SMOKE-ifort/data/inventory/MYD02/meicsrc.MYD02.txt

     Value for IOAPI_CHECK_HEADERS not defined;returning default:   FALSE
      
     "AREA" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/inventory/MYD02/meic.MYD02.ncf"
     File type GRDDED3 
     Execution ID "????????????????"
     Grid name ""
     Dimensions: 2000000 rows, 1 cols, 1 lays, 4 vbles
     NetCDF ID:     65536  opened as READONLY            
     Time-independent data.
      
     "TMP_POL_FILE" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/inventory/MYD02/meic.MYD02_dat/CO.ncf"
     File type GRDDED3 
     Execution ID "????????????????"
     Grid name ""
     Dimensions: 402808 rows, 1 cols, 1 lays, 7 vbles
     NetCDF ID:    131072  opened as READONLY            
     Time-independent data.
     Closing file TMP_POL_FILE
      
     "TMP_POL_FILE" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/inventory/MYD02/meic.MYD02_dat/NH3.ncf"
     File type GRDDED3 
     Execution ID "????????????????"
     Grid name ""
     Dimensions: 343107 rows, 1 cols, 1 lays, 7 vbles
     NetCDF ID:    131072  opened as READONLY            
     Time-independent data.
     Closing file TMP_POL_FILE
      
     "TMP_POL_FILE" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/inventory/MYD02/meic.MYD02_dat/NOX.ncf"
     File type GRDDED3 
     Execution ID "????????????????"
     Grid name ""
     Dimensions: 402808 rows, 1 cols, 1 lays, 7 vbles
     NetCDF ID:    131072  opened as READONLY            
     Time-independent data.
     Closing file TMP_POL_FILE
      
     "TMP_POL_FILE" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/inventory/MYD02/meic.MYD02_dat/PM10.ncf"
     File type GRDDED3 
     Execution ID "????????????????"
     Grid name ""
     Dimensions: 384167 rows, 1 cols, 1 lays, 7 vbles
     NetCDF ID:    131072  opened as READONLY            
     Time-independent data.
     Closing file TMP_POL_FILE
      
     "TMP_POL_FILE" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/inventory/MYD02/meic.MYD02_dat/PM2_5.ncf"
     File type GRDDED3 
     Execution ID "????????????????"
     Grid name ""
     Dimensions: 384167 rows, 1 cols, 1 lays, 7 vbles
     NetCDF ID:    131072  opened as READONLY            
     Time-independent data.
     Closing file TMP_POL_FILE
      
     "TMP_POL_FILE" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/inventory/MYD02/meic.MYD02_dat/SO2.ncf"
     File type GRDDED3 
     Execution ID "????????????????"
     Grid name ""
     Dimensions: 372633 rows, 1 cols, 1 lays, 7 vbles
     NetCDF ID:    131072  opened as READONLY            
     Time-independent data.
     Closing file TMP_POL_FILE
      
     "TMP_POL_FILE" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/inventory/MYD02/meic.MYD02_dat/VOC.ncf"
     File type GRDDED3 
     Execution ID "????????????????"
     Grid name ""
     Dimensions: 402965 rows, 1 cols, 1 lays, 7 vbles
     NetCDF ID:    131072  opened as READONLY            
     Time-independent data.
     Closing file TMP_POL_FILE
      
     "TMP_POL_FILE" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/inventory/MYD02/meic.MYD02_dat/PMC.ncf"
     File type GRDDED3 
     Execution ID "????????????????"
     Grid name ""
     Dimensions: 377769 rows, 1 cols, 1 lays, 7 vbles
     NetCDF ID:    131072  opened as READONLY            
     Time-independent data.
     Closing file TMP_POL_FILE
     Value for PROMPTFLAG:  N returning FALSE
      
     "ATMP1" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/run_MYD02/scenario/atmp.meic.MYD02.2020162.1.ncf"
     File type GRDDED3 
     Execution ID "????????????????"
     Grid name ""
     Dimensions: 2000000 rows, 1 cols, 1 lays, 4 vbles
     NetCDF ID:    131072  opened as READONLY            
     Starting date and time  2020162:000000 (0:00:00   June 10, 2020)
     Timestep                          010000 (1:00:00 hh:mm:ss)
     Maximum current record number        25
      
     "ATMP2" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/run_MYD02/scenario/atmp.meic.MYD02.2020162.2.ncf"
     File type GRDDED3 
     Execution ID "????????????????"
     Grid name ""
     Dimensions: 2000000 rows, 1 cols, 1 lays, 4 vbles
     NetCDF ID:    196608  opened as READONLY            
     Starting date and time  2020162:000000 (0:00:00   June 10, 2020)
     Timestep                          010000 (1:00:00 hh:mm:ss)
     Maximum current record number        25
     NOTE: Time zone initialized using ATMP             hourly emissions file.
     
     Value for PROMPTFLAG:  N returning FALSE
      
     "AGMAT" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/run_MYD02/static/agmat.meic.MYD02.MYD02.ncf"
     File type SMATRX3 
     Execution ID "????????????????"
     Grid name "MYd02_CROSS"
     Dimensions: 18904 rows, 334305 cols, 1 lays, 1 vbles
     NetCDF ID:    262144  opened as READONLY            
     Time-independent data.
      
     "ASMAT" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/run_MYD02/static/asmat_l.meic.cmaq_cb05_soa.MYD02.ncf"
     File type GRDDED3 
     Execution ID "????????????????"
     Grid name ""
     Dimensions: 2000000 rows, 1 cols, 1 lays, 32 vbles
     NetCDF ID:    327680  opened as READONLY            
     Time-independent data.
     NOTE: Speciation type initialized using area speciation matrix.

     File "INVTABLE" opened for input on unit:  96
     /home/atmos/MODEL/SMOKE-ifort/data/inventory/grid/other/invtable_hapcap_cb05soa.txt

     Value for USE_EXP_GEO_CODES:  Y returning TRUE
     Value for G_STDATE:  2020162
     Value for G_STTIME:  0
     Value for G_TSTEP:  10000
     Value for G_RUNLEN:  250000
     Value for PROMPTFLAG:  N returning FALSE
     Using default value   2020162  for query:
     "Enter simulation starting date (YYYYDDD)|(YYYYMMDD)"
     Using date  2020162
     Using default value         0  for query:
     "Enter simulation starting time (HHMMSS)"
     Using default value        25  for query:
     "Enter output duration (hours)"
     Output Time Zone: 00 
           Start Date: June 10, 2020 
           Start Time: 000000 HHMMSS 
            Time Step:   001 hour 
             Duration:   025 hours
     NOTE: Output grid "MYd02_CROSS" set; described as 
          No description available
     Value for MRG_GRDOUT_UNIT:  'moles/s'
     Value for MRG_GRDOUT_UNIT:  'moles/s'
     Value for MRG_TOTOUT_UNIT:  'moles/day'
     Value for MRG_TOTOUT_UNIT:  'moles/day'
     NOTE: Units conversion case not found for: "g" with "moles" 
          No conversion will be made.
     NOTE: Units conversion case not found for: "g" with "moles" 
          No conversion will be made.
     NOTE: Units conversion case not found for: "g" with "moles" 
          No conversion will be made.
     NOTE: Units conversion case not found for: "g" with "moles" 
          No conversion will be made.
     NOTE: Units conversion case not found for: "g" with "moles" 
          No conversion will be made.
     NOTE: Units conversion case not found for: "g" with "moles" 
          No conversion will be made.
     NOTE: Units conversion case not found for: "g" with "moles" 
          No conversion will be made.
     NOTE: Units conversion case not found for: "g" with "moles" 
          No conversion will be made.
     NOTE: Units conversion case not found for: "g" with "moles" 
          No conversion will be made.
     NOTE: Units conversion case not found for: "g" with "moles" 
          No conversion will be made.
     NOTE: Units conversion case not found for: "g" with "moles" 
          No conversion will be made.
     NOTE: Units conversion case not found for: "g" with "moles" 
          No conversion will be made.
     Reading source data from inventory file...
     Value for SMK_DEFAULT_TZONE:  5
     Reading geographic region names and time zones...

     File "GEOCODE_LEVEL1" opened for input on unit:  95
     /home/atmos/MODEL/SMOKE-ifort/data/ge_dat/meic/geocode1_edgar.txt


     File "GEOCODE_LEVEL2" opened for input on unit:  95
     /home/atmos/MODEL/SMOKE-ifort/data/ge_dat/meic/geocode2_edgar.txt


     File "GEOCODE_LEVEL3" opened for input on unit:  95
     /home/atmos/MODEL/SMOKE-ifort/data/ge_dat/meic/geocode3_edgar.txt


     File "GEOCODE_LEVEL4" opened for input on unit:  95
     /home/atmos/MODEL/SMOKE-ifort/data/ge_dat/meic/geocode4_edgar.txt

     ERROR: Gridding matrix dimension is inconsistent with records count! 
          Delete gridding matrix and recreate it.
     
     *** ERROR ABORT in subroutine RDGMAT

All my log files for this run are listed as follows:
grdmat.meic.MYD02.MYD02.log.txt (13.2 KB)
smkinven.meic.MYD02.log.txt (12.8 KB)
smkmerge.meic.MYD02.20200610.MYD02.log.txt (13.7 KB)
spcmat.meic.MYD02.cmaq_cb05_soa.log.txt (19.0 KB)
temporal.meic.MYD02.20200610.log.txt (29.5 KB)

2

We are wondering if you may have changed the grid dimensions (NCOL/NROW) at some point and there are files with the older grid dimensions still around. SMOKE will crash if it tries to write to an existing file that has different dimensions. If so, we recommend deleting everything in the intermediate and premerged directories for that sector and rerun.

3

Hi Alison. Thank you for replying. I have deleted everything in the intermediate and premerged directories and rerun several times, and the same error still around. I compared the smkmerge log file of my case with that of EDGAR and found that smkmerge opened two ‘ATMP’ files (ATMP1 and ATMP2) in my case, while EDGAR only opened one. I further checked these two files, and the pollutant factors were stored separately (as shown in the figure below).I wonder if this issue is causing the error. It’s strange that the same method, when applied to the MIX inventory, worked fine, yet here it generated two ATMP files and occurred the error.

     "ATMP1" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/run_MYD02/scenario/atmp.meic.MYD02.2020162.1.ncf"
     File type GRDDED3 
     Execution ID "????????????????"
     Grid name ""
     Dimensions: 2000000 rows, 1 cols, 1 lays, 4 vbles
     NetCDF ID:    131072  opened as READONLY            
     Starting date and time  2020162:000000 (0:00:00   June 10, 2020)
     Timestep                          010000 (1:00:00 hh:mm:ss)
     Maximum current record number        25
      
     "ATMP2" opened as OLD:READ-ONLY   
     File name "/home/atmos/MODEL/SMOKE-ifort/data/run_MYD02/scenario/atmp.meic.MYD02.2020162.2.ncf"
     File type GRDDED3 
     Execution ID "????????????????"
     Grid name ""
     Dimensions: 2000000 rows, 1 cols, 1 lays, 4 vbles
     NetCDF ID:    196608  opened as READONLY            
     Starting date and time  2020162:000000 (0:00:00   June 10, 2020)
     Timestep                          010000 (1:00:00 hh:mm:ss)
     Maximum current record number        25
     NOTE: Time zone initialized using ATMP             hourly emissions file.

atmp_files_screenshot
atmp_files_screenshot1170×761 140 KB

4

The ATMP files getting split into two components could be an I/O API issue. A while ago there was a limit of 120 variables per file, so when we had more than 120 pollutants in SMOKE runs, SMOKE/IOAPI would output multiple ATMP files per day. Although this wasn’t a problem as I/O API would handle this situation. Newer versions of the I/O API support additional variables in each file.

We note that there is a utility to MEIC into CMAQ-ready format posted here:

5

Hi, Alison. Thank you for your explanation about the multiple ATMP files. However, in my case, there were 7 pollutants (CO, NH3, NOx, PM10, PM2.5, SO2, VOC, and PMC), but the ATMP files were still split into two files. Moreover, there was an instruction in the temporal log file of EDGAR stating, “Number of variables per file array is not allocated for file set ATMP; using default of 2048 variables per file,” but I do not see this in my case (see the figure below).

temperal_log_screenshot
temperal_log_screenshot1920×1079 329 KB

6

@eyth.alison Alison, I understand what you mean, that ATMP files splitting into two was not caused the error. Later, I used MEIC to create a inventory for a previous simulation area, and it was successful. I only modified the GRIDDESC, GRIDCRO2D, and the variable “IOAPI_GRIDNAME_1”. Does this mean there was an issue with the simulation area settings? I have checked the GRIDDESC, GRIDCRO2D file many times,and make sure there was no problems. What exactly went wrong? :face_with_spiral_eyes:

7

We Can you provide your GRIDDESC and GRIDCRO2D for both the domain that worked and the domain that didn’t work?

That may shed some light on the issue.

8

Hi, Alison. Thank you very much for still paying attention to my issue. The worked domain files are as following:
GRIDCRO2D_tj_9km.txt (182.8 KB)
GRIDDESC_tj_9km.txt (193 Bytes)
And the maybe problem files are:
GRIDCRO2D_MYD02.txt (455.2 KB)
GRIDDESC_MYD02.txt (201 Bytes)
In the problem case, I have two nested domains. The ‘MYD02’ files are for domain2, while domain1 worked fine. The files for domain1 are also provided:
GRIDCRO2D_MYD01.txt (586.4 KB)
GRIDDESC_MYD01.txt (201 Bytes)
Furthermore, recently I discovered that some areas within domain2 are outside of China, while the MEIC inventory only includes the mainland of China. Consequently, I combined the MEIC and MIX inventories and utilized the new inventory for domain2, yet the problem persisted.

9

Hi,

We were unable to reproduce the error using the MYD02 domain and the GRIDCRO2D that you provided when applied to an HTAP gridded inventory.

You indicated that the domain1 worked for you. Could you please share the grdmat and smkmerge logs from the MYD01 domain for comparison?

10

Hi, Beidler. Apologies for the late response. I have successfully used the HTAP inventory for the MYD02 domain too, but encountered the same error when attempting to use the MIX inventory, which previously worked for the MYD01 domain. Now I have no idea whether the error lies with the inventory or the simulation domain settings.
I uploaded all the log files from the MYD01 and part of the inventories I used(via google drive) as below:
grdmat.mix.MYD01.MYD01.log.txt (13.2 KB)
smkinven.mix.MYD01.log.txt (15.0 KB)
smkmerge.mix.MYD01.20200701.MYD01.log.txt (54.1 KB)
spcmat.mix.MYD01.cmaq_cb05_soa.log.txt (18.9 KB)
temporal.mix.MYD01.20200701.log.txt (28.9 KB)

11

@james.beidler Hi, Beidler. I have opened the google drive authority for you. Can you get the inventory files? Is there any wrong with my inventory?

12

Unfortunately I was not able to reproduce the error shown in the original post using the datasets that you shared.
I’ve attached the smkmerge log from my test run with the MEIC transportation inventory if you’d like to compare.
smkmerge.txt (51.9 KB)

How is you arinv configured? Have you tried running each MEIC or MIX sector separately?

14

Beidler, thank you for your previous guidance. I have followed your suggestion to run each sector separately, and all have been successful except for the agriculture sector due to it only has data for NH3, with all other pollutants showing zero values.

Additionally, I have managed to run for four out of the five sectors without any issues. However, when attempting to run all sectors together, it failed.

I have uploaded all the sector data to Google Drive, and I have attached my arinv file below for your review.
arinv.mix_meic_fuse.lst.txt (3.9 KB)
arinv.mix_meic_fuse.part4.lst.txt (3.1 KB)

15

If you remove all pollutants but NH3 for agriculture from your arinv you should be able to run this sector. Rather than trying to run all of the sectors together you should run them separately and then combine them with mrggrid.

16

Hi, Beidler. Thank for your suggestions. I have successfully run each sector, but they all belong to area sources. How should I name the output files and merge them? The merge subroutine processes area, nonroad, point, and so on. Should I modify the setmerge_files.scr file to process multiple area outputs?

17

Hi, Beidler. I have figured out how to merge results from different sectors; it’s just a matter of modifying the output filenames and the merged file names in the “setmerge_files.scr” file.
@james.beidler @eyth.alison Thank you again for your help!