Running pre and post CMAQ analysis by AMET

AMET
1

Dear all;

I have experienced some problem with running AMET using “aqProject_pre_and_post.csh”. The log is attached below.

/usr/local/apps/ioapi/ioapi-3.2/Linux2_x86_64ifortmpi/m3utilio.mod/yijia: Not a directory.

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).  Copyright (C) 1992-2002 MCNC   
 and Carlie J. Coats, Jr., and 2003-2006 Baron Advanced          
 Meteorological Systems, LLC and released under the GNU LGPL     
 License, version 2.1.  See URL                                  
                                                                 
     http://www.baronams.com/products/ioapi/LGPL.txt             
                                                                 
 for conditions of use.                                          
                                                                 
 Library release tag: $JDate: 2005333 $                          
                                                                 
 $Id: @(#) ioapi library version 3.0 $
 Version with PARMS3.EXT/PARAMETER::MXVARS3=50000
 netCDF version "3.6.2" of Jan  6 2011 09:16:35 $
  
  
 Missing environment variable EXECUTION_ID

API Started
Value for GENSPEC not defined;returning default: FALSE
Value for IOAPI_CHECK_HEADERS not defined;returning default: FALSE

 "INFILE1" opened as OLD:READ-ONLY   
 File name "/gpfs_common/share03/cmaq/CMAQ_DATA/output_Jan3d_new_CCTM_v53_intel_2014_12US1_Jan3d/m3tshift/CCTM_ACONC_v53_intel_2014_12US1_Jan3d_20140101.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 217 vbles
 NetCDF ID:         3  opened as READONLY            
 Starting date and time  2014001:000000 (0:00:00   Jan. 1, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number        24
  
 Value for IOAPI_CHECK_HEADERS not defined;returning default:   FALSE
  
 "OUTFILE" opened as OLD:READ-WRITE  
 File name "/gpfs_common/share03/cmaq/AMET_v14/output/v53_intel_2014_12US1_Jan3d/COMBINE_ACONC_v53_intel_2014_12US1_Jan3d_201401.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 134 vbles
 NetCDF ID:         4  opened as READWRITE           
 Starting date and time  2014001:000000 (0:00:00   Jan. 1, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number         0

Copying Variables at time:2014001: 0

 >>--->> WARNING in subroutine DESC3
 Invalid file name argument

ERROR Invalid syntax for field: DENS[2]

ERROR Cannot read DENS from INFILE2

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).  Copyright (C) 1992-2002 MCNC   
 and Carlie J. Coats, Jr., and 2003-2006 Baron Advanced          
 Meteorological Systems, LLC and released under the GNU LGPL     
 License, version 2.1.  See URL                                  
                                                                 
     http://www.baronams.com/products/ioapi/LGPL.txt             
                                                                 
 for conditions of use.                                          
                                                                 
 Library release tag: $JDate: 2005333 $                          
                                                                 
 $Id: @(#) ioapi library version 3.0 $
 Version with PARMS3.EXT/PARAMETER::MXVARS3=50000
 netCDF version "3.6.2" of Jan  6 2011 09:16:35 $
  
  
 Missing environment variable EXECUTION_ID

API Started
Value for GENSPEC not defined;returning default: FALSE
Value for IOAPI_CHECK_HEADERS not defined;returning default: FALSE

 "INFILE1" opened as OLD:READ-ONLY   
 File name "/gpfs_common/share03/cmaq/CMAQ_DATA/output_Jan3d_new_CCTM_v53_intel_2014_12US1_Jan3d/m3tshift/CCTM_ACONC_v53_intel_2014_12US1_Jan3d_20140102.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 217 vbles
 NetCDF ID:         3  opened as READONLY            
 Starting date and time  2014002:000000 (0:00:00   Jan. 2, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number        24
  
 Value for IOAPI_CHECK_HEADERS not defined;returning default:   FALSE
  
 "OUTFILE" opened as OLD:READ-WRITE  
 File name "/gpfs_common/share03/cmaq/AMET_v14/output/v53_intel_2014_12US1_Jan3d/COMBINE_ACONC_v53_intel_2014_12US1_Jan3d_201401.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 134 vbles
 NetCDF ID:         4  opened as READWRITE           
 Starting date and time  2014001:000000 (0:00:00   Jan. 1, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number         0

Copying Variables at time:2014002: 0

 >>--->> WARNING in subroutine DESC3
 Invalid file name argument

ERROR Invalid syntax for field: DENS[2]

ERROR Cannot read DENS from INFILE2

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).  Copyright (C) 1992-2002 MCNC   
 and Carlie J. Coats, Jr., and 2003-2006 Baron Advanced          
 Meteorological Systems, LLC and released under the GNU LGPL     
 License, version 2.1.  See URL                                  
                                                                 
     http://www.baronams.com/products/ioapi/LGPL.txt             
                                                                 
 for conditions of use.                                          
                                                                 
 Library release tag: $JDate: 2005333 $                          
                                                                 
 $Id: @(#) ioapi library version 3.0 $
 Version with PARMS3.EXT/PARAMETER::MXVARS3=50000
 netCDF version "3.6.2" of Jan  6 2011 09:16:35 $
  
  
 Missing environment variable EXECUTION_ID

API Started
Value for GENSPEC not defined;returning default: FALSE
Value for IOAPI_CHECK_HEADERS not defined;returning default: FALSE

 "INFILE1" opened as OLD:READ-ONLY   
 File name "/gpfs_common/share03/cmaq/CMAQ_DATA/output_Jan3d_new_CCTM_v53_intel_2014_12US1_Jan3d/m3tshift/CCTM_ACONC_v53_intel_2014_12US1_Jan3d_20140103.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 217 vbles
 NetCDF ID:         3  opened as READONLY            
 Starting date and time  2014003:000000 (0:00:00   Jan. 3, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number        24
  
 Value for IOAPI_CHECK_HEADERS not defined;returning default:   FALSE
  
 "OUTFILE" opened as OLD:READ-WRITE  
 File name "/gpfs_common/share03/cmaq/AMET_v14/output/v53_intel_2014_12US1_Jan3d/COMBINE_ACONC_v53_intel_2014_12US1_Jan3d_201401.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 134 vbles
 NetCDF ID:         4  opened as READWRITE           
 Starting date and time  2014001:000000 (0:00:00   Jan. 1, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number         0

Copying Variables at time:2014003: 0

 >>--->> WARNING in subroutine DESC3
 Invalid file name argument

ERROR Invalid syntax for field: DENS[2]

ERROR Cannot read DENS from INFILE2

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).  Copyright (C) 1992-2002 MCNC   
 and Carlie J. Coats, Jr., and 2003-2006 Baron Advanced          
 Meteorological Systems, LLC and released under the GNU LGPL     
 License, version 2.1.  See URL                                  
                                                                 
     http://www.baronams.com/products/ioapi/LGPL.txt             
                                                                 
 for conditions of use.                                          
                                                                 
 Library release tag: $JDate: 2005333 $                          
                                                                 
 $Id: @(#) ioapi library version 3.0 $
 Version with PARMS3.EXT/PARAMETER::MXVARS3=50000
 netCDF version "3.6.2" of Jan  6 2011 09:16:35 $
  
  
 Missing environment variable EXECUTION_ID

API Started
Value for GENSPEC not defined;returning default: FALSE
Value for IOAPI_CHECK_HEADERS not defined;returning default: FALSE

 "INFILE1" opened as OLD:READ-ONLY   
 File name "/gpfs_common/share03/cmaq/CMAQ_DATA/output_Jan3d_new_CCTM_v53_intel_2014_12US1_Jan3d/m3tshift/CCTM_DRYDEP_v53_intel_2014_12US1_Jan3d_20140101.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 162 vbles
 NetCDF ID:         3  opened as READONLY            
 Starting date and time  2014001:010000 (1:00:00   Jan. 1, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number        24
  
 "INFILE2" opened as OLD:READ-ONLY   
 File name "/gpfs_common/share03/cmaq/CMAQ_DATA/output_Jan3d_new_CCTM_v53_intel_2014_12US1_Jan3d/m3tshift/CCTM_WETDEP1_v53_intel_2014_12US1_Jan3d_20140101.nc"
 File type GRDDED3 
 Execution ID "CMAQ_CCTMv53_yzhao28_20190821_200716_189196760"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 130 vbles
 NetCDF ID:         4  opened as READONLY            
 Starting date and time  2014001:010000 (1:00:00   Jan. 1, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number        24

file INFILE2 needs converion: F

 Value for IOAPI_CHECK_HEADERS not defined;returning default:   FALSE
  
 "OUTFILE" opened as OLD:READ-WRITE  
 File name "/gpfs_common/share03/cmaq/AMET_v14/output/v53_intel_2014_12US1_Jan3d/COMBINE_DEP_v53_intel_2014_12US1_Jan3d_201401.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 102 vbles
 NetCDF ID:         5  opened as READWRITE           
 Starting date and time  2014001:010000 (1:00:00   Jan. 1, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number         0

Copying Variables at time:2014001: 10000

 >>--->> WARNING in subroutine DESC3
 Invalid file name argument

ERROR Invalid syntax for field: RC[3]

ERROR Cannot read RC from INFILE3

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).  Copyright (C) 1992-2002 MCNC   
 and Carlie J. Coats, Jr., and 2003-2006 Baron Advanced          
 Meteorological Systems, LLC and released under the GNU LGPL     
 License, version 2.1.  See URL                                  
                                                                 
     http://www.baronams.com/products/ioapi/LGPL.txt             
                                                                 
 for conditions of use.                                          
                                                                 
 Library release tag: $JDate: 2005333 $                          
                                                                 
 $Id: @(#) ioapi library version 3.0 $
 Version with PARMS3.EXT/PARAMETER::MXVARS3=50000
 netCDF version "3.6.2" of Jan  6 2011 09:16:35 $
  
  
 Missing environment variable EXECUTION_ID

API Started
Value for GENSPEC not defined;returning default: FALSE
Value for IOAPI_CHECK_HEADERS not defined;returning default: FALSE

 "INFILE1" opened as OLD:READ-ONLY   
 File name "/gpfs_common/share03/cmaq/CMAQ_DATA/output_Jan3d_new_CCTM_v53_intel_2014_12US1_Jan3d/m3tshift/CCTM_DRYDEP_v53_intel_2014_12US1_Jan3d_20140102.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 162 vbles
 NetCDF ID:         3  opened as READONLY            
 Starting date and time  2014002:010000 (1:00:00   Jan. 2, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number        24
  
 "INFILE2" opened as OLD:READ-ONLY   
 File name "/gpfs_common/share03/cmaq/CMAQ_DATA/output_Jan3d_new_CCTM_v53_intel_2014_12US1_Jan3d/m3tshift/CCTM_WETDEP1_v53_intel_2014_12US1_Jan3d_20140102.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 130 vbles
 NetCDF ID:         4  opened as READONLY            
 Starting date and time  2014002:010000 (1:00:00   Jan. 2, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number        24

file INFILE2 needs converion: F

 Value for IOAPI_CHECK_HEADERS not defined;returning default:   FALSE
  
 "OUTFILE" opened as OLD:READ-WRITE  
 File name "/gpfs_common/share03/cmaq/AMET_v14/output/v53_intel_2014_12US1_Jan3d/COMBINE_DEP_v53_intel_2014_12US1_Jan3d_201401.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 102 vbles
 NetCDF ID:         5  opened as READWRITE           
 Starting date and time  2014001:010000 (1:00:00   Jan. 1, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number         0

Copying Variables at time:2014002: 10000

 >>--->> WARNING in subroutine DESC3
 Invalid file name argument

ERROR Invalid syntax for field: RC[3]

ERROR Cannot read RC from INFILE3

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3      
 I/O Applications Programming Interface, [I/O API] which is      
 built on top of the netCDF I/O library (Copyright 1993, 1996    
 University Corporation for Atmospheric Research/Unidata         
 Program) and the PVM parallel-programming library (from         
 Oak Ridge National Laboratory).  Copyright (C) 1992-2002 MCNC   
 and Carlie J. Coats, Jr., and 2003-2006 Baron Advanced          
 Meteorological Systems, LLC and released under the GNU LGPL     
 License, version 2.1.  See URL                                  
                                                                 
     http://www.baronams.com/products/ioapi/LGPL.txt             
                                                                 
 for conditions of use.                                          
                                                                 
 Library release tag: $JDate: 2005333 $                          
                                                                 
 $Id: @(#) ioapi library version 3.0 $
 Version with PARMS3.EXT/PARAMETER::MXVARS3=50000
 netCDF version "3.6.2" of Jan  6 2011 09:16:35 $
  
  
 Missing environment variable EXECUTION_ID

API Started
Value for GENSPEC not defined;returning default: FALSE
Value for IOAPI_CHECK_HEADERS not defined;returning default: FALSE

 "INFILE1" opened as OLD:READ-ONLY   
 File name "/gpfs_common/share03/cmaq/CMAQ_DATA/output_Jan3d_new_CCTM_v53_intel_2014_12US1_Jan3d/m3tshift/CCTM_DRYDEP_v53_intel_2014_12US1_Jan3d_20140103.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 162 vbles
 NetCDF ID:         3  opened as READONLY            
 Starting date and time  2014003:010000 (1:00:00   Jan. 3, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number        24
  
 "INFILE2" opened as OLD:READ-ONLY   
 File name "/gpfs_common/share03/cmaq/CMAQ_DATA/output_Jan3d_new_CCTM_v53_intel_2014_12US1_Jan3d/m3tshift/CCTM_WETDEP1_v53_intel_2014_12US1_Jan3d_20140103.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 130 vbles
 NetCDF ID:         4  opened as READONLY            
 Starting date and time  2014003:010000 (1:00:00   Jan. 3, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number        24

file INFILE2 needs converion: F

 Value for IOAPI_CHECK_HEADERS not defined;returning default:   FALSE
  
 "OUTFILE" opened as OLD:READ-WRITE  
 File name "/gpfs_common/share03/cmaq/AMET_v14/output/v53_intel_2014_12US1_Jan3d/COMBINE_DEP_v53_intel_2014_12US1_Jan3d_201401.nc"
 File type GRDDED3 
 Execution ID "????????????????"
 Grid name "12US1"
 Dimensions: 299 rows, 459 cols, 1 lays, 102 vbles
 NetCDF ID:         5  opened as READWRITE           
 Starting date and time  2014001:010000 (1:00:00   Jan. 1, 2014)
 Timestep                          010000 (1:00:00 hh:mm:ss)
 Maximum current record number         0

Copying Variables at time:2014003: 10000

 >>--->> WARNING in subroutine DESC3
 Invalid file name argument

ERROR Invalid syntax for field: RC[3]

ERROR Cannot read RC from INFILE3
Setting up AMET database if needed
Warning message:
In library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :
there is no package called ‘RMySQL’
Error in dbConnect(MySQL(), user = mysql_login, password = mysql_pass, :
could not find function “dbConnect”
Execution halted
Error setting up AMET database

I assume there is something wrong with my inputs or script settings? Have you seen this before?

Thank you very much!

Yijia

2

one notes that this is a very old I/O API library…

Something seems to have happened to the file-name argument for DESC3(). Version at least 3.1 (2010) would log the file-name argument so that you might have a better chance…

I don’t know about those “Invalid syntax…” error messages…

3

Thank you for replying! I think there might some issue with reading the mcip files. DENS means air density and RC means precipitation. I think I put the correct path there in the script.

Yijia

4

Hi Yijia,

Yes, there does appear to be an issue reading the met variables out of the MCIP files. That should be the first issue you try to address. Not sure if it’s related to an issue with I/O API or not as suggested. You’ll need to do a little digger there.

I also noticed at the end of the log file you sent an error with the RMySQL package in your R installation. Apparently it is missing based on the error message. You’ll need to install the RMySQL package in order to use the database with AMET.

Wyat

5

Dear Wyat;

I will look into the mcip inputs.

The RMySQL makes me very confused. Actually I am not using database in AMET. I just want to compare the obs with model outputs. I am following the “AMET aqProject Pre- and Post- Analysis Script Guide v1.4” on AMET/doc in GitHub. It did not tell me to how to get away from the database setup. I don’t think I need to set up a database with my purpose of only comparing obs and model. Am I understanding right? The script I am running is “AMET_v14/scripts_db/aqExample/aqProject_pre_and_post.csh”. Is it the correct one to run? If not, would you please point me to the right one?

Appreciate it a lot!

Yijia

6

Since it sounds like you just want the paired model/obs data, you should set the LOAD_SITEX flag to F, but keep the WRITE_SITEX and RUN_SITEX flags set to T. That will avoid AMET attempting to load the site compare output into the database.

That should work for you.

7

Since I am not using project or database connection, I want to run the script in the directory of “scripts_analysis”. For instance, I am trying to get the “run_scatterplot.csh” to work. However, it keeps giving me error of either “mysql not loaded” or “source config_file failed”. I suspect I did not know how to configure the running script.Or I missed somewhere.

In the “run_scatterplot.csh”, there is no places to set the locations of obs and model outputs. I have used the combine program to pre-analyze the CMAQ outputs already. So I want to link the observations and combined outputs to the running script. Where should I do this?

In the User’s Guide, since it is mainly used to utilize AMET database, the instruction of how to pair and compare obs and model without database is not very clear. Could you give me some hints if possible?

Thank you a lot!

Yijia

8

Error in source(config_file) :
cannot coerce type ‘closure’ to vector of type ‘character’
Calls: source -> withVisible -> eval -> eval -> source
Execution halted

By comment all the setting on mysql database, I got this error message. I don’t plan to use any database connect so I don’t want to set up.

9

Hi Yijia,

I assume in the run_scatterplot.csh script you set the AMET_DB flag to F, correct? That’s the first thing you need to do. Second, you need to set the environment variable OUTDIR to the location of your site compare output files (the csv files). Note that regardless of whether or not you’re using the database, you still need to specify the environment variables AMETBASE, AMET_PROJECT and MYSQL_CONFIG (I realize MYSQL_CONFIG is confusing since you’re not using the database, but it’s really an AMET config). And if you don’t plan on doing a model-to-model comparison, comment out AMET_PROJECT2.

Please try doing those two steps if you have not already and report back any issues. Thanks.

Wyat

10

Thank you very much, Wyat. I did not know I need to input the site compare output. I understand now.

Best;
Yijia