CMAS Data Warehouse

abi0010 · November 22, 2021, 2:24pm

Hi… what is the procedure for downloading multiple from the CMAS Data Warehouse? I was looking for a procedure and I couldn’t find the link to it.

EQUATESv1.0: Emissions, WRF/MCIP, CMAQv5.3.2 Data – 2002-2017 US_12km and NHEMI_108km - CMAS Data Warehouse (unc.edu)

lizadams · November 22, 2021, 4:09pm

To download multiple files from equates, sample download scripts have been provided.

The scripts require gdrive.
The link for instructions on how to install gdrive is available here:

Sample scripts to download equates data.

github.com

lizadams/EQUATES_BENCHMARK/blob/main/gdrive_scripts/data/2017_12US1/emis/gdrive_download_equates_emis.csh

#! /bin/csh -f
# script to download files matching query using gdrive
# # NOTE: Please use the gdrive executable for linux available from the CMAS Center Google Drive to avoid Error:403: Rate Limit Exceeded errors
# # see README Data Download Instructions https://docs.google.com/document/d/1e7B94zFkbKygVWfrhGwEZL51jF4fGXGXZbvi6KzXYQ4/edit
# #
# # usage example 
# # ./SCRIPT_NAME
# # ./gdrive_download_equates_emis.csh
#
#
#set the month to be downloaded
# in this example, the emissions inputs for the month of December 2016 will be downloaded
set year = 2016
set month = 12
set runto = 12

gdrive list --name-width 200 --absolute -q "trashed = false and name = 'static_emis_files_EQUATES_v1.0.tar'" --no-header  | cut -d " " -f1 - | xargs -L 1 gdrive download -r --timeout 0
gdrive list --name-width 200 --absolute -q "trashed = false and name = 'model_ready_emis_${year}_stackgroups_epicsoil_EQUATES_v1.0.tar'" --no-header  | cut -d " " -f1 - | xargs -L 1 gdrive download -r --timeout 0

while ($month <= $runto)

This file has been truncated. show original

github.com

lizadams/EQUATES_BENCHMARK/blob/main/gdrive_scripts/data/2017_12US1/icbc/gdrive_download_equates.icbc.csh

#! /bin/csh -f
# script to download files matching query using gdrive
# # NOTE: Please use the gdrive executable for linux available from the CMAS Center Google Drive to avoid Error:403: Rate Limit Exceeded errors
# # see README Data Download Instructions https://docs.google.com/document/d/1e7B94zFkbKygVWfrhGwEZL51jF4fGXGXZbvi6KzXYQ4/edit
# #
# # usage example 
# # ./SCRIPT_NAME
# # ./gdrive_download_equates.icbc.csh
#
# in this example, the input files for the month of December 2016 will be downloaded
# note, the script does not handle change in calendar year
set year = 2016
set month = 12
set runto = 12


while ($month <= $runto)
	  echo "Process files ending in $month"
	  #	   gdrive list --name-width 200 --absolute -q "trashed = false and name = 'BCON_CONC_12US1_CMAQv53_TS_108NHEMI_regrid_${month}.nc'" --no-header  | cut -d " " -f1 - | xargs -L 1 gdrive download -r --timeout 0
	    gdrive list --name-width 200 --absolute -q "trashed = false and name = 'CCTM_CGRID_v532_cb6r3_ae7_aq_WR413_MYR_STAGE_${year}_12US1_${year}${month}15.nc'" --no-header  | cut -d " " -f1 - | xargs -L 1 gdrive download -r --timeout 0

This file has been truncated. show original

github.com

lizadams/EQUATES_BENCHMARK/blob/main/gdrive_scripts/data/2017_12US1/met/mcip_v51_wrf_v411_noltng/gdrive_download_equates_mcip.csh

#! /bin/csh -f
# script to download files matching query using gdrive
# # NOTE: Please use the gdrive executable for linux available from the CMAS Center Google Drive to avoid Error:403: Rate Limit Exceeded errors
# # see README Data Download Instructions https://docs.google.com/document/d/1e7B94zFkbKygVWfrhGwEZL51jF4fGXGXZbvi6KzXYQ4/edit
## NOTE: This is a EQUATES 2017 Data Download Script
# #
# # usage example 
# # ./SCRIPT_NAME
# # ./gdrive_download_equates_mcip.csh
#
# note, in this example, the script will download from December 15, 2016 to December 31, 2016
# note, this script does not handle changes in the calendar year
set pdate = 20161215 
set runto = 20161231


while ($pdate <= $runto)
	  echo "Process files ending in $pdate"
	   gdrive list --name-width 200 --absolute -q "trashed = false and name = 'MCIPv51_WRFv411_noltng_12US1.35L.${pdate}.tar'" --no-header  | cut -d " " -f1 - | xargs -L 1 gdrive download -r --timeout 0
	   tar -xvf MCIPv51_WRFv411_noltng_12US1.35L.${pdate}.tar

This file has been truncated. show original

The above scripts will download data for the benchmark case described in the README.md available here:

abi0010 · December 1, 2021, 3:30pm

Thank you very much!

abi0010 · December 1, 2021, 6:46pm

One more question: apologies if i missed it - where can I find the 10-day spin up data for December 2015? for equates data?

lizadams · December 1, 2021, 8:48pm

Hi Abi,

Spin-up data for EQUATES can obtained as follows:
December 2015 Meteorology:
https://drive.google.com/drive/folders/1JvtoA7B-HSSkmSQ42mbsZRpijfxEzYfy?usp=sharing
December 2015 ICBCs:
https://drive.google.com/drive/folders/1n5GKkaHjhIDyEoheYmUC2P_TyQBCnjJv?usp=sharing
2015 Emissions:
https://drive.google.com/drive/folders/1p2AsX0L7vu4CH0UHRIDb9XirfhGvzWfu?usp=sharing

Note the ICs have been provided for the 1st and 15th day of each month, so you can start from Dec. 16 and run until Dec. 31 for 16 days of spinup. Please refer to the example benchmark run script for an example (EQUATES_BENCHMARK/run_cctm_2017_12US1_EQUATES.csh at main · lizadams/EQUATES_BENCHMARK · GitHub)

abi0010 · December 1, 2021, 9:55pm

Thanks you for this Liz! Very much appreciated!

Topic		Replies	Views
Monthly files with daily average H-CMAQ output:from 2017 onwards EQUATES	2	263	August 13, 2022
CONC output from EQUATES 12US1 runs EQUATES cmaq , equates	2	188	September 26, 2023
Download daily average hemispheric CMAQ output for BCON - "google drive quota exceeded" error EQUATES admin , cmaq	4	494	February 15, 2023
Date of the emission data from EQUATES EQUATES cmaq , equates	2	193	October 12, 2023
Missing October H-CMAQ output for 2016 dataset Ancillary Inputs	5	316	April 11, 2022

CMAS Data Warehouse

Related Topics