CMAQ - I/O API installation

djensen · February 23, 2021, 2:25am

Hello,

I want to install CMAQ. However, before that, I used the following link to install netCDF-C, netCDF-Fortran, and I/O API:

github.com

USEPA/CMAQ/blob/master/DOCS/Users_Guide/Tutorials/CMAQ_UG_tutorial_build_library_intel.md

## Install netCDF-C

1. Download netCDF-C from the following website https://www.unidata.ucar.edu/downloads/netcdf/index.jsp

```
wget ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-c-4.7.0.tar.gz
```

2. Untar the file 

```
tar -xzvf netcdf-c-4.7.0.tar.gz
```

3. Change directories into the package
```
cd netcdf-c-4.7.0
```

3. Verify that no modules are currently loaded using module commands.

This file has been truncated. show original

In the above website, I found a link to download different versions of I/O API:
https://www.cmascenter.org/ioapi/documentation/all_versions/html/AVAIL.html

So, I downloaded v3.2. Then, I followed the instruction in README.txt.

Then I did the following step (Step 6):
6. Customize “Makefile” for your system:
The default directory for both executables and object
libraries is in directory “…/$BIN” relative to the source
code directories for the I/O API and tools. Edit the
“Makefile” to put “BASEDIR”, “SRCDIR”, and “OBJDIR” wherever
you want it (if you want somewhere other than the default
“./$BIN” location). Look at the “Makefile” header-comment
for further customization options.

Then I typed “make”, however, I received the following errors:

Makefile:200: warning: overriding recipe for target `/CMAQ_libs//Linux3 Makefile:197: warning: ignoring old recipe for target` /CMAQ_libs//Linu3’
mkdir -p /CMAQ_libs/Linux3/Linux3
mkdir -p /CMAQ_libs/Linux3/ioapi/fixed_src
(cd /CMAQ_libs/Linux3/ioapi ; make fixed_src)
make[1]: Entering directory `/CMAQ_libs/Linux3/ioapi' make[1]: Nothing to be done for` fixed_src’.
make[1]: Leaving directory `CMAQ_libs/Linux3/ioapi' (cd /CMAQ_libs/Linux3/ioapi ; make all) make[1]: Entering directory` CMAQ_libs/Linux3/ioapi’
make[1]: * No rule to make target `all'. Stop. make[1]: Leaving directory` /CMAQ_libs/Linux3/ioapi’
make: * [all] Error 2

Then, I have tried to delete it and install the lower version of I/O API (e.g. 3.1), however, I received the same error.

Can you please help me in this regard?

Thank you.

lizadams · February 23, 2021, 2:53am

There seems to be an issue with this version I/O API for the top level Makefile.

Please try the following.
On line 210 in the top level makefile is the following line:

configure: {IODIR}/Makefile {TOOLDIR}/Makefile

Change this to:

configure:

Delete the prerequisite “$(IODIR)/Makefile & $(TOOLDIR)/Makefile”

cd ioapi
rm Makefile

cd …
cd m3tools
rm Makefile
cd …

set the following environment variables

setenv BIN Linux2_x86_64gfort
setenv CPLMODE nocpl

Then rerun the top level make.

If this fails when trying to build m3tools, then try to specify the directory path to the netcdf C and netcdf Fortran include directories.

Edit the $(TOOLDIR)/Makefile to directly specify the location of the netCDF Fortran and netCDF C Include files.

LIBS = -L${OBJDIR} -lioapi -L <path_to_netcdf-fortran>/lib -lnetcdff -L <path_to_netcdf-c-4.7.0>/lib -lnetcdf (OMPLIBS) (ARCHLIB) $(ARCHLIBS)

cjcoats · February 23, 2021, 12:17pm

The authoritative version of the build and installation instructions for I/O API Versions 3.0, 3.1, and 3.2 comes from the I/O API web pages, and can be found at either of the following (which duplicate each other:

https://www.cmascenter.org/ioapi/documentation/all_versions/html/AVAIL.html
https://cjcoats.github.io/ioapi/AVAIL.html
Or you can use the README.txt at the top level of the I/O API directory sub-tree.

Note in particular the

For each desired configuration of machine, compiler-set, and other options, setenv BIN , where matches the extension on one of the ioapi/Makeinclude.* (creating a new one, if necessary). Then do a top-level make dirs to build the object-directories

.
`

djensen · February 24, 2021, 8:05pm

Thank you for you help.

I exactly used the above links and downloaded 3.2, 3.1, and 3.0. I also followed the instruction in README.txt. However, I received the above error.

djensen · February 25, 2021, 12:47am

Hi Liz,

I removed all of my previous ioapi (versions 3.1, 3.2, and 3.0) and installed ioapi3.2 from the scratch.

I am trying to compile the library intel2020 and mpi.

Then, in he top folder, I make a copy of Makefile.template to Makefile and modify line 210 based on your above instruction.

The I use this library for the system:
module load intel/2020
module load intel/mpi-2020u2
module load netcdf/4.7.4-intel2020
module load hdf5/1.12.0-intel2020
export NETCDF_LIBDIR=/shared/centos7/netcdf/4.7.4-intel2020/lib
export NETCDF_INCDIR=/shared/centos7/netcdf/4.7.4-intel2020/include

Then I use these export commands:
export BIN=Linux2_x86_64ifortmpi
export INSTALL=/…/WRFCMAQ/CMAQ_libs/
export BASEDIR=/…/WRFCMAQ/CMAQ_libs/ioapi-3.2
export CPLMODE=nocpl
make

Then, I type “make” and find the following error:
mkdir -p /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/Linux2_x86_64ifortmpi
(cd /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi ; make fixed_src)
make[1]: Entering directory /.../WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi' make[1]: Nothing to be done for fixed_src’.
make[1]: Leaving directory /.../WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi' (cd /.../WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi ; sed -e 's|IOAPI_BASE|/.../WRFCMAQ/CMAQ_libs/ioapi-3.2|' -e 's|LIBINSTALL||' -e 's|BININSTALL||' -e 's|IOAPI_DEFS||' -e 's|NCFLIBS|-lnetcdff -lnetcdf|' -e 's|MAKEINCLUDE|include /.../WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi/Makeinclude|' -e 's|PVMINCLUDE|include |' < Makefile.nocpl.sed > Makefile ) (cd /.../WRFCMAQ/CMAQ_libs/ioapi-3.2/m3tools ; sed -e 's|IOAPI_BASE|/.../WRFCMAQ/CMAQ_libs/ioapi-3.2|' -e 's|LIBINSTALL||' -e 's|BININSTALL||' -e 's|IOAPI_DEFS||' -e 's|NCFLIBS|-lnetcdff -lnetcdf|' -e 's|MAKEINCLUDE|include /.../WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi/Makeinclude|' -e 's|PVMINCLUDE|include |' < Makefile.nocpl.sed > Makefile ) (cd /.../WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi ; make BIN=Linux2_x86_64ifortmpi all) make[1]: Entering directory /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi’
cd /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/Linux2_x86_64ifortmpi; mpif90 -auto -warn notruncated_source -Bstatic -static-intel -c -DAUTO_ARRAYS=1 -DF90=1 -DFLDMN=1 -DFSTR_L=int -DIOAPI_NO_STDOUT=1 -DAVOID_FLUSH=1 -DBIT32=1 -O3 -unroll -stack_temps -safe_cray_ptr -convert big_endian -assume byterecl -traceback -xHost -openmp -qopenmp -DAUTO_ARRAYS=1 -DF90=1 -DFLDMN=1 -DFSTR_L=int -DIOAPI_NO_STDOUT=1 -DAVOID_FLUSH=1 -DBIT32=1 -I/…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi/m3utilio.f
gfortran: error: notruncated_source: No such file or directory
gfortran: error: big_endian: No such file or directory
gfortran: error: byterecl: No such file or directory
gfortran: error: unrecognized command line option ‘-auto’
gfortran: error: unrecognized command line option ‘-warn’
gfortran: error: unrecognized command line option ‘-static-intel’
gfortran: error: unrecognized command line option ‘-stack_temps’
gfortran: error: unrecognized command line option ‘-safe_cray_ptr’
gfortran: error: unrecognized command line option ‘-convert’
gfortran: error: unrecognized command line option ‘-assume’
gfortran: error: unrecognized command line option ‘-traceback’
gfortran: error: unrecognized command line option ‘-qopenmp’
make[1]: *** [m3utilio.o] Error 1
make[1]: Leaving directory `/…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi’
make: *** [all] Error 2

Besides, I try to make the following modification and compile Makefile in m3tools:
LIBS = -L${OBJDIR} -lioapi -L/shared/centos7/netcdf/4.7.4-intel2020/lib -lnetcdff -L/shared/centos7/netcdf/4.7.4-intel2020/lib -lnetcdf (OMPLIBS) (ARCHLIB) $(ARCHLIBS)

However, after the above modification, once I typed “make”, I received the following error:
make: *** No rule to make target m3utilio.mod', needed by airs2m3.o’. Stop.

Thank you in advance for your help.

djensen · February 25, 2021, 12:48am

Hi Liz,

I removed all of my previous ioapi (versions 3.1, 3.2, and 3.0) and installed ioapi3.2 from the scratch.

I am trying to compile the library intel2020 and mpi.

Then, in he top folder, I make a copy of Makefile.template to Makefile and modify line 210 based on your above instruction.

The I use this library for the system:
module load intel/2020
module load intel/mpi-2020u2
module load netcdf/4.7.4-intel2020
module load hdf5/1.12.0-intel2020
export NETCDF_LIBDIR=/shared/centos7/netcdf/4.7.4-intel2020/lib
export NETCDF_INCDIR=/shared/centos7/netcdf/4.7.4-intel2020/include

Then I use these export commands:
export BIN=Linux2_x86_64ifortmpi
export INSTALL=/…/WRFCMAQ/CMAQ_libs/
export BASEDIR=/…/WRFCMAQ/CMAQ_libs/ioapi-3.2
export CPLMODE=nocpl
make

Then, I type “make” and find the following error:
mkdir -p /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/Linux2_x86_64ifortmpi
(cd /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi ; make fixed_src)
make[1]: Entering directory /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi’ make[1]: Nothing to be done for fixed_src’.
make[1]: Leaving directory /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi’ (cd /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi ; sed -e ‘s|IOAPI_BASE|/…/WRFCMAQ/CMAQ_libs/ioapi-3.2|’ -e ‘s|LIBINSTALL||’ -e ‘s|BININSTALL||’ -e ‘s|IOAPI_DEFS||’ -e ‘s|NCFLIBS|-lnetcdff -lnetcdf|’ -e ‘s|MAKEINCLUDE|include /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi/Makeinclude|’ -e ‘s|PVMINCLUDE|include |’ < Makefile.nocpl.sed > Makefile ) (cd /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/m3tools ; sed -e ‘s|IOAPI_BASE|/…/WRFCMAQ/CMAQ_libs/ioapi-3.2|’ -e ‘s|LIBINSTALL||’ -e ‘s|BININSTALL||’ -e ‘s|IOAPI_DEFS||’ -e ‘s|NCFLIBS|-lnetcdff -lnetcdf|’ -e ‘s|MAKEINCLUDE|include /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi/Makeinclude|’ -e ‘s|PVMINCLUDE|include |’ < Makefile.nocpl.sed > Makefile ) (cd /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi ; make BIN=Linux2_x86_64ifortmpi all) make[1]: Entering directory /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi’
cd /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/Linux2_x86_64ifortmpi; mpif90 -auto -warn notruncated_source -Bstatic -static-intel -c -DAUTO_ARRAYS=1 -DF90=1 -DFLDMN=1 -DFSTR_L=int -DIOAPI_NO_STDOUT=1 -DAVOID_FLUSH=1 -DBIT32=1 -O3 -unroll -stack_temps -safe_cray_ptr -convert big_endian -assume byterecl -traceback -xHost -openmp -qopenmp -DAUTO_ARRAYS=1 -DF90=1 -DFLDMN=1 -DFSTR_L=int -DIOAPI_NO_STDOUT=1 -DAVOID_FLUSH=1 -DBIT32=1 -I/…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi /…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi/m3utilio.f
gfortran: error: notruncated_source: No such file or directory
gfortran: error: big_endian: No such file or directory
gfortran: error: byterecl: No such file or directory
gfortran: error: unrecognized command line option ‘-auto’
gfortran: error: unrecognized command line option ‘-warn’
gfortran: error: unrecognized command line option ‘-static-intel’
gfortran: error: unrecognized command line option ‘-stack_temps’
gfortran: error: unrecognized command line option ‘-safe_cray_ptr’
gfortran: error: unrecognized command line option ‘-convert’
gfortran: error: unrecognized command line option ‘-assume’
gfortran: error: unrecognized command line option ‘-traceback’
gfortran: error: unrecognized command line option ‘-qopenmp’
make[1]: *** [m3utilio.o] Error 1
make[1]: Leaving directory `/…/WRFCMAQ/CMAQ_libs/ioapi-3.2/ioapi’
make: *** [all] Error 2

Besides, I try to make the following modification and compile Makefile in m3tools:
LIBS = -L${OBJDIR} -lioapi -L/shared/centos7/netcdf/4.7.4-intel2020/lib -lnetcdff -L/shared/centos7/netcdf/4.7.4-intel2020/lib -lnetcdf
(OMPLIBS) (ARCHLIB) $(ARCHLIBS)

However, after the above modification, once I typed “make”, I received the following error:
make: *** No rule to make target m3utilio.mod’, needed by airs2m3.o’. Stop.

Thank you in advance for your help.

cjcoats · February 26, 2021, 8:34pm

No, No, No!

See item 8 in the build instructions https://www.cmascenter.org/ioapi/documentation/all_versions/html/AVAIL.html#build.

Acquire and build the netCDF library, libnetcdf.a (and libnetcdff.a for netCDF-4.1 or later), for each configuration…
Vendor supplied netCDF libraries frequently cause problems, in part because they are compiled with compilers that are not link-compatible with the Fortran compilers used by the I/O API and the various models
Build your own netCDF C and Fortran libraries!

See item 5 in the build instructions: BIN=Linux2_x86_64ifortmpi is for building the distributed-I/O version, and also requires that you build the relevant PnetCDF library
If you are doing an Intel icc/ifort based build, what in the world are you doing using gfortran? Use the same compiler-system for all the models. Note that command line options for gfortran are very different to command line options for ifort.

djensen · February 27, 2021, 3:16am

Hello,

Thank you.

I exported the compiler (as below):
export CC=icc
export CXX=icpc
export F77=ifort
export FC=ifort
export F90=ifort
export CPP=‘icpc -E’
export CXXCPP=‘icpc -E’

Next, I changed the BIN:
export BIN=Linux2_x86_64ifort

Then, I change the content of the Makeinclude.$BIN in ioapi folder:
-openmp to -qopenmp

Finally, I got the following error:
cd /home/niknejad.s/WRFCMAQ/CMAQ_libs/ioapi-3.2/Linux2_x86_64ifort; ifort -auto -warn notruncated_source -Bstatic -static-intel /home/niknejad.s/WRFCMAQ/CMAQ_libs/ioapi-3.2/Linux2_x86_64ifort/airs2m3.o -L/home/niknejad.s/WRFCMAQ/CMAQ_libs/ioapi-3.2/Linux2_x86_64ifort -lioapi -lnetcdff -lnetcdf -qopenmp -Bstatic -o airs2m3
ld: /shared/centos7/intel/parallel_studio_xe_2020_cluster_edition/compilers_and_libraries_2020.1.217/linux/bin/intel64/…/…/lib/LLVMgold.so: error loading plugin: libimf.so: cannot open shared object file: No such file or directory
make[1]: *** [airs2m3] Error 1
make[1]: Leaving directory `/…/WRFCMAQ/CMAQ_libs/ioapi-3.2/m3tools’
make: *** [all] Error 2

I would appreciate if you could help me in this regard.

Thank you.

cjcoats · February 27, 2021, 12:32pm

libimf.so is part of the Intel math runtime-libraries. This would indicate that you did not have a fully set-up set of Intel compilers when you did this.

See your systems administrator (or whoever installed ifort and icc): somewhere in that installation is a set of scripts that set up various environment variables like LD_LIBRARY_PATH for the compilers. He needs to tell them how to source these scripts before you start to use the compilers. (The usual practice is to have this done as part of login-processing of /etc/profile.d/ or else as part of your .bashrc; or for systems using the MODULE software-management environment as part of the module load command.)

djensen · March 13, 2021, 7:50am

Hello,

Thank you very much for your help.

I have tried compiling netcdf (and hdf5) with intel 2020 compilers + MPI and Intel2020 + mvapich2 , based on the instructions here:

github.com

USEPA/CMAQ/blob/master/DOCS/Users_Guide/Tutorials/CMAQ_UG_tutorial_build_library_intel.md

## Install netCDF-C

1. Download netCDF-C from the following website https://www.unidata.ucar.edu/downloads/netcdf/index.jsp

```
wget ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-c-4.7.0.tar.gz
```

2. Untar the file 

```
tar -xzvf netcdf-c-4.7.0.tar.gz
```

3. Change directories into the package
```
cd netcdf-c-4.7.0
```

3. Verify that no modules are currently loaded using module commands.

This file has been truncated. show original

However, it seems that using Intel 2020 compilers might be the issue. I’ll continue with some combinations of the intel 2021 compilers, and will try openmpi instead of mvapich2. Unfortunately, we don’t have intel compilers of earlier versions than 2019 as they have in their example. Additionally, they mention that newer intel compiler versions (>18.0) have some incompatibilities.
https://www.cmascenter.org/ioapi/documentation/all_versions/html/AVAIL.html
In this case, is it ok with switching to GNU compilers?
Also, the configuration and compilation notes suggested I did not build with the MPI option, can you please tell me if I can use a version without MPI?
For your information, since I want to install CMAQ later, I am not sure about MPI. However, from what I understand from the documentation of ioapi, it can be compiled without the option of MPI (which is related to using a parallel version of netCDF and other libraries), but CMAQ needs parallelism to run with more processors.

Thank you in advance for your assistance.

lizadams · April 13, 2021, 2:05am

I/O API can be build with the gnu compilers without MPI
The same gnu compilers should be used to build CMAQ with MPI, using the non-mpi version of I/O API as a library.