# Compiling CMAQ on Cheyenne at NCAR

**URL:** <https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995>\
**Category:** Compiling\
**Created:** [January 5, 2021, 2:31am UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995 "2021-01-05T02:31:53Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ckuo6](https://avatars.discourse-cdn.com/v4/letter/c/b9bd4f/32.png) [@ckuo6](https://forum.cmascenter.org/u/ckuo6)\
**Post date:** [January 5, 2021, 2:31am UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/1 "2021-01-05T02:31:53Z")

</div>

Hi Guys,

I tried to compile CMAQ on Cheyenne in NCAR.  
My ioapi version is 3.1, the compiler is intel, and the related environment settings in config\_cmaq.csh (line 80-105) is:

```
case intel:

    #> I/O API, netCDF, and MPI library locations
    setenv IOAPI_INCL_DIR /glade/scratch/chengpin/ioapi_3_1/Linux2_x86_64ifort #ioapi_inc_intel #> I/O API include header files
    setenv IOAPI_LIB_DIR /glade/scratch/chengpin/ioapi_3_1/Linux2_x86_64ifort #> I/O API libraries
    setenv NETCDF_LIB_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/lib #netcdf_lib_intel #> netCDF C directory path
    setenv NETCDF_INCL_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/include #> netCDF C directory path
    setenv NETCDFF_LIB_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/lib #> netCDF Fortran directory path
    setenv NETCDFF_INCL_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/include #> netCDF Fortran directory path
    setenv MPI_LIB_DIR /glade/u/apps/ch/opt/mpt/2.22 #> MPI directory path

    #> Compiler Aliases and Flags
    #> set the compiler flag -qopt-report=5 to get a model optimization report in the build directory with the optrpt extension
    setenv myFC ifort
    setenv myCC icc
    setenv myFSTD "-O3 -fno-alias -mp1 -fp-model source -ftz -simd -align all -xHost -vec-guard-write -unroll-aggressive"
    setenv myDBG "-O0 -g -check bounds -check uninit -fpe0 -fno-alias -ftrapuv -traceback"
    setenv myLINK_FLAG #"-qopenmp-simd" openMP not supported w/ CMAQ
    setenv myFFLAGS "-fixed -132"
    setenv myFRFLAGS "-free"
    setenv myCFLAGS "-O2"
    setenv extra_lib "-lcurl"
    #chsetenv extra_lib ""
    setenv mpi_lib "-lmpich" #> No Library specification needed for mpiifort
                         #> -lmpich for mvapich
                         #> -lmpi for openmpi

```

And I ./bldit\_cctm.csh under /CCTM/scripts, some error messages occurred, which is:

ptrwrite3.f(65): error #5102: Cannot open include file ‘STATE3.EXT’  
INCLUDE ‘STATE3.EXT’  
--------------^  
ptrwrite3.f(116): error #6404: This name does not have a type, and must have an explicit type. [FINIT3]  
IF ( .NOT. FINIT3 ) THEN  
-----------------^  
ptrwrite3.f(118): error #6404: This name does not have a type, and must have an explicit type. [LOGDEV]  
WRITE( LOGDEV,91010 ) ‘WRITE3(): I/O API not yet initialized.’  
----------------^  
ptrwrite3.f(141): error #6404: This name does not have a type, and must have an explicit type. [COUNT3]  
FID = INDEX1 ( FIL16, COUNT3, FLIST3 )  
------------------------------^  
ptrwrite3.f(141): error #6404: This name does not have a type, and must have an explicit type. [FLIST3]  
FID = INDEX1 ( FIL16, COUNT3, FLIST3 )  
--------------------------------------^  
ptrwrite3.f(151): error #6404: This name does not have a type, and must have an explicit type. [SDATE3]  
STEP = JSTEP3 ( JDATE, JTIME, SDATE3( FID ), STIME3( FID ),  
---------------------------------------^  
ptrwrite3.f(151): error #6404: This name does not have a type, and must have an explicit type. [STIME3]  
STEP = JSTEP3 ( JDATE, JTIME, SDATE3( FID ), STIME3( FID ),  
------------------------------------------------------^  
ptrwrite3.f(152): error #6404: This name does not have a type, and must have an explicit type. [TSTEP3]  
& ABS( TSTEP3( FID ) ) )  
------------------------------^  
ptrwrite3.f(152): warning #7319: This argument’s data type is incompatible with this intrinsic procedure; procedure assumed EXTERNAL. [ABS]  
& ABS( TSTEP3( FID ) ) )  
------------------------------^  
ptrwrite3.f(152): error #6404: This name does not have a type, and must have an explicit type. [ABS]  
& ABS( TSTEP3( FID ) ) )  
-------------------------^  
ptrwrite3.f(288): error #6410: This name has not been declared as an array or a function. [MXREC3]  
MXREC3( FID ) = MAX( MXREC3( FID ), ABS( STEP ) )  
---------^  
ptrwrite3.f(288): warning #7319: This argument’s data type is incompatible with this intrinsic procedure; procedure assumed EXTERNAL. [MAX]  
MXREC3( FID ) = MAX( MXREC3( FID ), ABS( STEP ) )  
------------------------------^  
ptrwrite3.f(288): error #6404: This name does not have a type, and must have an explicit type. [MAX]  
MXREC3( FID ) = MAX( MXREC3( FID ), ABS( STEP ) )  
-------------------------^  
compilation aborted for ptrwrite3.f (code 1)  
Makefile:448: recipe for target ‘ptrwrite3.o’ failed  
make: \*\*\* [ptrwrite3.o] Error 1  
make: \*\*\* Waiting for unfinished jobs…  
pshut3.F(53): error #5102: Cannot open include file ‘STATE3.EXT’  
INCLUDE ‘STATE3.EXT’  
----------------^  
pshut3.F(72): error #6404: This name does not have a type, and must have an explicit type. [FINIT3]  
IF (.NOT. FINIT3) THEN  
------------------^  
pshut3.F(82): error #6404: This name does not have a type, and must have an explicit type. [COUNT3]  
DO FILE = 1, COUNT3  
---------------------^  
pshut3.F(84): error #6404: This name does not have a type, and must have an explicit type. [CDFID3]  
FID = CDFID3(FILE)  
-----------------^  
pshut3.F(86): error #6404: This name does not have a type, and must have an explicit type. [FLIST3]  
IF ((FID .GE. 0) .AND. (FLIST3(FILE) .NE. CMISS3)) THEN  
-----------------------------------^  
pshut3.F(86): warning #6043: This Hollerith or character constant is too long and cannot be used in the current numeric context. [’???’]  
IF ((FID .GE. 0) .AND. (FLIST3(FILE) .NE. CMISS3)) THEN  
-----------------------------------------------------^  
pshut3.F(99): error #6404: This name does not have a type, and must have an explicit type. [FTYPE3]  
IF( FTYPE3( FILE ) .EQ. LOC\_MPIGRD3 ) THEN  
------------------^  
pshut3.F(118): error #6404: This name does not have a type, and must have an explicit type. [LOGDEV]  
WRITE( LOGDEV,91010 )  
------------------------------^  
pshut3.F(120): error #6054: A CHARACTER data type is required in this context. [FLIST3]  
& ‘File name: ’ // FLIST3( FILE ) ,  
-----------------------------------------^  
pshut3.F(128): error #6410: This name has not been declared as an array or a function. [FLIST3]  
FLIST3(FILE) = CMISS3  
-----------^  
pshut3.F(129): error #6404: This name does not have a type, and must have an explicit type. [NVARS3]  
DO V = 1, NVARS3( FILE )  
---------------------^  
pshut3.F(130): error #6410: This name has not been declared as an array or a function. [ILAST3]  
ILAST3(V,FILE) = IMISS3  
--------------^  
pshut3.F(131): error #6410: This name has not been declared as an array or a function. [LDATE3]  
LDATE3(V,FILE) = IMISS3  
--------------^  
pshut3.F(132): error #6410: This name has not been declared as an array or a function. [LTIME3]  
LTIME3(V,FILE) = IMISS3  
--------------^  
pshut3.F(133): error #6410: This name has not been declared as an array or a function. [NDATE3]  
NDATE3(V,FILE) = IMISS3  
--------------^  
pshut3.F(134): error #6410: This name has not been declared as an array or a function. [NTIME3]  
NTIME3(V,FILE) = IMISS3  
--------------^  
pshut3.F(140): error #6303: The assignment operation or the binary expression operation is invalid for the data types of the two operands.  
FINIT3 = .FALSE.  
-----------------^  
compilation aborted for pshut3.F (code 1)  
Makefile:445: recipe for target ‘pshut3.o’ failed  
make: \*\*\* [pshut3.o] Error 1  
pwrite3.f(62): error #5102: Cannot open include file ‘STATE3.EXT’  
INCLUDE ‘STATE3.EXT’  
--------------^  
pwrite3.f(114): error #6404: This name does not have a type, and must have an explicit type. [FINIT3]  
IF ( .NOT. FINIT3 ) THEN  
-----------------^  
pwrite3.f(116): error #6404: This name does not have a type, and must have an explicit type. [LOGDEV]  
WRITE( LOGDEV,91010 ) ‘WRITE3(): I/O API not yet initialized.’  
----------------^  
pwrite3.f(142): error #6404: This name does not have a type, and must have an explicit type. [COUNT3]  
FID = INDEX1 ( FIL16, COUNT3, FLIST3 )  
------------------------------^  
pwrite3.f(142): error #6404: This name does not have a type, and must have an explicit type. [FLIST3]  
FID = INDEX1 ( FIL16, COUNT3, FLIST3 )  
--------------------------------------^  
pwrite3.f(152): error #6404: This name does not have a type, and must have an explicit type. [SDATE3]  
STEP = JSTEP3 ( JDATE, JTIME, SDATE3( FID ), STIME3( FID ),  
---------------------------------------^  
pwrite3.f(152): error #6404: This name does not have a type, and must have an explicit type. [STIME3]  
STEP = JSTEP3 ( JDATE, JTIME, SDATE3( FID ), STIME3( FID ),  
------------------------------------------------------^  
pwrite3.f(153): error #6404: This name does not have a type, and must have an explicit type. [TSTEP3]  
& ABS( TSTEP3( FID ) ) )  
------------------------------^  
pwrite3.f(153): warning #7319: This argument’s data type is incompatible with this intrinsic procedure; procedure assumed EXTERNAL. [ABS]  
& ABS( TSTEP3( FID ) ) )  
------------------------------^  
pwrite3.f(153): error #6404: This name does not have a type, and must have an explicit type. [ABS]  
& ABS( TSTEP3( FID ) ) )  
-------------------------^  
pwrite3.f(300): error #6410: This name has not been declared as an array or a function. [MXREC3]  
MXREC3( FID ) = MAX( MXREC3( FID ), ABS( STEP ) )  
---------^  
pwrite3.f(300): warning #7319: This argument’s data type is incompatible with this intrinsic procedure; procedure assumed EXTERNAL. [MAX]  
MXREC3( FID ) = MAX( MXREC3( FID ), ABS( STEP ) )  
------------------------------^  
pwrite3.f(300): error #6404: This name does not have a type, and must have an explicit type. [MAX]  
MXREC3( FID ) = MAX( MXREC3( FID ), ABS( STEP ) )  
-------------------------^  
compilation aborted for pwrite3.f (code 1)  
Makefile:448: recipe for target ‘pwrite3.o’ failed  
make: \*\*\* [pwrite3.o] Error 1  
CSQY\_DATA.F(524): remark #8291: Recommended relationship between field width ‘W’ and the number of fractional digits ‘D’ in this edit descriptor is ‘W\>=D+7’.  
READ( PHOTAB\_UNIT,’(A16,7X,F8.3,1X,40(1PE12.6,2X))’ )  
-----------------------------------------------------^  
CSQY\_DATA.F(540): remark #8291: Recommended relationship between field width ‘W’ and the number of fractional digits ‘D’ in this edit descriptor is ‘W\>=D+7’.  
READ( PHOTAB\_UNIT,’(A16,7X,F8.3,1X,40(1PE12.6,2X))’ )  
-----------------------------------------------------^  
CSQY\_DATA.F(577): remark #8291: Recommended relationship between field width ‘W’ and the number of fractional digits ‘D’ in this edit descriptor is ‘W\>=D+7’.  
READ( PHOTAB\_UNIT,’(A16,7X,F8.3,1X,40(1PE12.6,2X))’ )  
--------------------------------------------------^  
CSQY\_DATA.F(1247): remark #8291: Recommended relationship between field width ‘W’ and the number of fractional digits ‘D’ in this edit descriptor is ‘W\>=D+7’.  
READ( OPTICS\_UNIT,’(A30,1X,40(1PE12.6,2X))’ ) LAND\_EXPECT,  
------------------------------------------^  
**ERROR** while running make command

I think some command was missing for ptrwrite3.o & pshut3.o, but I am not sure.  
Did anyone have the same issue with Cheyenne?  
Thanks.

---

<div class="post-metadata">

**Author:** ![cjcoats](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@cjcoats](https://forum.cmascenter.org/u/cjcoats)\
**Post date:** [January 5, 2021, 12:39pm UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/2 "2021-01-05T12:39:07Z")

</div>

Before we get started here: **For netCDF4, you must be using I/O API 3.2**. See [https://cjcoats.github.io/ioapi/NEWSTUFF.html#oct302015](https://cjcoats.github.io/ioapi/NEWSTUFF.html#oct302015):

> Recent (post-4.4) versions of netCDF-Fortran have dropped the `NC*()` interfaces used; a major and tedious re-write was necessary to re-cast all 800+ `NC*()` calls in terms of netCDF-3-style `NF_*()` interfaces

**For compile errors, always include the _complete command line_ for the compile command that failed.,**

In this case, you must be missing an include-path directive to the correct I/O API `INCLUDE`-files: Since CMAQ uses a non-Standard “fixed-132” source format, this is the _ioapi/fixed\_src_ directory in your I/O API installation. Typically, this directive in your command line might be something like

_-I${HOME}/Models/ioapi-3.2/ioapi/fixed\_src_

.

---

<div class="post-metadata">

**Author:** ![ckuo6](https://avatars.discourse-cdn.com/v4/letter/c/b9bd4f/32.png) [@ckuo6](https://forum.cmascenter.org/u/ckuo6)\
**Post date:** [January 5, 2021, 4:31pm UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/3 "2021-01-05T16:31:59Z")

</div>

Hi cjcoats,

Thanks for your suggestion. I have compiled ioapi 3.2 with pgi compiler, and set the environment in cmaq\_config.sh as showed

```
case pgi:

    #> I/O API, netCDF, and MPI library locations
    setenv IOAPI_INCL_DIR /glade/scratch/chengpin/ioapi-3.2/Linux4_x86_64pg #> I/O API include header files
    setenv IOAPI_LIB_DIR /glade/scratch/chengpin/ioapi-3.2/Linux4_x86_64pg #> I/O API libraries
    setenv NETCDF_LIB_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/lib #> netCDF C directory path
    setenv NETCDF_INCL_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/include #> netCDF C directory path
    setenv NETCDFF_LIB_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/lib #> netCDF Fortran directory path
    setenv NETCDFF_INCL_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/include #> netCDF Fortran directory path
    setenv MPI_LIB_DIR /glade/u/apps/ch/opt/mpt/2.22 #> MPI directory path

    #> Compiler Aliases and Flags
    setenv myFC ifort
    setenv myCC pgcc
    setenv myLINK_FLAG # "-mp" openMP not supported w/ CMAQ
    setenv myFSTD "-O3"
    setenv myDBG "-O0 -g -Mbounds -Mchkptr -traceback -Ktrap=fp"
    setenv myFFLAGS "-Mfixed -Mextend -mcmodel=medium -tp px"
    setenv myFRFLAGS "-Mfree -Mextend -mcmodel=medium -tp px"
    setenv myCFLAGS "-O2"
    #setenv extra_lib "-lextra"
    #setenv mpi_lib "-lmpi" #> -lmpich for mvapich or -lmpi for openmpi
    setenv extra_lib "-lcurl"
    setenv mpi_lib "-lmpich" #> -lmpich for mvapich or -lmpi for openmpi

    breaksw

```

I also tried to modified the directory to /glade/scratch/chengpin/ioapi-3.2/Linux4\_x86\_64pg/ioapi/fixed\_src as your suggestion, and there are still some issues when compiling CCTM.

I show error message part in the log file here.

if ( 1 || ! -f /glade/scratch/chengpin/CMAQ\_REPO/UTIL/bldmake/bldmake\_pgi.exe ) then  
cd /glade/scratch/chengpin/CMAQ\_REPO/UTIL/bldmake/scripts  
./bldit\_bldmake.csh  
rm: No match.  
ifort: command line warning #10006: ignoring unknown option ‘-Mfixed’  
ifort: command line warning #10006: ignoring unknown option ‘-Mextend’  
ifort: command line remark #10148: option ‘-tp px’ not supported  
cfg\_module.f(47): error #5120: Unterminated character constant  
& (/ '.F ', '.f ', '.c ', ‘.F90’, ‘.f90’ /)  
-------------------------------------------------------------------^  
cfg\_module.f(47): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: , : (/ /)  
& (/ '.F ', ‘.f ‘, ‘.c ‘, ‘.F90’, ‘.f90’ /)  
----------------------------------------------------------------------------^  
cfg\_module.f(246): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
If ( (Len\_Trim(record) + Len\_Trim(nextrecord)) .Ge. (Len(record)) ) Then  
----------------------------------------------------------------------------------^  
cfg\_module.f(247): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ) ,  
Write(\*,’(//“ **ERROR** The maximum record length exceeded”)’)  
-------------------------------------------------------------------------^  
cfg\_module.f(287): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* \<CHAR\_CON\_KIND\_PARAM\> \<CHAR\_NAM\_KIND\_PARAM\> \<CHARACTER\_CONSTANT\> …  
If ( verbose ) Write( \*, ‘("make options set to ",a)’ ) Trim( make\_options )  
--------------------------------------------------------------------------------------^  
cfg\_module.f(301): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
If ( verbose ) Write( \*, ‘("Repository set to ",a)’ ) Trim( repo )  
----------------------------------------------------------------------------^  
cfg\_module.f(308): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
If ( verbose ) Write( \*, ‘("Mechanism set to ",a)’ ) Trim( mechanism )  
--------------------------------------------------------------------------------^  
cfg\_module.f(315): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
If ( verbose ) Write( \*, ‘("LIB\_BASE set to ",a)’ ) Trim( lib\_base )  
------------------------------------------------------------------------------^  
cfg\_module.f(354): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
If ( verbose ) Write( \*, ‘("F\_COMPILER set to ",a)’ ) Trim( f\_compiler )  
----------------------------------------------------------------------------------^  
cfg\_module.f(373): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
If ( verbose ) Write( \*, ‘("f\_flags set to ",a)’ ) Trim( f\_flags )  
----------------------------------------------------------------------------^  
cfg\_module.f(379): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
If ( verbose ) Write( \*, ‘("f90\_flags set to ",a)’ ) Trim( f90\_flags )  
--------------------------------------------------------------------------------^  
cfg\_module.f(386): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
If ( verbose ) Write( \*, ‘("C\_COMPILER set to ",a)’ ) Trim( c\_compiler )  
----------------------------------------------------------------------------------^  
cfg\_module.f(392): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
If ( verbose ) Write( \*, ‘("C\_FLAGS set to ",a)’ ) Trim( c\_flags )  
----------------------------------------------------------------------------^  
cfg\_module.f(405): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
If ( verbose ) Write( \*, ‘("CPP\_FLAGS set to ",a)’ ) Trim( cpp\_flags )  
--------------------------------------------------------------------------------^  
cfg\_module.f(412): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
If ( verbose ) Write( \*, ‘("LINKER set to ",a)’ ) Trim( linker )  
--------------------------------------------------------------------------^  
cfg\_module.f(418): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
If ( verbose ) Write( \*, ‘("LINK\_FLAGS set to ",a)’ ) Trim( link\_flags )  
----------------------------------------------------------------------------------^  
cfg\_module.f(430): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
If ( verbose ) Write( \*, ‘("NETCDF set to ",a)’ ) Trim( netcdf )  
--------------------------------------------------------------------------^  
cfg\_module.f(432): catastrophic error: Too many errors, exiting  
compilation aborted for cfg\_module.f (code 1)  
ifort: command line warning #10006: ignoring unknown option ‘-Mfixed’  
ifort: command line warning #10006: ignoring unknown option ‘-Mextend’  
ifort: command line remark #10148: option ‘-tp px’ not supported  
bldmake.f(73): error #5120: Unterminated character constant  
If ( status .ne. 0 ) Call error\_msg( ‘Cannot create FILE [Makefile]’ )  
-------------------------------------------^  
bldmake.f(73): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) , :  
If ( status .ne. 0 ) Call error\_msg( ‘Cannot create FILE [Makefile]’ )  
----------------------------------------------------------------------------^  
bldmake.f(147): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
Call help\_msg(‘Invalid configuration file argument:’ // Trim( cfgFile ) )  
---------------------------------------------------------------------------------^  
bldmake.f(161): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
Call help\_msg(‘Invalid arguments on command line:’ // Trim( argv ) )  
------------------------------------------------------------------------------^  
bldmake.f(208): error #5120: Unterminated character constant  
Call help\_msg( ‘Invalid arguments [’ // Trim(argv) // ‘] on command line’ )  
--------------------------------------------------------------^  
bldmake.f(208): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) , :  
Call help\_msg( ‘Invalid arguments [’ // Trim(argv) // ‘] on command line’ )  
-----------------------------------------------------------------------------------^  
bldmake.f(225): error #5082: Syntax error, found IDENTIFIER ‘DATEVALUES’ when expecting one of: :: ) ( , : \* ; . (/ + - ] /) ’ \*\* \> PRIVATE / …  
& dateValues( 2 ), dateValues( 3 ), dateValues( 1 ),  
---------^  
bldmake.f(250): error #5120: Unterminated character constant  
Write( _,’(" -git\_local Does NOT copy source files to BLD directory")’ )  
-----------------^  
bldmake.f(250): error #5082: Syntax error, found ‘’’ when expecting one of: ( \* \<CHAR\_CON\_KIND\_PARAM\> \<CHAR\_NAM\_KIND\_PARAM\> \<CHARACTER\_CONSTANT\> …  
Write( ,’(" -git\_local Does NOT copy source files to BLD directory")’ )  
---------------^  
bldmake.f(252): error #5120: Unterminated character constant  
Write( ,’(" -debug\_cctm Execute make with DEBUG option set to TRUE")’ )  
-----------------^  
bldmake.f(252): error #5082: Syntax error, found ‘’’ when expecting one of: ( \* \<CHAR\_CON\_KIND\_PARAM\> \<CHAR\_NAM\_KIND\_PARAM\> \<CHARACTER\_CONSTANT\> …  
Write( ,’(" -debug\_cctm Execute make with DEBUG option set to TRUE")’ )  
---------------^  
bldmake.f(253): error #5120: Unterminated character constant  
Write( ,’(" -isam\_cctm Execute make with ISAM option set to TRUE")’ )  
-----------------^  
bldmake.f(253): error #5082: Syntax error, found ‘’’ when expecting one of: ( \* \<CHAR\_CON\_KIND\_PARAM\> \<CHAR\_NAM\_KIND\_PARAM\> \<CHARACTER\_CONSTANT\> …  
Write( ,’(" -isam\_cctm Execute make with ISAM option set to TRUE")’ )  
---------------^  
bldmake.f(293): error #5120: Unterminated character constant  
Write( lfn, ‘("# Makefile generated for serial execution using program bldmake")’ )  
-----------------------^  
bldmake.f(293): error #5082: Syntax error, found ‘’’ when expecting one of: ( \* \<CHAR\_CON\_KIND\_PARAM\> \<CHAR\_NAM\_KIND\_PARAM\> \<CHARACTER\_CONSTANT\> …  
Write( lfn, ‘("# Makefile generated for serial execution using program bldmake")’ )  
---------------------^  
bldmake.f(295): error #5120: Unterminated character constant  
Write( lfn, ‘("# Makefile generated for parallel execution using program bldmake")’ )  
-----------------------^  
bldmake.f(295): error #5082: Syntax error, found ‘’’ when expecting one of: ( \* \<CHAR\_CON\_KIND\_PARAM\> \<CHAR\_NAM\_KIND\_PARAM\> \<CHARACTER\_CONSTANT\> …  
Write( lfn, ‘("# Makefile generated for parallel execution using program bldmake")’ )  
---------------------^  
bldmake.f(298): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
Write( lfn, ‘("# Generation date [",a,"]")’ ) Trim( currentDate )  
------------------------------------------------------------------------------^  
bldmake.f(299): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
Write( lfn, ‘("# Configuration file [",a,"]")’ ) Trim( cfgFile )  
--------------------------------------------------------------------------^  
bldmake.f(302): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: :: ) ( , : \* . % + - [ . = \*\* / // .LT. \< .LE. \<= .EQ. == …  
& Write( lfn, ‘("# With mechanism [",a,"]")’ ) Trim( mechanism )  
----------------------------------------------------------------------------^  
bldmake.f(307): error #5120: Unterminated character constant  
Write( lfn, ‘("# Full Compiler Paths when Makefile was Built:")’ )  
-----------------------^  
bldmake.f(309): catastrophic error: Too many errors, exiting  
compilation aborted for bldmake.f (code 1)  
ifort: command line warning #10006: ignoring unknown option ‘-Mfixed’  
ifort: command line warning #10006: ignoring unknown option ‘-Mextend’  
ifort: command line remark #10148: option ‘-tp px’ not supported  
parser.f(107): error #6404: This name does not have a type, and must have an explicit type. [NFIELD]  
If ( Index( delimiter, record( i:i ) ) .Gt. 0 ) nfields = nfields+1  
------------------------------------------------------------------^  
parser.f(213): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ) ,  
Subroutine getParsedField( record, delimiter, nth, field, includeDel )  
----------------------------------------------------------------------------^  
parser.f(213): error #6404: This name does not have a type, and must have an explicit type. [INCLUDED]  
Subroutine getParsedField( record, delimiter, nth, field, includeDel )  
----------------------------------------------------------------^  
parser.f(313): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( : % [ . = =\>  
If ( string( 1:1 ) .Ne. ‘"’ .And. string( last:last ) .Ne. ‘"’ ) Return  
-----------------------------------------------------------------------------^  
parser.f(313): error #6343: The statement following the Logical IF statement is invalid.  
If ( string( 1:1 ) .Ne. ‘"’ .And. string( last:last ) .Ne. ‘"’ ) Return  
------^  
parser.f(355): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( \<CHAR\_CON\_KIND\_PARAM\> \<CHAR\_NAM\_KIND\_PARAM\> \<CHARACTER\_CONSTANT\> \<INTEGER\_CONSTANT\> …  
If ( infield .And. string( i:i ) .Eq. inchar ) string( i:i ) = outchar  
-------------------------------------------------------------------------------^  
parser.f(376): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( \<CHAR\_CON\_KIND\_PARAM\> \<CHAR\_NAM\_KIND\_PARAM\> \<CHARACTER\_CONSTANT\> \<INTEGER\_CONSTANT\> …  
if ( ( k .Ge. 97 ) .And. ( k .Le. 122 ) ) str( i:i ) = Char( k - 32 )  
-----------------------------------------------------------------------------^  
compilation aborted for parser.f (code 1)  
ifort: command line warning #10006: ignoring unknown option ‘-Mfixed’  
ifort: command line warning #10006: ignoring unknown option ‘-Mextend’  
ifort: command line remark #10148: option ‘-tp px’ not supported  
utils.f(183): error #5120: Unterminated character constant  
Write(,’(“ **ERROR** opening scratch file to capture which results”/)’)  
------------------^  
utils.f(183): error #5082: Syntax error, found ‘’’ when expecting one of: ( \* \<CHAR\_CON\_KIND\_PARAM\> \<CHAR\_NAM\_KIND\_PARAM\> \<CHARACTER\_CONSTANT\> …  
Write(,’(“ **ERROR** opening scratch file to capture which results”/)’)  
----------------^  
utils.f(176): error #6404: This name does not have a type, and must have an explicit type. [TRI]  
Write(,’(" **ERROR** writing “,a,” to ",a/)’) Trim(cmdline), Trim(scrfile)  
---------------------------------------------------------------------^  
utils.f(232): error #5120: Unterminated character constant  
Write(,’(“ **ERROR** openning scratch file to capture pwd results”/)’)  
------------------^  
utils.f(232): error #5082: Syntax error, found ‘’’ when expecting one of: ( \* \<CHAR\_CON\_KIND\_PARAM\> \<CHAR\_NAM\_KIND\_PARAM\> \<CHARACTER\_CONSTANT\> …  
Write(,’(“ **ERROR** openning scratch file to capture pwd results”/)’)  
----------------^  
utils.f(225): error #6404: This name does not have a type, and must have an explicit type. [TRI]  
Write(_,’(" **ERROR** writing “,a,” to ",a/)’) Trim(cmdline), Trim(scrfile)  
---------------------------------------------------------------------^  
compilation aborted for utils.f (code 1)  
ifort: No match.

_ **ERROR** _ BLDMAKE Compile failed

---

<div class="post-metadata">

**Author:** ![cgnolte](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/cgnolte/32/53_2.png) [@cgnolte](https://forum.cmascenter.org/u/cgnolte)\
**Post date:** [January 5, 2021, 5:33pm UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/4 "2021-01-05T17:33:32Z")

</div>

> [@ckuo6](#):
>
> I show error message part in the log file here.
> 
> if ( 1 || ! -f /glade/scratch/chengpin/CMAQ\_REPO/UTIL/bldmake/bldmake\_pgi.exe ) then  
> cd /glade/scratch/chengpin/CMAQ\_REPO/UTIL/bldmake/scripts  
> ./bldit\_bldmake.csh  
> rm: No match.  
> ifort: command line warning #10006: ignoring unknown option ‘-Mfixed’

The first problem you’re having is that you think you’re using PGI, but in fact you’re getting Intel (ifort), which does not understand any of the PGI-formatted compiler directives.

---

<div class="post-metadata">

**Author:** ![ckuo6](https://avatars.discourse-cdn.com/v4/letter/c/b9bd4f/32.png) [@ckuo6](https://forum.cmascenter.org/u/ckuo6)\
**Post date:** [January 5, 2021, 6:26pm UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/5 "2021-01-05T18:26:35Z")

</div>

Hi cgnolte,

Really thanks for your response.

I think I need to ask one basic question which is how to know the compiler.  
I tried the myFC as the default setting (mpifort) or the ioapi compiler (pgi), and neither one can work.  
I will try to test other possible terms. Anyway, thanks for your time.

---

<div class="post-metadata">

**Author:** ![ckuo6](https://avatars.discourse-cdn.com/v4/letter/c/b9bd4f/32.png) [@ckuo6](https://forum.cmascenter.org/u/ckuo6)\
**Post date:** [January 5, 2021, 11:32pm UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/6 "2021-01-05T23:32:42Z")

</div>

For the update, I tried to module load pgi to replace intel before compilation, and change the environment settings to:

```
case pgi:

    #> I/O API, netCDF, and MPI library locations
    setenv IOAPI_INCL_DIR /glade/scratch/chengpin/ioapi-3.2/Linux4_x86_64pg #> I/O API include header files
    setenv IOAPI_LIB_DIR /glade/scratch/chengpin/ioapi-3.2/ioapi/fixed_src #> I/O API libraries
    setenv NETCDF_LIB_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/pgi/20.4/lib #> netCDF C directory path
    setenv NETCDF_INCL_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/pgi/20.4/include #> netCDF C directory path
    setenv NETCDFF_LIB_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/pgi/20.4/lib #> netCDF Fortran directory path
    setenv NETCDFF_INCL_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/pgi/20.4/include #> netCDF Fortran directory path
    setenv MPI_LIB_DIR /glade/u/apps/ch/opt/mpt/2.22 #> MPI directory path

    #> Compiler Aliases and Flags
    setenv myFC pgi
    setenv myCC pgcc
    setenv myLINK_FLAG # "-mp" openMP not supported w/ CMAQ
    setenv myFSTD "-O3"
    setenv myDBG "-O0 -g -Mbounds -Mchkptr -traceback -Ktrap=fp"
    setenv myFFLAGS "-Mfixed -Mextend -mcmodel=medium -tp px"
    setenv myFRFLAGS "-Mfree -Mextend -mcmodel=medium -tp px"
    setenv myCFLAGS "-O2"
    #setenv extra_lib "-lextra"
    #setenv mpi_lib "-lmpi" #> -lmpich for mvapich or -lmpi for openmpi
    setenv extra_lib "-lcurl"
    setenv mpi_lib "-lmpich" #> -lmpich for mvapich or -lmpi for openmpi

    breaksw

```

And I compiled under CCTM/scripts using:

./bldit\_cctm.csh pgi |& tee bldit.cctm.log

BLD\_CCTM\_v532\_pgi showed up in the same directory but only CCTM\_v532.cfg.bld in it, I cannot compile CCTM there.

I still try to fix it with my little coding capability.

Here’s the log file:

Compiler is set to pgi  
setenv CCTM\_SRC /glade/u/home/chengpin/CCTM/src  
set GlobInc = /glade/u/home/chengpin/CCTM/src/ICL  
set Mechs = /glade/u/home/chengpin/CCTM/src/MECHS  
setenv REPOROOT /glade/u/home/chengpin/CCTM/src  
set CompileBLDMAKE  
set CopySrc  
set ParOpt  
set make\_options = -j  
if ( 0 ) then  
if ( 0 ) then  
set VRSN = v532  
endif  
set EXEC = CCTM\_v532.exe  
set CFG = CCTM\_v532.cfg  
set ModGrid = grid/cartesian  
set DepMod = m3dry  
set ModAdv = wrf\_cons  
set ModHdiff = hdiff/multiscale  
set ModVdiff = vdiff/acm2\_m3dry  
set ModDepv = depv/m3dry  
set ModEmis = emis/emis  
set ModBiog = biog/beis3  
set ModPlmrs = plrise/smoke  
set ModCgrds = spcs/cgrid\_spcs\_nml  
set ModPhot = phot/inline  
set Mechanism = cb6r3\_ae7\_aq  
set ModGas = gas/ebi\_cb6r3\_ae7\_aq  
set ModAero = aero/aero7  
set ModCloud = cloud/acm\_ae7  
set ModUtil = util/util  
set ModDiag = diag  
set Tracer = trac0  
set ModPa = procan/pa  
set ModPvO3 = pv\_o3  
set ModISAM = isam  
set ModDDM3D = ddm3d  
setenv FC pgi  
set FP = pgi  
set CC = pgcc  
setenv BLDER /glade/scratch/chengpin/CMAQ\_REPO/UTIL/bldmake/bldmake\_pgi.exe  
set NETCDF = /glade/scratch/chengpin/CMAQ\_REPO/lib/x86\_64/pgi/netcdf/lib -lnetcdf  
set NETCDFF = /glade/scratch/chengpin/CMAQ\_REPO/lib/x86\_64/pgi/netcdff/lib -lnetcdff  
set PNETCDF = /glade/scratch/chengpin/CMAQ\_REPO/lib/x86\_64/pgi/pnetcdf/lib -lpnetcdf  
set FSTD = -O3  
set DBG = -O0 -g -Mbounds -Mchkptr -traceback -Ktrap=fp  
setenv F\_FLAGS -Mfixed -Mextend -mcmodel=medium -tp px  
set F90\_FLAGS = -Mfree -Mextend -mcmodel=medium -tp px  
set CPP\_FLAGS =  
set C\_FLAGS = -O2 -DFLDMN -I  
set LINK\_FLAGS =  
if ( ! -e /glade/u/home/chengpin || ! -e /glade/scratch/chengpin/CMAQ\_REPO/lib/x86\_64/pgi ) then  
echo Model repository base path: /glade/u/home/chengpin  
Model repository base path: /glade/u/home/chengpin  
echo library path: /glade/scratch/chengpin/CMAQ\_REPO/lib/x86\_64/pgi  
library path: /glade/scratch/chengpin/CMAQ\_REPO/lib/x86\_64/pgi  
if ( 0 ) then  
setenv CMAQ\_MODEL /glade/u/home/chengpin  
echo default Model repository path: /glade/u/home/chengpin  
default Model repository path: /glade/u/home/chengpin  
endif  
set BLD\_OS = `uname -s`  
uname -s  
if ( Linux != Linux ) then  
if ( 0 ) then  
if ( 0 ) then  
set PIO =  
endif  
if ( m3dry == m3dry ) then  
set cpp\_depmod = -Dm3dry\_opt  
else if ( m3dry == stage ) then  
if ( 1 ) then  
echo Parallel; set MPI flags  
Parallel; set MPI flags  
set ModStenex = STENEX/se  
set ModPario = PARIO  
set ModPar = par/mpi  
set PARIO = /glade/u/home/chengpin/PARIO  
set STENEX = /glade/u/home/chengpin/STENEX  
set PAR = ( -Dparallel )  
set Popt = SE  
set seL = se\_snl  
set LIB2 = -lioapi  
set LIB3 = -lmpich -lcurl  
set Str1 = ( // Parallel / Include message passing definitions )  
set Str2 = ( include SUBST\_MPI mpif.h ; )  
else  
if ( 0 ) then  
set SENS =  
endif  
set ModMech = MECHS/cb6r3\_ae7\_aq  
if ( cb6r3\_ae7\_aq == cb6r3m\_ae7\_kmtbr ) then  
set ModTrac = MECHS/trac0  
if ( 0 ) then  
set POT =  
endif  
set Bld = /glade/scratch/chengpin/CMAQ\_REPO/CCTM/scripts/BLD\_CCTM\_v532\_pgi  
if ( ! -e /glade/scratch/chengpin/CMAQ\_REPO/CCTM/scripts/BLD\_CCTM\_v532\_pgi ) then  
if ( ! -d /glade/scratch/chengpin/CMAQ\_REPO/CCTM/scripts/BLD\_CCTM\_v532\_pgi ) then  
endif  
cd /glade/scratch/chengpin/CMAQ\_REPO/CCTM/scripts/BLD\_CCTM\_v532\_pgi  
set ICL\_PAR = /glade/u/home/chengpin/CCTM/src/ICL/fixed/mpi  
set ICL\_CONST = /glade/u/home/chengpin/CCTM/src/ICL/fixed/const  
set ICL\_FILES = /glade/u/home/chengpin/CCTM/src/ICL/fixed/filenames  
set ICL\_EMCTL = /glade/u/home/chengpin/CCTM/src/ICL/fixed/emctrl  
if ( 1 ) then  
set ICL\_MPI = .  
endif  
if ( 1 ) then  
/bin/cp: No match.  
/bin/cp: No match.  
/bin/cp: No match.  
/bin/cp: No match.  
if ( 1 ) then  
/bin/cp -fp ./mpif.h /glade/scratch/chengpin/CMAQ\_REPO/CCTM/scripts/BLD\_CCTM\_v532\_pgi  
/bin/cp: cannot stat ‘./mpif.h’: No such file or directory  
endif  
set ICL\_PAR = .  
set ICL\_CONST = .  
set ICL\_FILES = .  
set ICL\_EMCTL = .  
if ( 1 ) then  
set ICL\_MPI = .  
endif  
endif  
set STX1 = ( -DSUBST\_BARRIER=SE\_BARRIER -DSUBST\_GLOBAL\_MAX=SE\_GLOBAL\_MAX -DSUBST\_GLOBAL\_MIN=SE\_GLOBAL\_MIN -DSUBST\_GLOBAL\_MIN\_DATA=SE\_GLOBAL\_MIN\_DATA -DSUBST\_GLOBAL\_TO\_LOCAL\_COORD=SE\_GLOBAL\_TO\_LOCAL\_COORD -DSUBST\_GLOBAL\_SUM=SE\_GLOBAL\_SUM -DSUBST\_GLOBAL\_LOGICAL=SE\_GLOBAL\_LOGICAL -DSUBST\_LOOP\_INDEX=SE\_LOOP\_INDEX -DSUBST\_SUBGRID\_INDEX=SE\_SUBGRID\_INDEX )  
set STX2 = ( -DSUBST\_HI\_LO\_BND\_PE=SE\_HI\_LO\_BND\_PE -DSUBST\_SUM\_CHK=SE\_SUM\_CHK -DSUBST\_SE\_INIT=SE\_INIT -DSUBST\_INIT\_ARRAY=SE\_INIT\_ARRAY -DSUBST\_COMM=SE\_COMM -DSUBST\_MY\_REGION=SE\_MY\_REGION -DSUBST\_SLICE=SE\_SLICE -DSUBST\_GATHER=SE\_GATHER -DSUBST\_DATA\_COPY=SE\_DATA\_COPY -DSUBST\_IN\_SYN=SE\_IN\_SYN )  
if ( wrf\_cons == wrf\_cons ) then  
set ModCpl = couple/gencoor\_wrf\_cons  
set ModHadv = hadv/ppm  
set ModVadv = vadv/wrf\_cons  
else if ( wrf\_cons == local\_cons ) then  
set Cfile = /glade/scratch/chengpin/CMAQ\_REPO/CCTM/scripts/BLD\_CCTM\_v532\_pgi/CCTM\_v532.cfg.bld  
set quote = "  
echo  
if ( 1 ) then  
echo make\_options “-j”;  
echo  
endif  
echo model CCTM\_v532.exe;  
echo  
echo repo /glade/u/home/chengpin/CCTM/src;  
echo  
echo mechanism cb6r3\_ae7\_aq;  
echo  
echo lib\_base /glade/scratch/chengpin/CMAQ\_REPO/lib/x86\_64/pgi;  
echo  
echo lib\_1 ioapi/lib;  
echo  
echo lib\_2 ioapi/include\_files;  
echo  
if ( 1 ) then  
echo lib\_3 “mpi -I.”;  
echo  
endif  
echo  
echo lib\_4 ioapi/lib;  
echo  
set text = " -Dparallel -Dm3dry\_opt -DSUBST\_BARRIER=SE\_BARRIER -DSUBST\_GLOBAL\_MAX=SE\_GLOBAL\_MAX -DSUBST\_GLOBAL\_MIN=SE\_GLOBAL\_MIN -DSUBST\_GLOBAL\_MIN\_DATA=SE\_GLOBAL\_MIN\_DATA -DSUBST\_GLOBAL\_TO\_LOCAL\_COORD=SE\_GLOBAL\_TO\_LOCAL\_COORD -DSUBST\_GLOBAL\_SUM=SE\_GLOBAL\_SUM -DSUBST\_GLOBAL\_LOGICAL=SE\_GLOBAL\_LOGICAL -DSUBST\_LOOP\_INDEX=SE\_LOOP\_INDEX -DSUBST\_SUBGRID\_INDEX=SE\_SUBGRID\_INDEX -DSUBST\_HI\_LO\_BND\_PE=SE\_HI\_LO\_BND\_PE -DSUBST\_SUM\_CHK=SE\_SUM\_CHK -DSUBST\_SE\_INIT=SE\_INIT -DSUBST\_INIT\_ARRAY=SE\_INIT\_ARRAY -DSUBST\_COMM=SE\_COMM -DSUBST\_MY\_REGION=SE\_MY\_REGION -DSUBST\_SLICE=SE\_SLICE -DSUBST\_GATHER=SE\_GATHER -DSUBST\_DATA\_COPY=SE\_DATA\_COPY -DSUBST\_IN\_SYN=SE\_IN\_SYN";  
echo cpp\_flags " -Dparallel -Dm3dry\_opt -DSUBST\_BARRIER=SE\_BARRIER -DSUBST\_GLOBAL\_MAX=SE\_GLOBAL\_MAX -DSUBST\_GLOBAL\_MIN=SE\_GLOBAL\_MIN -DSUBST\_GLOBAL\_MIN\_DATA=SE\_GLOBAL\_MIN\_DATA -DSUBST\_GLOBAL\_TO\_LOCAL\_COORD=SE\_GLOBAL\_TO\_LOCAL\_COORD -DSUBST\_GLOBAL\_SUM=SE\_GLOBAL\_SUM -DSUBST\_GLOBAL\_LOGICAL=SE\_GLOBAL\_LOGICAL -DSUBST\_LOOP\_INDEX=SE\_LOOP\_INDEX -DSUBST\_SUBGRID\_INDEX=SE\_SUBGRID\_INDEX -DSUBST\_HI\_LO\_BND\_PE=SE\_HI\_LO\_BND\_PE -DSUBST\_SUM\_CHK=SE\_SUM\_CHK -DSUBST\_SE\_INIT=SE\_INIT -DSUBST\_INIT\_ARRAY=SE\_INIT\_ARRAY -DSUBST\_COMM=SE\_COMM -DSUBST\_MY\_REGION=SE\_MY\_REGION -DSUBST\_SLICE=SE\_SLICE -DSUBST\_GATHER=SE\_GATHER -DSUBST\_DATA\_COPY=SE\_DATA\_COPY -DSUBST\_IN\_SYN=SE\_IN\_SYN";  
echo  
echo f\_compiler pgi;  
echo  
echo fstd “-O3”;  
echo  
echo dbg “-O0 -g -Mbounds -Mchkptr -traceback -Ktrap=fp”;  
echo  
echo f\_flags “-Mfixed -Mextend -mcmodel=medium -tp px”;  
echo  
echo f90\_flags “-Mfree -Mextend -mcmodel=medium -tp px”;  
echo  
echo c\_compiler pgcc;  
echo  
echo c\_flags “-O2 -DFLDMN -I”;  
echo  
echo link\_flags “”;  
echo  
echo ioapi “-lioapi”;  
echo  
echo netcdf “-lnetcdf”;  
echo  
echo netcdff “-lnetcdff”;  
echo  
if ( 1 ) then  
echo mpich “-lmpich -lcurl”;  
echo  
endif  
echo include SUBST\_PE\_COMM ./PE\_COMM.EXT;  
echo include SUBST\_CONST ./CONST.EXT;  
echo include SUBST\_FILES\_ID ./FILES\_CTM.EXT;  
echo include SUBST\_EMISPRM ./EMISPRM.EXT;  
echo  
if ( 1 ) then  
echo // Parallel / Include message passing definitions  
echo include SUBST\_MPI ./mpif.h;  
endif  
echo  
set text = stenex or se\_noop  
echo // options are stenex or se\_noop  
echo Module STENEX/se;  
if ( 1 ) then  
set text = // parallel executable; stenex and pario included  
echo // parallel executable; stenex and pario included  
echo Module PARIO;  
else  
echo  
set text = par, par\_nodistr and par\_noop  
echo // options are par, par\_nodistr and par\_noop  
if ( 1 ) then  
echo Module par/mpi;  
endif  
echo  
if ( 0 ) then  
set text = driver  
echo // options are driver  
echo Module driver;  
echo  
set text = cartesian  
echo // options are cartesian  
echo Module grid/cartesian;  
echo  
set text = Init  
echo // options are Init  
echo Module init;  
echo  
set text = gencoor\_wrf\_cons and gencoor\_local\_cons  
echo // options are gencoor\_wrf\_cons and gencoor\_local\_cons  
echo Module couple/gencoor\_wrf\_cons;  
echo  
set text = ppm  
echo // options are ppm  
echo Module hadv/ppm;  
echo  
set text = wrf\_cons and local\_cons  
echo // options are wrf\_cons and local\_cons  
echo Module vadv/wrf\_cons;  
echo  
set text = multiscale  
echo // options are multiscale  
echo Module hdiff/multiscale;  
echo  
set text = acm2\_m3dry or acm2\_stage  
echo // options are acm2\_m3dry or acm2\_stage  
echo Module vdiff/acm2\_m3dry;  
echo  
set text = m3dry or stage  
echo // options are m3dry or stage  
echo Module depv/m3dry;  
echo  
set text = emis  
echo // options are emis  
echo Module emis/emis;  
echo  
set text = beis3  
echo // options are beis3  
echo Module biog/beis3;  
echo  
set text = smoke  
echo // options are smoke  
echo Module plrise/smoke;  
echo  
set text = cgrid\_spcs\_nml and cgrid\_spcs\_icl  
echo // options are cgrid\_spcs\_nml and cgrid\_spcs\_icl  
echo Module spcs/cgrid\_spcs\_nml;  
echo  
set text = inline and table  
echo // options are inline and table  
echo Module phot/inline;  
echo  
set text = gas chemistry solvers  
echo // gas chemistry solvers  
set text = smvgear, ros3, and ebi\_; see ‘gas chemistry mechanisms’ for   
echo // options are smvgear, ros3, and ebi\_; see ‘gas chemistry mechanisms’ for   
echo Module gas/ebi\_cb6r3\_ae7\_aq;  
echo  
set MechList = cb6mp\_ae6\_aq, cb6r3\_ae6\_aq, cb6r3\_ae7\_aq, cb6r3\_ae7\_aqkmt2, cb6r3m\_ae7\_kmtbr, racm2\_ae6\_aq, saprc07tc\_ae6\_aq, saprc07tic\_ae6i\_aq, saprc07tic\_ae6i\_aqkmti, saprc07tic\_ae7i\_aq, saprc07tic\_ae7i\_aqkmt2  
set text = gas chemistry mechanisms  
echo // gas chemistry mechanisms  
set text = cb6mp\_ae6\_aq, cb6r3\_ae6\_aq, cb6r3\_ae7\_aq, cb6r3\_ae7\_aqkmt2, cb6r3m\_ae7\_kmtbr, racm2\_ae6\_aq, saprc07tc\_ae6\_aq, saprc07tic\_ae6i\_aq, saprc07tic\_ae6i\_aqkmti, saprc07tic\_ae7i\_aq, saprc07tic\_ae7i\_aqkmt2  
echo // options are cb6mp\_ae6\_aq, cb6r3\_ae6\_aq, cb6r3\_ae7\_aq, cb6r3\_ae7\_aqkmt2, cb6r3m\_ae7\_kmtbr, racm2\_ae6\_aq, saprc07tc\_ae6\_aq, saprc07tic\_ae6i\_aq, saprc07tic\_ae6i\_aqkmti, saprc07tic\_ae7i\_aq, saprc07tic\_ae7i\_aqkmt2  
echo Module MECHS/cb6r3\_ae7\_aq;  
echo  
set text = tracer modules  
echo // tracer modules  
echo // options are trac0, trac1  
echo Module MECHS/trac0;  
echo

if ( 0 ) then  
set text = aero6  
echo // options are aero6  
echo Module aero/aero7;  
echo  
set text = acm\_ae6, acm\_ae6\_kmt, acm\_ae7\_kmt2, acm\_ae6\_mp, acm\_ae7  
echo // options are acm\_ae6, acm\_ae6\_kmt, acm\_ae7\_kmt2, acm\_ae6\_mp, acm\_ae7  
echo Module cloud/acm\_ae7;  
echo  
set text = // compile for inline process analysis  
echo // compile for inline process analysis  
echo Module procan/pa;  
echo  
set text = // compile for integrated source apportionment method  
echo // compile for integrated source apportionment method  
echo Module isam;  
echo  
if ( 0 ) then  
set text = util  
echo // options are util  
echo Module util/util;  
echo  
set text = diag  
echo // options are diag  
echo Module diag;  
echo  
set text = stm  
echo // options are stm  
echo Module stm;  
echo  
set text = cio  
echo // options are cio  
echo Module cio;  
echo  
if ( 0 ) then  
unalias mv rm  
if ( 1 || ! -f /glade/scratch/chengpin/CMAQ\_REPO/UTIL/bldmake/bldmake\_pgi.exe ) then  
cd /glade/u/home/chengpin/UTIL/bldmake/scripts  
/glade/u/home/chengpin/UTIL/bldmake/scripts: No such file or directory.

---

<div class="post-metadata">

**Author:** ![cgnolte](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.cmascenter.org/cgnolte/32/53_2.png) [@cgnolte](https://forum.cmascenter.org/u/cgnolte)\
**Post date:** [January 6, 2021, 1:48am UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/7 "2021-01-06T01:48:24Z")

</div>

First, what version of CMAQ are you using?

Your problem is in the model build step, before the compiler even tries to compile.

You are arriving at this portion of the build script:

```
#> Recompile BLDMAKE from source if requested or if it does not exist
 if ( $?CompileBLDMAKE || ! -f $BLDER ) then
   cd ${CMAQ_REPO}/UTIL/bldmake/scripts
   ./bldit_bldmake.csh
 endif

```

But you are getting a “no such file or directory” error when you try to cd to `${CMAQ_REPO}/UTIL/bldmake/scripts`. Please check the value of CMAQ\_REPO. Is that path correct?

---

<div class="post-metadata">

**Author:** ![ckuo6](https://avatars.discourse-cdn.com/v4/letter/c/b9bd4f/32.png) [@ckuo6](https://forum.cmascenter.org/u/ckuo6)\
**Post date:** [January 6, 2021, 3:55am UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/8 "2021-01-06T03:55:18Z")

</div>

Dear @cgnolte

Thanks for your suggestion and time.  
The CMAQ version is 5.3.2, and I have modified the ${CMAQ\_REPO} and made sure it exist.  
Now the compilation stopped, and the error message showed:

if ( 1 || ! -f /glade/scratch/chengpin/CMAQ\_REPO/UTIL/bldmake/bldmake\_pgi.exe ) then  
cd /glade/u/home/chengpin/CMAQ\_REPO/UTIL/bldmake/scripts  
./bldit\_bldmake.csh  
rm: No match.  
pgi: Command not found.  
pgi: Command not found.  
pgi: Command not found.  
pgi: Command not found.  
pgi: No match.

_ **ERROR** _ BLDMAKE Compile failed

But I already set the compiler to pgi and am still trying to figure out the reason.

---

<div class="post-metadata">

**Author:** ![ckuo6](https://avatars.discourse-cdn.com/v4/letter/c/b9bd4f/32.png) [@ckuo6](https://forum.cmascenter.org/u/ckuo6)\
**Post date:** [January 7, 2021, 2:57am UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/9 "2021-01-07T02:57:01Z")

</div>

Update my status.  
Now it can compile with the compiler pgf90, which is the same in /ioapi-3.2/ioapi/Makeinclude.Linux4\_x86\_64pg

But CCTM compilation stops with the following error:

“distr\_env.c”, line 52: warning: variable “numprocs” was set but never used  
int myid, numprocs;  
^

“distr\_env.c”, line 54: warning: variable “ret” was set but never used  
int ret, i, error;  
^

“distr\_env.c”, line 54: warning: variable “error” was set but never used  
int ret, i, error;

The whole log file is uploaded.  
Did anyone encounter the similar issue or have any thought about the error?  
[bldit.cctm.log.txt](https://forum.cmascenter.org/uploads/short-url/zeG1wvAEYqqDxEZwwBT380ERGaJ.txt) (184.9 KB)

---

<div class="post-metadata">

**Author:** ![cjcoats](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@cjcoats](https://forum.cmascenter.org/u/cjcoats)\
**Post date:** [January 7, 2021, 12:39pm UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/10 "2021-01-07T12:39:01Z")

</div>

That’s just a warning, not an error. It’s just a hold-over (I think) from a previous and more primitive generation of this routine that did use that variable.

---

<div class="post-metadata">

**Author:** ![ckuo6](https://avatars.discourse-cdn.com/v4/letter/c/b9bd4f/32.png) [@ckuo6](https://forum.cmascenter.org/u/ckuo6)\
**Post date:** [January 7, 2021, 4:33pm UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/11 "2021-01-07T16:33:23Z")

</div>

Hi, @cjcoats,

I am not sure these errors will cause compilation failure in the end or not.  
Because in the end, it shows:

Makefile generated  
**ERROR** while running make command

---

<div class="post-metadata">

**Author:** ![cjcoats](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@cjcoats](https://forum.cmascenter.org/u/cjcoats)\
**Post date:** [January 7, 2021, 4:38pm UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/12 "2021-01-07T16:38:45Z")

</div>

Now, what you’re seeing is a failure to find the MPI implementation specific libraries and header files. Go back and read the “building the CCTM” instructions.

---

<div class="post-metadata">

**Author:** ![ckuo6](https://avatars.discourse-cdn.com/v4/letter/c/b9bd4f/32.png) [@ckuo6](https://forum.cmascenter.org/u/ckuo6)\
**Post date:** [January 9, 2021, 4:48am UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/13 "2021-01-09T04:48:41Z")

</div>

Hi all

Update my latest status.  
Now I used intel-compiled IOAPI 3.2 to compile CCTM (Linux2\_x86\_64ifort).  
CMAQ version is 5.3.2.

Module list is

1. ncarenv/1.3 2) intel/19.0.5 3) ncarcompilers/0.5.0 4) mpt/2.22 5) netcdf/4.7.4

Environment setting in config\_cmaq.csh is

```
case intel:

    #> I/O API, netCDF, and MPI library locations
    setenv IOAPI_INCL_DIR /glade/scratch/chengpin/ioapi-3.2/ioapi/fixed_src #ioapi_inc_intel #> I/O API include header files
    setenv IOAPI_LIB_DIR /glade/scratch/chengpin/ioapi-3.2/Linux2_x86_64ifort #> I/O API libraries
    setenv NETCDF_LIB_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/lib #netcdf_lib_intel #> netCDF C directory path
    setenv NETCDF_INCL_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/include #> netCDF C directory path
    setenv NETCDFF_LIB_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/lib #> netCDF Fortran directory path
    setenv NETCDFF_INCL_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/include #> netCDF Fortran directory path
    setenv MPI_LIB_DIR /glade/u/apps/ch/opt/mpt/2.22 #> MPI directory path

    #> Compiler Aliases and Flags
    #> set the compiler flag -qopt-report=5 to get a model optimization report in the build directory with the optrpt extension
    setenv myFC ifort
    setenv myCC icc
    setenv myFSTD "-O3 -fno-alias -mp1 -fp-model source -ftz -simd -align all -xHost -vec-guard-write -unroll-aggressive"
    setenv myDBG "-O0 -g -check bounds -check uninit -fpe0 -fno-alias -ftrapuv -traceback"
    setenv myLINK_FLAG #"-qopenmp-simd" openMP not supported w/ CMAQ
    setenv myFFLAGS "-fixed -132"
    setenv myFRFLAGS "-free"
    setenv myCFLAGS "-O2"
    setenv extra_lib "-lcurl"
    #chsetenv extra_lib ""
    setenv mpi_lib "-lmpich" #> No Library specification needed for mpiifort
                         #> -lmpich for mvapich
                         #> -lmpi for openmpi

    breaksw

```

Now I got an error message with:

pm3warn.f(54): error #7002: Error in opening the compiled module file. Check INCLUDE paths. [M3UTILIO]  
USE M3UTILIO ! i/o api  
----------^  
compilation aborted for pm3warn.f (code 1)  
subdmap.f(72): error #7002: Error in opening the compiled module file. Check INCLUDE paths. [M3UTILIO]  
USE M3UTILIO ! i/o api  
----------^  
compilation aborted for subdmap.f (code 1)

I checked the netcdf and netcdff lib/include path, and they do exist.  
Did anyone encounter a similar problem before?  
Thanks.

---

<div class="post-metadata">

**Author:** ![cjcoats](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@cjcoats](https://forum.cmascenter.org/u/cjcoats)\
**Post date:** [January 9, 2021, 12:26pm UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/14 "2021-01-09T12:26:29Z")

</div>

Now you’ve either botched the I/O API :: CMAQ match-up or you’ve botched the INCLUDE directives. To repeat myself:

- **For compile errors, always include the _complete command line_ for the compile command that failed.** , and
- Go back and read the “building the CCTM” instructions.

---

<div class="post-metadata">

**Author:** ![ckuo6](https://avatars.discourse-cdn.com/v4/letter/c/b9bd4f/32.png) [@ckuo6](https://forum.cmascenter.org/u/ckuo6)\
**Post date:** [January 11, 2021, 4:18am UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/15 "2021-01-11T04:18:43Z")

</div>

Hope it’s my last question. I tried to detailed my steps but it could be redundant. Thanks for reading.

Thanks to @cjcoats 's advice.  
I read the IOAPI and CCTM compiling instructions carefully, and the error still occurred at CCTM compiling.

Module reset first.

1. ncarenv/1.3 2) intel/19.0.5 3) ncarcompilers/0.5.0 4) mpt/2.22 5) netcdf/4.7.4

In IOAPI 3.2, I set

```auto
setenv BIN Linux2_x86_64ifort
setenv BASEDIR $cwd
setenv CPLMODE nocpl

```

command out the following setting in the file, /ioapi/Makeinclude.Linux2\_x86\_64ifort:

```auto
OMPFLAGS = # -fopenmp 
OMPLIBS = # -fopenmp
...
ARCHLIB = #-Bstatic

```

I asked NCAR help desk and they provided me the build script here.

#!/bin/bash  
BLDID=$(date +%m%dT%H%M)  
DEBUG=false  
DEBUGFLAGS="-O0 -g -traceback"  
IOVER=3.2-20200828

module purge  
module reset

rm -fr ioapi-IOVER tar -xf ioapi-{IOVER}.tar.gz

cd ioapi-${IOVER}

export BIN=Linux2\_x86\_64ifort  
export BASEDIR=$PWD

export CPLMODE=nocpl  
ln -s Makefile.template Makefile  
cp ioapi/Makefile.$CPLMODE ioapi/Makefile  
cp m3tools/Makefile.$CPLMODE m3tools/Makefile

for F in $(find . -name Makefile); do  
sed -i “s|(BASEDIR =).\*|\1 $PWD|” $F  
done

F=$(find . -name Makeinclude.$BIN)  
sed -i “s|-openmp|-qopenmp|g” $F  
sed -i ‘/^ARCHFLAGS/a \ -DIOAPI\_NCF4=1 \’ $F  
[[$DEBUG == true]] && sed -i “s|-O3|$DEBUGFLAGS|g” $F

The modified Makeinclude.Linux2\_x86\_64ifort also in the gz file, ioapi-3.2-20200828.tar.gz.  
Then I ./build.sh and IOAPI 3.2 was successfully compiled.

Second, in config\_cmaq.sh, I modified the fold directories and modified the netcdf, netcdff, and mpi’s lib and include directories. I used openMPI here and module load openMPI.  
The module list is

1. ncarenv/1.3 2) intel/19.0.5 3) ncarcompilers/0.5.0 4) netcdf/4.7.4 5) openmpi/4.0.5

Environment setting in config\_cmaq.sh is

```
    #> I/O API, netCDF, and MPI library locations
    setenv IOAPI_INCL_DIR /glade/scratch/chengpin/ioapi-3.2/ioapi/fixed_src
    setenv IOAPI_LIB_DIR /glade/scratch/chengpin/ioapi-3.2/Linux2_x86_64gfort
    setenv NETCDF_LIB_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/lib     
    setenv NETCDF_INCL_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/include 
    setenv NETCDFF_LIB_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/lib     
    setenv NETCDFF_INCL_DIR /glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/include 
    setenv MPI_LIB_DIR /glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5 

    #> Compiler Aliases and Flags
    #> set the compiler flag -qopt-report=5 to get a model optimization report in the build directory with the optrpt extension
    setenv myFC ifort
    setenv myCC icc
    setenv myFSTD "-O3 -fno-alias -mp1 -fp-model source -ftz -simd -align all -xHost -vec-guard-write -unroll-aggressive"
    setenv myDBG "-O0 -g -check bounds -check uninit -fpe0 -fno-alias -ftrapuv -traceback"
    setenv myLINK_FLAG "-qopenmp" 
    setenv myFFLAGS "-fixed -132"
    setenv myFRFLAGS "-free"
    setenv myCFLAGS "-O2"
    setenv extra_lib "-lcurl"
    setenv mpi_lib "-lmpi"  
    breaksw

```

Then I compiled icon and bcon before CCTM, and they succeeded.  
When I used ./bldit\_cctm.csh intel |& tee build\_cctm.log to compile CCTM, the log showed the error which I considered that could be due to mpi lib file settings, because bcon and icon would not use mpi for compilation. l listed 5 lines here and the detailed log file in the attachment.  
[build\_cctm.log.txt](https://forum.cmascenter.org/uploads/short-url/5SFAdnrgsCuvyKRHn3ut8ry1cBd.txt) (408.9 KB)

/usr/lib64/gcc/x86\_64-suse-linux/4.8/…/…/…/…/x86\_64-suse-linux/bin/ld: se\_data\_send\_module.o: in function `se_data_send_module_mp_se_1d_data_send_': se_data_send_module.f:(.text+0x433): undefined reference to `mpi\_send\_’  
/usr/lib64/gcc/x86\_64-suse-linux/4.8/…/…/…/…/x86\_64-suse-linux/bin/ld: se\_data\_send\_module.o: in function `se_data_send_module_mp_se_2d_data_send_': se_data_send_module.f:(.text+0xc84): undefined reference to `mpi\_send\_’  
/usr/lib64/gcc/x86\_64-suse-linux/4.8/…/…/…/…/x86\_64-suse-linux/bin/ld: se\_data\_send\_module.o: in function `se_data_send_module_mp_se_2de_data_send_': se_data_send_module.f:(.text+0x1455): undefined reference to `mpi\_send\_’  
/usr/lib64/gcc/x86\_64-suse-linux/4.8/…/…/…/…/x86\_64-suse-linux/bin/ld: se\_data\_send\_module.o: in function `se_data_send_module_mp_se_3d_data_send_': se_data_send_module.f:(.text+0x1f11): undefined reference to `mpi\_send\_’  
/usr/lib64/gcc/x86\_64-suse-linux/4.8/…/…/…/…/x86\_64-suse-linux/bin/ld: se\_data\_send\_module.o: in function `se_data_send_module_mp_se_3de_data_send_': se_data_send_module.f:(.text+0x28af): undefined reference to `mpi\_send\_’

So I module show openMPI to show the path here:

Built on Sun Nov 22 20:33:42 MST 2020  
Modules used:  
ncarenv/1.3  
intel/19.0.5  
ncarcompilers/0.5.0  
]])  
prepend\_path(“PATH”,"/glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/bin")  
prepend\_path(“CPATH”,"/glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/include")  
prepend\_path(“FPATH”,"/glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/include")  
prepend\_path(“PKG\_CONFIG\_PATH”,"/glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/lib/pkgconfig")  
prepend\_path(“LIBRARY\_PATH”,"/glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/lib")  
prepend\_path(“LD\_LIBRARY\_PATH”,"/glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/lib")  
prepend\_path(“MANPATH”,"/glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/share/man")  
setenv(“MPI\_ROOT”,"/glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/")  
setenv(“MPI\_VERSION”,“4.0.5”)  
append\_path(“MODULEPATH”,"/glade/u/apps/ch/modulefiles/default/openmpi/4.0.5/intel/19.0.5")  
prepend\_path(“PATH”,"/glade/u/apps/ch/opt/ncarcompilers/0.5.0/intel/19.0.5/mpi")  
setenv(“MPICC\_CC”,“icc”)  
setenv(“MPICXX\_CXX”,“icpc”)  
setenv(“MPIF90\_F90”,“ifort”)  
setenv(“NCAR\_ROOT\_OPENMPI”,"/glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/")  
setenv(“NCAR\_RANK\_OPENMPI”,“1000”)  
setenv(“NCAR\_LIBS\_OPENMPI”,"-lrt -ldl")  
setenv(“NCAR\_LDFLAGS\_OPENMPI”,"/glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/lib")  
setenv(“LMOD\_MPI”,“openmpi”)  
setenv(“LMOD\_MPI\_VERSION”,“4.0.5”)  
setenv(“UCX\_NET\_DEVICES”,“mlx5\_0:1”)

What in /glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/lib include:  
libmca\_common\_dstore.la  
libmca\_common\_sm.so.40  
libmpi\_usempif08.la  
libopen-pal.so.40  
mpi\_f08\_interfaces.mod  
libmca\_common\_dstore.so  
libmca\_common\_sm.so.40.20.0  
libmpi\_usempif08.so  
libopen-pal.so.40.20.5  
mpi\_f08.mod  
libmca\_common\_dstore.so.1  
libmca\_common\_ucx.la  
libmpi\_usempif08.so.40  
libopen-rte.la  
mpi\_f08\_types.mod  
libmca\_common\_dstore.so.1.0.2  
libmca\_common\_ucx.so  
libmpi\_usempif08.so.40.21.1  
libopen-rte.so  
mpi.mod  
libmca\_common\_monitoring.la  
libmca\_common\_ucx.so.40  
libmpi\_usempi\_ignore\_tkr.la  
libopen-rte.so.40  
ompi\_monitoring\_prof.la  
libmca\_common\_monitoring.so  
libmca\_common\_ucx.so.40.20.2  
libmpi\_usempi\_ignore\_tkr.so  
libopen-rte.so.40.20.5  
ompi\_monitoring\_prof.so  
libmca\_common\_monitoring.so.50  
libmpi.la  
libmpi\_usempi\_ignore\_tkr.so.40  
liboshmem.la  
openmpi  
libmca\_common\_monitoring.so.50.10.0  
libmpi\_mpifh.la  
libmpi\_usempi\_ignore\_tkr.so.40.20.0  
liboshmem.so  
pkgconfig  
libmca\_common\_ompio.la  
libmpi\_mpifh.so  
libompitrace.la  
liboshmem.so.40  
pmix  
libmca\_common\_ompio.so  
libmpi\_mpifh.so.40  
libompitrace.so  
liboshmem.so.40.22.1  
pmpi\_f08\_interfaces.mod  
libmca\_common\_ompio.so.41  
libmpi\_mpifh.so.40.20.2  
libompitrace.so.40  
mpi\_ext.mod  
libmca\_common\_ompio.so.41.19.4  
libmpi.so  
libompitrace.so.40.20.0  
mpi\_f08\_callbacks.mod  
libmca\_common\_sm.la  
libmpi.so.40  
libopen-pal.la  
mpi\_f08\_ext.mod  
libmca\_common\_sm.so  
libmpi.so.40.20.5  
libopen-pal.so  
mpi\_f08\_interfaces\_callbacks.mod

What in /glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/include include:  
mpi-ext.h  
mpif-constants.h  
mpif.h  
mpif-io-handles.h  
mpi.h  
openmpi  
pshmemx.h  
shmem.h  
mpif-c-constants-decl.h  
mpif-externals.h  
mpif-handles.h  
mpif-sentinels.h  
mpi\_portable\_platform.h  
openshmem  
shmem-compat.h  
shmemx.h  
mpif-config.h  
mpif-ext.h  
mpif-io-constants.h  
mpif-sizeof.h  
mpp  
pshmem.h  
shmem.fh

I still don’t know why config\_cmaq.sh cannot capture the mpi lib and include file.  
Or did I missing something during the compiling?  
Thanks for your time to read my words.

---

<div class="post-metadata">

**Author:** ![cjcoats](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@cjcoats](https://forum.cmascenter.org/u/cjcoats)\
**Post date:** [January 11, 2021, 3:34pm UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/16 "2021-01-11T15:34:12Z")

</div>

o repeat myself again:

- **For compile errors, always include the _complete command line_ for the compile command that failed.** ,

and don’t make me have to spend lots of time digging through the log to find problems.

```
ifort -qopenmp se_bndy_copy_info_ext.o ... subhfile.o -L/glade/scratch/chengpin/CMAQ_REPO/lib/x86_64/intel/ioapi/lib -lioapi -L/glade/scratch/chengpin/CMAQ_REPO/lib/x86_64/intel/netcdff/lib -lnetcdff -L/glade/scratch/chengpin/CMAQ_REPO/lib/x86_64/intel/netcdf/lib -lnetcdf -o CCTM_v532.exe

```

There’s nothing here about MPI libraries. And that’s the problem.

---

<div class="post-metadata">

**Author:** ![ckuo6](https://avatars.discourse-cdn.com/v4/letter/c/b9bd4f/32.png) [@ckuo6](https://forum.cmascenter.org/u/ckuo6)\
**Post date:** [January 11, 2021, 4:26pm UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/17 "2021-01-11T16:26:21Z")

</div>

That’s the issue I have. The lib and include specify the path which has the following files, but I don’t realize why the CCTM compilation cannot capture them.

These are what under /glade/scratch/chengpin/CMAQ\_REPO/lib/x86\_64/intel/netcdf/lib  
libh5bzip2.la  
libhdf5hl\_fortran.la  
libnetcdff.la  
libh5bzip2.so  
libhdf5\_hl\_fortran.so  
libnetcdff.settings  
libhdf5.a  
libhdf5hl\_fortran.so  
libnetcdff.so  
libhdf5\_cpp.a  
libhdf5hl\_fortran.so.100  
libnetcdff.so.7  
libhdf5\_cpp.la  
libhdf5hl\_fortran.so.100.0.6  
libnetcdff.so.7.0.0  
libhdf5\_cpp.so  
libhdf5\_hl.la  
libnetcdf.la  
libhdf5\_cpp.so.103  
libhdf5\_hl.so  
libnetcdf.settings  
libhdf5\_cpp.so.103.3.0  
libhdf5\_hl.so.100  
libnetcdf.so  
libhdf5\_fortran.a  
libhdf5\_hl.so.100.1.4  
libnetcdf.so.18  
libhdf5\_fortran.la  
libhdf5.la  
libnetcdf.so.18.0.0  
libhdf5\_fortran.so  
libhdf5.settings  
libsz.a  
libhdf5\_fortran.so.102  
libhdf5.so  
libsz.la  
libhdf5\_fortran.so.102.1.0  
libhdf5.so.103  
libsz.so  
libhdf5\_hl.a  
libhdf5.so.103.3.0  
libsz.so.2  
libhdf5\_hl\_cpp.a  
libnetcdf.a  
libsz.so.2.0.0  
libhdf5\_hl\_cpp.la  
libnetcdf\_c++4.a  
libz.a  
libhdf5\_hl\_cpp.so  
libnetcdf\_c++4.la  
libz.so  
libhdf5\_hl\_cpp.so.100  
libnetcdf\_c++4.so  
libz.so.1  
libhdf5\_hl\_cpp.so.100.1.5  
libnetcdf\_c++4.so.1  
libz.so.1.2.11  
libhdf5\_hl\_fortran.a  
libnetcdf\_c++4.so.1.1.0  
pkgconfig  
libhdf5hl\_fortran.a  
libnetcdff.a

These are what under /glade/scratch/chengpin/CMAQ\_REPO/lib/x86\_64/intel/netcdff/lib  
libh5bzip2.la  
libhdf5hl\_fortran.la  
libnetcdff.la  
libh5bzip2.so  
libhdf5\_hl\_fortran.so  
libnetcdff.settings  
libhdf5.a  
libhdf5hl\_fortran.so  
libnetcdff.so  
libhdf5\_cpp.a  
libhdf5hl\_fortran.so.100  
libnetcdff.so.7  
libhdf5\_cpp.la  
libhdf5hl\_fortran.so.100.0.6  
libnetcdff.so.7.0.0  
libhdf5\_cpp.so  
libhdf5\_hl.la  
libnetcdf.la  
libhdf5\_cpp.so.103  
libhdf5\_hl.so  
libnetcdf.settings  
libhdf5\_cpp.so.103.3.0  
libhdf5\_hl.so.100  
libnetcdf.so  
libhdf5\_fortran.a  
libhdf5\_hl.so.100.1.4  
libnetcdf.so.18  
libhdf5\_fortran.la  
libhdf5.la  
libnetcdf.so.18.0.0  
libhdf5\_fortran.so  
libhdf5.settings  
libsz.a  
libhdf5\_fortran.so.102  
libhdf5.so  
libsz.la  
libhdf5\_fortran.so.102.1.0  
libhdf5.so.103  
libsz.so  
libhdf5\_hl.a  
libhdf5.so.103.3.0  
libsz.so.2  
libhdf5\_hl\_cpp.a  
libnetcdf.a  
libsz.so.2.0.0  
libhdf5\_hl\_cpp.la  
libnetcdf\_c++4.a  
libz.a  
libhdf5\_hl\_cpp.so  
libnetcdf\_c++4.la  
libz.so  
libhdf5\_hl\_cpp.so.100  
libnetcdf\_c++4.so  
libz.so.1  
libhdf5\_hl\_cpp.so.100.1.5  
libnetcdf\_c++4.so.1  
libz.so.1.2.11  
libhdf5\_hl\_fortran.a  
libnetcdf\_c++4.so.1.1.0  
pkgconfig  
libhdf5hl\_fortran.a  
libnetcdff.a

Or said, did the lib lack the files with the specific name?  
What are the file names we need under lib and include?

And sorry for spending your time.

---

<div class="post-metadata">

**Author:** ![lizadams](https://avatars.discourse-cdn.com/v4/letter/l/49beb7/32.png) [@lizadams](https://forum.cmascenter.org/u/lizadams)\
**Post date:** [January 11, 2021, 5:03pm UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/18 "2021-01-11T17:03:54Z")

</div>

Hi,  
If your FC definition is set to mpifort, and your modules are loaded correctly, then the mpi libraries will be found automatically.

Please check the output of the following command:

```auto
mpifort -show

```

It should display something like the following if you have the following modules loaded.  
Currently Loaded Modules:

1. gcc/9.1.0 2) openmpi\_4.0.1/gcc\_9.1.0

```auto
gfortran -I/nas/longleaf/apps-dogwood/mpi/gcc_9.1.0/openmpi_4.0.1/include -pthread -I/nas/longleaf/apps-dogwood/mpi/gcc_9.1.0/openmpi_4.0.1/lib -Wl,-rpath -Wl,/nas/longleaf/apps-dogwood/mpi/gcc_9.1.0/openmpi_4.0.1/lib -Wl,--enable-new-dtags -L/nas/longleaf/apps-dogwood/mpi/gcc_9.1.0/openmpi_4.0.1/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi

```

---

<div class="post-metadata">

**Author:** ![ckuo6](https://avatars.discourse-cdn.com/v4/letter/c/b9bd4f/32.png) [@ckuo6](https://forum.cmascenter.org/u/ckuo6)\
**Post date:** [January 11, 2021, 5:59pm UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/19 "2021-01-11T17:59:28Z")

</div>

Hi @lizadams  
Thanks for your replying.

Here’s what I got using mpifort -show

ifort -I/glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/include -Wl,-Bstatic -lnetcdff -lnetcdf -lhdf5hl\_fortran -lhdf5\_hl -lhdf5\_fortran -lhdf5 -lsz -lz -Wl,-Bdynamic -lm -ldl -lrt -ldl -Wl,–disable-new-dtags -L/glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/lib -L/glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/lib -Wl,-rpath,/glade/u/apps/ch/opt/netcdf/4.7.4/intel/19.0.5/lib -Wl,-rpath,/glade/u/apps/ch/opt/openmpi/4.0.5/intel/19.0.5/lib

Here I have a question.  
If I used mpt/2.22 for IOAPI compilation and openMPI/4.0.5 for CCTM compilation.  
Will this different MPI contradict themself?

---

<div class="post-metadata">

**Author:** ![cjcoats](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@cjcoats](https://forum.cmascenter.org/u/cjcoats)\
**Post date:** [January 11, 2021, 6:29pm UTC](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995/20 "2021-01-11T18:29:18Z")

</div>

If you compiled the pnetcdf-enabled version, then you need to use the same MPI version throughout.  
If you didn’t, the I/O API library is independent of MPI version (mainly depends on Fortran compiler and optimization-level).

[Next page](https://forum.cmascenter.org/t/compiling-cmaq-on-cheyenne-at-ncar/1995.md?page=2)