Error in runing CCTM-ISAM

CMAQ Run Time Errors and Issues
1

Hello，all, I got some problems in runing CMAQ5.0.2-ISAM, could anyone help me ?

The model crashed down once I qsub theCCTM-ISAM runing script. But CCTM can run successfully. I have got the same errors before, I solved them by reducing tracked variables from 1253 to 543. Now, I can’t run ISAM again even with vaiables of 543.
What should I do to resolve this error, please? It’s been days I have been on this. Kindly help please.

The ERROR messages in CTM_log_files showed as showed below:

 >>--->> WARNING in subroutine GET_AERO_EMIS:INTERPX
 Error reading PMC from EMIS_1
 M3WARN:  DTBUF 0:01:30   Dec. 27, 2019 (2019361:000130)

 >>--->> WARNING in subroutine GET_AERO_EMIS on PE 086
 Could not read PMC from EMIS_1
 M3WARN:  DTBUF 0:01:30   Dec. 27, 2019 (2019361:000130)
 Error allocating internal buffer for BUFVGTP3()
 Timesteps:  LAST  2019361:000000; NEXT  2019361:010000

 >>--->> WARNING in subroutine GET_SSEMIS:INTERPX
 Error reading QV from MET_CRO_3D
 M3WARN:  DTBUF 0:01:30   Dec. 27, 2019 (2019361:000130)


 *** ERROR ABORT in subroutine GET_SSEMIS on PE 086
 Could not interpolate QV from MET_CRO_3D

PM3EXIT: DTBUF 0:01:30 Dec. 27, 2019
Date and time 0:01:30 Dec. 27, 2019 (2019361:000130)

In the
[cli_53]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 538976288) - process 53
[cli_7]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 538976288) - process 7
[cli_173]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 538976288) - process 173
[cli_189]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 538976288) - process 189
[cli_169]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 538976288) - process 169
[cli_122]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 538976288) - process 122
[cli_123]: aborting job:

[proxy:0:0@node1] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:912): assert (!closed) failed
[proxy:0:0@node1] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:0@node1] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event
[proxy:0:1@node4] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:912): assert (!closed) failed
[proxy:0:1@node4] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status

2

Hi Ybzhang,

This appears to be an I/O API error message about CMAQ not being able to extract the required variables from the input files. One suggestion is to examine the header information for the input files that may be causing the ERROR ABORT.

From your log, it looks like there are two errors

Could not read PMC from EMIS_1
 M3WARN:  DTBUF 0:01:30   Dec. 27, 2019 (2019361:000130)
 Error allocating internal buffer for BUFVGTP3()

and

Error reading QV from MET_CRO_3D

I am going to describe a method to examine the input file for MET_CRO_3D using the benchmark data that I have, as I don’t have access to your LOG files.

Find the path and filename of the MET_CRO_3D file in your CTM_LOG file using grep to search for the MET_CRO_3D definition, and then use the ncdump -h command to print out the header information for that file and save it to a txt file, and upload it to this bug report.

Here is an example of how to do this.

grep -A 1 MET_CRO_3D CTM_LOG_031.v531_intel_Bench_2016_12SE1_20160702

output:

     "MET_CRO_3D" opened as OLD:READ-ONLY   
     File name "/proj/ie/proj/CMAS/CMAQ/CMAQ_v5.3.1/openmpi_3.1.4_intel_18.2/data/CMAQv5.3.1_Benchmark_2Day_Input/2016_12SE1/met/mcipv5.0/METCRO3D_160702.nc"

Use the ncdump -h command to export the header information in the NetCDF file to a txt file.

ncdump -h /proj/ie/proj/CMAS/CMAQ/CMAQ_v5.3.1/openmpi_3.1.4_intel_18.2/data/CMAQv5.3.1_Benchmark_2Day_Input/2016_12SE1/met/mcipv5.0/METCRO3D_160702.nc > METCRO3D_160702.header.txt

Then examine the header information in the METCRO3D_160702.header.txt file to find how QV is defined.
float QV(TSTEP, LAY, ROW, COL) ;
QV:long_name = "QV " ;
QV:units = "kg kg-1 " ;
QV:var_desc = "water vapor mixing ratio " ;

Is the issue that the MET files don’t have data for the dates/times period that you are running CMAQ?

You will need to use ncdump -h to examine the PMC variable in the EMIS_1 to determine if this variable is available in the input file.

Thanks,
Liz