Basically, CMAQ is exiting in like few seconds from the moment it starts to run with the error above in the standard output. The standard error is below:
intable: Subscript out of range.
intable: Subscript out of range.
intable: Subscript out of range.
intable: Subscript out of range.
intable: Subscript out of range.
intable: Subscript out of range.
intable: Subscript out of range.
intable: Subscript out of range.
/bin/rm: No match.
srun: TOPOLOGY: warning -- no switch can reach all nodes through its descendants.Do not use route/topology
Warning: Process to core binding is enabled and OMP_NUM_THREADS is set to non-zero (1) value
If your program has OpenMP sections, this can cause over-subscription of cores and consequently poor performance
To avoid this, please re-run your application after setting MV2_ENABLE_AFFINITY=0
Use MV2_USE_THREAD_WARNING=0 to suppress this message
srun: error: c-207-9: tasks 0,20,22,26-27,29: Exited with exit code 2
srun: error: c-207-23: tasks 39-40,51-52,54,57-58: Exited with exit code 2
tail: cannot open ābuff_CMAQ_CCTMv531_user_20200511_194816_704849307.txtā for reading: No such file or directory
(standard_in) 2: syntax error
(standard_in) 1: syntax error
(standard_in) 1: syntax error
What the āI/O STATUS = 10ā means is compiler-dependent. If you are using Intel ifort, then it means that OPEN(...STATUS = 'NEW'...) as in INITLOG3 refuses to over-write an existing file. Remove the $LOGFILE before running CMAQ.
Actually, I have CLOBBER_DATA set as TRUE, and I do not see log files - only the small file CCTM_v531_intel17.2_2016_CONUS_20140102.cfg as the output. I am a little unsure about the mpirun part of the code:
Edit: Actually, before the log files are placed in user specified $LOGDIR, they are initially at CMAQ_Project/ and may have been moved under $LOGDIR if CMAQ runs.
run_cmaq_rasel.csh (804 Bytes) Hy Iām facing similar kinds of errors. Hereās my job error output:
*** ERROR in INIT3/INITLOG3 ***
Error opening log file on unit 99
I/O STATUS = 17
File: CTM_LOG_000.v532_gcc_AQF5X_20200601
srun: error: NODE067: tasks 48-63: Exited with exit code 2
srun: error: NODE065: tasks 32-47: Exited with exit code 2
srun: error: NODE063: tasks 0-15: Exited with exit code 2
srun: error: NODE064: tasks 16-31: Exited with exit code 2
real 2.25
user 0.01
sys 0.01
** Runscript Detected an Error: CGRID file was not written. **
** This indicates that CMAQ was interrupted or an issue **
** exists with writing output. The runscript will now **
** abort rather than proceeding to subsequent days. **
Start Day: 2020-06-01
End Day: 2020-06-01
Number of Simulation Days: 1
Domain Name: AQF5X
Number of Grid Cells: 4099550 (ROW x COL x LAY)
Number of Layers: 35
Number of Processes: 64
All times are in seconds.
Num Day Wall Time
01 2020-06-01 2.25
Total Time = 2.25
Avg. Time = 2.25
What compiler was used to build this CCTM executable? ā I/O status values vary from compiler vendor to compiler vendor, and āI/O STATUS = 17ā cannot be interpreted without knowing what compiler was used.
At a further guess, the log-file already existed when you tried to run the program: is this the case?
The I/O API refuses to clobber existing logfile-data because that would ādestroy the chain of evidenceā, disrupting the integrity of the model-management system.
@cjcoats
I removed the log files (CTM_LOG_061.v532_gccā¦) in my run directory before I run my script. However, still, itās giving me the same error. After run If when I open log file I see following error:
ā>> WARNING in subroutine RDTFLAG
Time step error for file: GR_EMIS_001
M3WARN: DTBUF 12:00:00 June 1, 2020 (2020153:120000)
>>--->> WARNING in subroutine XTRACT3
Time step not available for file: GR_EMIS_001
M3WARN: DTBUF 12:00:00 June 1, 2020 (2020153:120000)
*** ERROR ABORT in subroutine retrieve_time_de on PE 000
Could not extract GR_EMIS_001 file
PM3EXIT: DTBUF 12:00:00 June 1, 2020
Date and time 12:00:00 June 1, 2020 (2020153:120000)
If I recall correctly, whenever I got āCould not extract GR_EMIS_001 fileā error, it would be because the 2d gridded emission stream fed to CMAQ as GR_EMIS_001 in the run script did not have a specific timestamp, which in your case would be 12:00:00 June 1, 2020 (2020153:120000). Could you try to do āncdump -v TFLAGā of the GR_EMIS_001 file and check if it has time stamp of 2020153:120000?
BTW, you can usually get some clue by looking at the log-message generated by routine OPEN3() for file GR_EMIS_001: it will tell you the fileās starting date&time, time-step, and number of steps ā something like:
"A" opened as OLD:READ-ONLY File name "/work/SCRATCH/HTR_CRO_2D_G0.std04b.200821806" File type GRDDED3 Execution ID "RTOFS-prod-2008218-06" Grid name "US_45_STD04B_CRO" Dimensions: 93 rows, 129 cols, 1 lays, 40 vbles NetCDF ID: 4 opened as READONLY Starting date and time 2008218:060000 (6:00:00 Aug. 5, 2008) Timestep 001500 (0:15:00 hh:mm:ss) Maximum current record number 481
indicating that file $A starts at 6:00:00 Aug. 5, 2008 and has 481 steps with 15-minute timesteps (for a duration of 32 hours (counting fenceposts on both ends --2008218:060000 through 2008218:140000 ). Thereās a reason I put in a lot of work on log-messages: theyāre there to help you.
@skunwar@cjcoats Iāve figured out the error. My emission input file name was different than the one Iāve mentioned in CCTM script. After correcting it, itās running fine. Thank you so much for your help.
Iām using gcc and gfort compilers with mpich. I have followed all the recommendations of this forum (CLOBBER_DATA TRUE, removed all LOGS) but I got the same error. Could you guide me to solve this error?
Hi,
Iām having the following issue with INIT3/INITLOG3:
*** ERROR in INIT3/INITLOG3 ***
Error opening log file on unit 99
I/O STATUS = 9
DESCRIPTION: permission to access file denied, unit 99, file /nasa/hpe/mpt/2.25_rhel79/lib/CTM_LOG_000.v532_intel_Bench_2016_12SE1_20160701
File: CTM_LOG_000.v532_intel_Bench_2016_12SE1_20160701
Iāve looked in the ioapi source code at init3.F90 and initlog3.F but havenāt been able to decode unit 99 or I/O STATUS = 9. Can you please let me know what this I/O STATUS is referring to? The issue seems to be that itās trying to write the CTM_LOG* files to the directory that contains mpirun, which is a directory containing my shared computerās module files. Iāve correctly defined $WORKDIR and $LOGDIR. Iām using an intel compiler.
This issue just began because the computerās operating system was changed from SLES to TOSS. On SLES, CMAQ ran correctly. Using the same CMAQ_REPO but re-building the CMAQ_PROJECT using new modules available on TOSS is what caused this error.
I/O error status numbers are compiler specific, so one needs to know what the underlying compiler for your mpif90 is (mpif90 is actually a script around conventional Fortran+C compilers that knows which libraries and which include-files to use, so you are highly dependent upon the underlying compiler.)
severe (9): Permission to access file denied
FOR$IOS_PERACCFIL. Check the permissions of the specified file and whether the network device is mapped and available. Make sure the correct file and device was being accessed. Change the protection, specific file, or process used before rerunning the program.
If itās some other compiler, google for that compiler and āruntime error codesā. If itās gfortran, well, donāt expect anything useful (thatās a major reason itās not my preferred compiler) ;-(
Thank you! Iām using Intel icc so that error code makes sense with the error in the log file:
*** ERROR in INIT3/INITLOG3 ***
Error opening log file on unit 99
I/O STATUS = 9
DESCRIPTION: permission to access file denied, unit 99, file /nasa/hpe/mpt/2.25_rhel79/lib/CTM_LOG_000.v532_intel_Bench_2016_12SE1_20160701
File: CTM_LOG_000.v532_intel_Bench_2016_12SE1_20160701
However, Iām still unsure why CMAQ is trying to write the CRM_LOG* files to that location (/nasa/hpe/mpt/2.25_rhel79/lib/), which is the $MPI_LIB_DIR. Could it be an issue with the MPI module that Iām using; maybe itās somehow changing the working directory to $MPI_LIB_DIR instead of $WORKDIR? Please let me know if youād like me to attach any of the CMAQ files.
