Met4moves fails at end of code

SMOKE
1

Hello,

I am having an issue running met4moves for a custom 4 km domain using SMOKE 5.1. It appears there aren’t any similar issues posted on here. I found run_met4moves_2016j_v2_12US2.csh and run_metcombine_2016j_12US2.csh on the 2016_beta platform. Everything works in the combine script, but I get this error running met4moves:

     "SMOKE_OUTFILE" opened as UNKNOWN(R-W)
     File name "/glade/derecho/scratch/lsheji/TAIWAN_SMOKE/ge_dat/afdust/met4moves/SMOKE_DAILY_2020fc_v2_20j_4NE5_2020001-2020059.ncf"
     File type GRDDED3
     Execution ID "????????????????"
     Grid name "4NE5_CROSS"
     Dimensions: 327 rows, 348 cols, 1 lays, 2 vbles
     NetCDF ID:    131072  opened as READWRITE
     Starting date and time  2020001:000000 (0:00:00   Jan. 1, 2020)
     Timestep                          240000 (24:00:00 hh:mm:ss)
     Maximum current record number         0

     File "MOVES_OUTFILE" opened for output on unit:  96
     /glade/derecho/scratch/lsheji/TAIWAN_SMOKE/ge_dat/afdust/met4moves/MOVES_DAILY_2020fc_v2_20j_4NE5_2020001-2020059.txt


     File "MOVES_RH_OUTFILE" opened for output on unit:  94
     /glade/derecho/scratch/lsheji/TAIWAN_SMOKE/ge_dat/afdust/met4moves/MOVES_RH_DAILY_2020fc_v2_20j_4NE5_2020001-2020059.txt

     Value for DAYLIGHT_EXEMPT not defined;returning default:   FALSE

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x145f322428ff in ???
#1  0x40e44d in ???
#2  0x4121ca in ???
#3  0x145f3222be6b in ???
#4  0x145f3222bf34 in ???
#5  0x403039 in ???
        at ../sysdeps/x86_64/start.S:120
#6  0xffffffffffffffff in ???
Segmentation fault (core dumped)

For reference, I am only running it for 2 months (Jan. and Feb.). I tried running it with csh in the terminal and by submitting it to the PBS on Derecho with the same error.

2

It could be useful to re-run with a debug/traceback version of the executable: then the backtrace would give subroutines and line numbers at which the seg-fault is occurring.

You will want to build using the flags from one of the I/O API’s *Makeinclude.dbg configuration files… you can use The Easy Guide® for Building SMOKE to show you how to build such an executable, if you need.

3

@cjcoats Here are the line numbers!

Backtrace for this error:
#0  0x155365c578ff in ???
#1  0x40e3ad in met4moves
	at /gpfs/csfs1/work/liweilai/nei2020/smoke/SMOKE_v5.2/SMOKE/src/movesmrg/met4moves.f:869
#2  0x41212a in main
	at /gpfs/csfs1/work/liweilai/nei2020/smoke/SMOKE_v5.2/SMOKE/src/movesmrg/met4moves.f:42
Segmentation fault (core dumped)

Examining line 869, it appears to be primarily a memory issue (assuming it is referencing the line prior):

 859         ALLOCATE( FUELCNTY( NREFC,NFUEL ), STAT=IOS )
 860         CALL CHECKMEM( IOS, 'FUELCNTY', PROGNAME )
 861         ALLOCATE( TKFUEL( NREFC,NFUEL,24 ), STAT=IOS )
 862         CALL CHECKMEM( IOS, 'TKFUEL', PROGNAME )
 863         ALLOCATE( TKREFHR( 24 ), STAT=IOS )
 864         CALL CHECKMEM( IOS, 'TKREFHR', PROGNAME )
 865         ALLOCATE( MAXTFUEL( NREFC,NFUEL ), STAT=IOS )
 866         CALL CHECKMEM( IOS, 'MAXTFUEL', PROGNAME )
 867         ALLOCATE( MINTFUEL( NREFC,NFUEL ), STAT=IOS )
 868         CALL CHECKMEM( IOS, 'MINTFUEL', PROGNAME )
 869         FUELIDX  = 0
 870         FUELCNTY = 0
 871         TKFUEL   = 0.0
 872         TKREFHR  = 0.0
 873         MAXTFUEL = BADVAL3
4

OK, then it’s a memory-use problem. What does the limit command on the system you’re using say about memoryuse and vmemoryuse? Can you set them to unlimited?

[This is made rather messier by the fact that some systems use “lazy allocation” where the ALLOCATEs are allowed to over-subscribe available memory, and then any actual failure doesn’t manifest until you try to actually use the memory, which is wha line 869 does. ;-( ]

5

@cjcoats I am allowed to set those, but they still fail in the same place with the same error…