Afdust_adj and othafdust_adj failing because of xportfrac

SMOKE
1

Hello,

I’m having issues running the Annual_othafdust_adj_12US1_2016ff_16j.csh script and possibly related, Annual_afdust_adj_12US1_2016ff_16j.csh

Here is the log file:

***testing*** SRGPRO_PATH set by SRGPRO input: /projects/b1045/SMOKE.EMF/2016_beta/ge_dat/gridding/CONUS_CANADA_MEXICO_4K_beta/
Linux2_x86_64ifort
NOTE: Default MONTH setting to 'jan'
NOTE: AUTO_DELETE script setting is not set.
  Default is N to not delete SMOKE NCF intermediate and output files
/bin/ls: No match.
/bin/ls: No match.
/bin/ls: No match.
/bin/ls: No match.
SCRIPT NOTE: Setting months with no partial months
months = 12
smk_dat = /projects/b1045/SMOKE.EMF/2016_beta/smoke4.6/scripts/smk_dates/2016/smk_merge_dates_201512.txt
days = 31
date = 20151208

 This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3
 I/O Applications Programming Interface, [I/O API] which is
 built on top of the netCDF I/O library (Copyright 1993, 1996
 University Corporation for Atmospheric Research/Unidata
 Program) and the PVM parallel-programming library (from
 Oak Ridge National Laboratory).  Copyright (C) 1992-2002 MCNC
 and Carlie J. Coats, Jr., and 2003-2006 Baron Advanced
 Meteorological Systems, LLC and released under the GNU LGPL
 License, version 2.1.  See URL

     http://www.baronams.com/products/ioapi/LGPL.txt

 for conditions of use.

 Library release tag: $JDate: 2005333 $

 $Id: @(#) ioapi library version 3.0 $
 netCDF version "3.6.2" of Dec  5 2008 11:55:27 $

ERROR: Running xportfrac adjustments

There’s a similar error with Annual_afdust_adj_12US1_2016ff_16j.CONUS4K_d02.csh

There error for this ends with the same message, but also includes a segmentation fault – here is the log file:

months = 12
smk_dat = /projects/b1045/SMOKE.EMF/2016_beta/smoke4.6/scripts/smk_dates/2016/smk_merge_dates_201512.txt
days = 31
date = 20151208
date = 20151209

     This program uses the EPA-AREAL/MCNC-EnvPgms/BAMS Models-3
     I/O Applications Programming Interface, [I/O API] which is
     built on top of the netCDF I/O library (Copyright 1993, 1996
     University Corporation for Atmospheric Research/Unidata
     Program) and the PVM parallel-programming library (from
     Oak Ridge National Laboratory).  Copyright (C) 1992-2002 MCNC
     and Carlie J. Coats, Jr., and 2003-2006 Baron Advanced
     Meteorological Systems, LLC and released under the GNU LGPL
     License, version 2.1.  See URL

         http://www.baronams.com/products/ioapi/LGPL.txt

     for conditions of use.

     Library release tag: $JDate: 2005333 $

     $Id: @(#) ioapi library version 3.0 $
     netCDF version "3.6.2" of Dec  5 2008 11:55:27 $

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
mult.x             000000000044C402  Unknown               Unknown  Unknown
mult.x             000000000045ED03  Unknown               Unknown  Unknown
mult.x             000000000045E8ED  Unknown               Unknown  Unknown
mult.x             0000000000444401  Unknown               Unknown  Unknown
mult.x             000000000043BEFD  rdvars_                   218  rdvars.F
mult.x             000000000042F37F  rdgrdded_                 108  rdgrdded.f
mult.x             000000000040DE50  read3_                    311  read3.F
mult.x             0000000000403727  Unknown               Unknown  Unknown
mult.x             00000000004030E2  Unknown               Unknown  Unknown
libc.so.6          00002B5A6A1C2555  Unknown               Unknown  Unknown
mult.x             0000000000403029  Unknown               Unknown  Unknown
ERROR: Running xportfrac adjustments

Because these are both dust_adj files and they have the same error, I think they’re related, but I’m not sure. Also, I cannot figure out why this error is coming up. The non-adj files complete without any errors. In a previous post about ptegu, the error was that I could not use 2015 – here I am again simply trying to create spin up files for 2015 so that I can get a model run for 2016, so perhaps I’ll have to shift dates again, but I’d like to know if there’s a separate issue I haven’t addressed.

To mitigate this problem, I have been removing intermediate log files before re-running the nonpoint sector. I also tried running 2016-01 to see if it would complete, and I got the same error about xportfrac adjustments.

Any insight would help, thanks

Stacy

2

As an update, I have continued to try to address this problem:

  1. I have checked that ulimit -s is in all run scripts.
  2. I have increased the memory for the scripts up to 64G, no change.
  3. I have deleted all the intermediate files and re-run the sector, no change.

A similar problem was brought up here: 2016 emission platform v1 adjust sectors - #6 by eyth.alison , but the solution was not explicitly stated.

3

What sort of error are you having and what platform are you running?

4

First of all, I note that this is a very old I/O API version – Library release tag: $JDate: 2005333 $
However, this is seg-faulting on a part of the code that has been working correctly (and un-changed) for more than 28 years, so…

Most probably, there is an error allocating the array to be returned by the READ3 call; but what?

You might try re-compiling SMOKE with trace-back and check-everything enabled – say for BIN=Linux2_x86_64dbg or Linux2_x86_64ifortdbg and re-running – this would give line-numbers for the SMOKE file at which the problem happens; then you could trace back the allocation-history of the variable being read by this READ3 call and find out why it is either not allocated, or allocated with the wrong size. Of course, check the dimensions on that input-file to see what that size should be…

5

@cjcoats @asmontgomery Xportfrac tool is not a SMOKE utility tool. So compiling SMOKE won’t help you much to debug. I do agree with @cjcoats that it is a segmentation fault coming from memory… Is this a precompiled executable from the NEI platform package? or did you compile your own?

6

Hello all, and thank you for your help!

I am running the 2016 beta platform so yes it is the precompiled executable from the NEI platform package.

7

Thank you for your help everyone – I have added in IOAPI_OFFSET_64 = “Y” in the runscripts and created a new xportfrac file, which seems to have addressed the error.

8

That only changes the netCDF handling of large arrays (etc.), and not necessarily the same issue for the rest of the program…