[Thread Prev][Thread Next][Index]

Re: [ferret_users] memory problems in ferret 7.20 (pyferret)

To: Martin Schmidt <martin.schmidt@xxxxxxxxxxxxxxxxx>
Subject: Re: [ferret_users] memory problems in ferret 7.20 (pyferret)
From: Karl Smith - NOAA Affiliate <karl.smith@xxxxxxxx>
Date: Wed, 20 Sep 2017 12:14:56 -0700
Arc-authentication-results: i=2; mx.google.com; dkim=pass header.i=@noaa-gov.20150623.gappssmtp.com header.s=20150623 header.b=EjICi6kj; spf=pass (google.com: domain of karl.smith@xxxxxxxx designates 209.85.220.41 as permitted sender) smtp.mailfrom=karl.smith@xxxxxxxx
Arc-authentication-results: i=1; mx.google.com; dkim=pass header.i=@noaa-gov.20150623.gappssmtp.com header.s=20150623 header.b=EjICi6kj; spf=pass (google.com: domain of karl.smith@xxxxxxxx designates 209.85.220.41 as permitted sender) smtp.mailfrom=karl.smith@xxxxxxxx
Arc-message-signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-archive:list-help:list-post:list-id:mailing-list:precedence:cc :to:subject:message-id:date:from:references:in-reply-to:mime-version :arc-authentication-results:arc-message-signature:dkim-signature :arc-authentication-results; bh=+rHwvQgLYeyR7eoQCLYst0hyG6yRbSeCuDAQ3RmyRq4=; b=ag/J1FM2DvzfjDSIAfbF30upU4Gbr5Nk1XjbU1oSYXE02c74H/IjdyiEI/jwJgLVdt nwil0eonqeaeLNsl+0zoRkR2kxPTdTLj/mi8wz/116J7FgqUXxeTJQMRKp29g2pzGQJm LFo9HcQWd3MWaGOpGMWVkB38bADw3adoTvwVoBhkJeAbP1RWM7OTv+hCSoXWHKN3UNWU QDEcM2iyeq75MtojDkD3O6opnPXTRh60ypaM+rtyFu1kAmPEVrxrKPf/jOg3SK8bCULv IDR0xSIjeIeTp/DKqNr7Balhs+bYptoYIkavzVAPYqq2akNG5rJLZNB/A0kBHkZS2q8V A2FQ==
Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:dkim-signature:arc-authentication-results; bh=+rHwvQgLYeyR7eoQCLYst0hyG6yRbSeCuDAQ3RmyRq4=; b=zr6Rqt6xfeEA2Aq/SxqVL7FY4cBFblsSLeJCT5bhrkGU97wuC/J6V5u+0GRHmRv8lY TOiBqYVVBwwnT4y5OftTtzFKReT1TPr3ioRFSxd6QmNl3BFg4qmztNYyvEQgrEftdRei dpogCJGST6PxuO4zFr/fPVg/dzyi99k4gXFL0CeSpGGnEkqSNIbTVBzHFydwUOwbBsVb x4cSyJT2Fg3X6OFXj4qIpgqAs51H71goUU+ZlU04Xhgv7fxe+gCIPQXdpkWaVEvFv0Lz qOfzHFe+yD6b47hRNOpOzjQBR3CXxnsCt0IvbqeAS2qmUo/fbJv9fuD3jXL7QgnGzxNi SRkg==
Arc-seal: i=2; a=rsa-sha256; t=1505934898; cv=pass; d=google.com; s=arc-20160816; b=rmDs7nSZ0fwLLqYNLjjduZ2bDFjKuz4gCoYYBWXd5XHYGqevs0lQY6/ZsgpTJI2p9b KKXQSGKj+9qHaLfcG4AB7UJzPRrBYWHc6AU/09xaoGxxFfdnqtxCATWzwRTlm41doWex NqwQJIx6VzfXNHnPqCAvLv0H7ElcNBBtEeIOS7Jdlq4hWf1P2IeHWlOVCLbou+ZQ4kDq ViObuSi5xP72q7aO1xUL1WSMNeEMfihRsWWQ7H0kWjgJzdve7dEY5V2p9S/xwTUkriQc jSoB3Zq1/d93Rt3pMiPj+Xy4+yOWYO/5VenBzQYXPEjWiGx7kk98ONZ5kRi9kwUaHnRB pw7g==
Arc-seal: i=1; a=rsa-sha256; t=1505934897; cv=none; d=google.com; s=arc-20160816; b=Epf1Zb+0YXdc7P5v8mEcqxW/ZP7bKwkwBMCXiX8/jOygRLF8twV4l8SXZmRWI3WjWS i/M7SVmM//PDSC+B0cuEqQvmegJMwy8vrfBnyCn6v4JKfVAacbpKtr+WkCQ5f7AhqlHp q1SWBHIot5F3SRiFcYs2DEDFYOys10U5YLvKbEcqcY5gDoG5v1773oeJSXwsIDfPtnJh 9BfSpcOCmg6JAwhJJ8jVrGomI4A6MKqzvie4tGGlqBet8HzDSQbdQlm4XP00y4NGzwmM tE1os0B3ufKYL6deRLp7wWoTGimT8MGLZlAH3oStoXlNitycyHK66MfYLiowADogIB8c 81aw==
Cc: ferret <ferret_users@xxxxxxxx>
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=noaa-gov.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:x-original-sender:x-original-authentication-results:precedence :mailing-list:list-id:list-post:list-help:list-archive; bh=+rHwvQgLYeyR7eoQCLYst0hyG6yRbSeCuDAQ3RmyRq4=; b=oLGncj40J5x7jPxPHxaoj7crGMWPNlYP98wZVFGFmdxMtIC/BKBYjg2KugV8/2Xhbv xVMjiJRzFaewRADCl0I9AHcQicKmIA+qSz9vHv7KaXZZK4AhmS23aws4c9HY4t8QzoTm BDGu/DBCMjZnetGgTIMWtKFoEgviFz+hiGBvFOqhstYgW8bno2RBIlROQwPz7InNbu8s UZDKPjRDZrkBANGMP3d4nl13EMZ4lfFyvQli9L4H271EAq9J6+FEJqvGCFSu45FlB58Y ZuXSqxcKoPVqKYS5/e5DzoVw8vEEw8VJReO47+qL/xaru2gSPSZ1w7Qg+ULRjwyWmnfn JUsw==
In-reply-to: <85011cbf-739d-c69a-5d66-a72611e3aef2@io-warnemuende.de>
List-archive: <https://groups.google.com/a/noaa.gov/group/ferret_users/>
List-help: <https://support.google.com/a/noaa.gov/bin/topic.py?topic=25838>, <mailto:ferret_users+help@noaa.gov>
List-id: <ferret_users.noaa.gov>
List-post: <https://groups.google.com/a/noaa.gov/group/ferret_users/post>, <mailto:ferret_users@noaa.gov>
Mailing-list: list ferret_users@xxxxxxxx; contact ferret_users+owners@xxxxxxxx
References: <CAORxmhuDzG858S8HUnNzfEVMskemtzxX7BgQiPe-bunN2zR+ag@mail.gmail.com> <85011cbf-739d-c69a-5d66-a72611e3aef2@io-warnemuende.de>
Sender: owner-ferret_users@xxxxxxxx

Hi Martin,

Yes, I am able to reproduce your result (using 7.2.1 just to make sure the problem was not associated with the fixes from 7.2.0 to 7.2.1), as well as the older results using ferret 6.9.7 and 7.1.0.

I suspect the problem is the handling of calculations that exceed the allocated memory size. In 7.2.0 the ability to split the calculation over a transformed axis was added, and I suspect this might be the source of this problem.

To show this a little more clearly, I modified your script to play with the memory sizes for each region. I also added a "can mem /all" to make sure ferret was always re-reading data from the source. Using Ferret 7.2.1, for the smaller region I got the value 1.263E+09 for all memory sizes. For the larger region, the values were 3.244E+08 (for 30), 8.262E+08 (for 50), 4.973E+09 (for 300), and 4.973E+09 (for 500). Using Ferret 6.9.7 or 7.1.0 with the smaller region I got the value 1.232E+09 for all memory sizes and with the larger region I got the value 4.760E+09 for all memory sizes.

I do notice that your lon, lat, and depth axes are irregularly spaced. This might be pertinent to this problem, but I am not sure at this time. Obviously this is a high priority issue in 7.2 to get fixed.

Best regards,

Karl

===

My script:

(definitions as before)

set region/t=1-jan-2010:31-dec-2010/x=13.5:14.7/y=-23.5:-22.5

set memory/size=30

can memory/all

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]

set memory/size=50

can memory/all

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]

set memory/size=300

can memory/all

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]

set memory/size=500

can memory/all

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]

! Increase in y

set region/t=1-jan-2010:31-dec-2010/x=13.5:14.7/y=-28:-22.5

set memory/size=30

can memory/all

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]

set memory/size=50

can memory/all

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]

set memory/size=300

can memory/all

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]

set memory/size=500

can memory/all

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]

On Tue, Sep 19, 2017 at 3:33 PM, Martin Schmidt <martin.schmidt@xxxxxxxxxxxxxxxxx> wrote:

Hi,

I find a strange behaviour with pyferret 7.20

<phy-8:mschmidt>ferret
        NOAA/PMEL TMAP
        PyFerret v7.2 (optimized)
        Linux 4.4.85-22-default - 09/19/17
        19-Sep-17 23:27

the binary is build on openSuse 42.3. netcdf/hdf/szip are build separately, all other ingredients come from the system. The compiler is gcc-5, python is 2.7.13.

The following example illustrates the problem. Averaging and integrating a bigger dataset. Data are model results kept on a public THREDDS, so it should be reproducable. I leave things as they are without streamlining. Don't ask for the meaning ...

use http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/genus_run_88_5day_ocean.nc
set region/t=1-jan-2010:31-dec-2010/x=13.5:14.7/y=-23.5:-22.5

let n2_prod_anamm = P_DET_ANAM_N2 * 13.25
let n2_prod_denit = P_DET_DENIT_NH4 * 5.3/2
let n2_prod_h2s   = P_H2S_OXNO3_SUL * 0.2
let n2_prod_sul   = P_SUL_OXNO3_SO4*3/5
let n2_prod = (n2_prod_anamm + n2_prod_denit + n2_prod_h2s + n2_prod_sul)

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]

yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
             VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + N2_PROD_SUL)
             DATA SET : MOM4 Baltic Sea 3 n.m.
             FILENAME : genus_run_88_5day_ocean.nc
             FILEPATH : http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/
             LONGITUDE: 13.5E to 14.7E (XY integ.)
             LATITUDE : 23.5S to 22.5S (XY integ.)
             DEPTH (m): -0.0009867 to 5700 (ZT summed)
             TIME     : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT summed) JULIAN
          1.263E+09

Now increase the region, the result should be growing significantly

set region/t=1-jan-2010:31-dec-2010/x=13.5:14.7/y=-28:-22.5 ! Note y!

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]

yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
             VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + N2_PROD_SUL)
             DATA SET : MOM4 Baltic Sea 3 n.m.
             FILENAME : genus_run_88_5day_ocean.nc
             FILEPATH : http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/
             LONGITUDE: 13.5E to 14.7E (XY integ.)
             LATITUDE : 28S to 22.5S (XY integ.)
             DEPTH (m): -0.0009867 to 5700 (ZT summed)
             TIME     : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT summed) JULIAN
          8.011E+08

This is strange, since the quantity is positive everywhere. Check with stats (not shown).

Now play with the memory size:

set memory/size=30 ---> 3.244E+08

set memory/size=50 ---> 3.244E+08 ! Is this stable? No!

can memory

set memory/size=50 ---> 8.262E+08

can memory

set memory/size=300 ---> 4.973E+09

I think, this is enough to show, that something does not work correctly.

The classical ferret shows similar results. Aggregating withouth THREDDS gives 4.973E+09 for the larger region without increasing the memory size!

Now a previous ferret version:

ferret
        NOAA/PMEL TMAP
        PyFERRET v7 (opt)
        Linux 2.6.32-642.1.1.el6.x86_64 - 07/01/16
        20-Sep-17 00:02

CAN MODE LOGO
SET MEM /SIZE=25.6
Cached data cleared from memory
SET MODE VERIFY
yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
             VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + N2_PROD_SUL)
             DATA SET : MOM4 Baltic Sea 3 n.m.
             FILENAME : genus_run_88_5day_ocean.nc
             FILEPATH : http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/
             LONGITUDE: 13.5E to 14.7E (XY integ.)
             LATITUDE : 23.5S to 22.5S (XY integ.)
             DEPTH (m): -0.0009867 to 5700 (ZT summed)
             TIME     : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT summed) JULIAN
          1.232E+09

Similar, but not the same as before! Make the region bigger now.

set region/t=1-jan-2010:31-dec-2010/x=13.5:14.7/y=-28:-22.5

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]

yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
             VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + N2_PROD_SUL)
             DATA SET : MOM4 Baltic Sea 3 n.m.
             FILENAME : genus_run_88_5day_ocean.nc
             FILEPATH : http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/
             LONGITUDE: 13.5E to 14.7E (XY integ.)
             LATITUDE : 28S to 22.5S (XY integ.)
             DEPTH (m): -0.0009867 to 5700 (ZT summed)
             TIME     : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT summed) JULIAN
          4.760E+09

Not in spectrum of the previous results!

Now with another elder ferret:

ferret
        NOAA/PMEL TMAP
        FERRET v6.96
        Linux 2.6.18-406.el5 64-bit - 12/02/15
        20-Sep-17 00:09

CAN MODE LOGO
SET MEM /SIZE=25.6
Cached data cleared from memory
SET MODE VERIFY

use "http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/genus_run_88_5day_ocean.nc"
....

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
             VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + N2_PROD_SUL)
             DATA SET : MOM4 Baltic Sea 3 n.m.
             FILENAME : genus_run_88_5day_ocean.nc
             FILEPATH : http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/
             LONGITUDE: 13.5E to 14.7E (XY integ.)
             LATITUDE : 23.5S to 22.5S (XY integ.)
             DEPTH (m): -0.0009867 to 5700 (ZT summed)
             TIME     : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT summed) JULIAN
          1.232E+09

!The same as with 7.0

set region/t=1-jan-2010:31-dec-2010/x=13.5:14.7/y=-28:-22.5

yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
             VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + N2_PROD_SUL)
             DATA SET : MOM4 Baltic Sea 3 n.m.
             FILENAME : genus_run_88_5day_ocean.nc
             FILEPATH : http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/
             LONGITUDE: 13.5E to 14.7E (XY integ.)
             LATITUDE : 28S to 22.5S (XY integ.)
             DEPTH (m): -0.0009867 to 5700 (ZT summed)
             TIME     : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT summed) JULIAN
          4.759E+09

Again another result - with higher precision the difference does not become small.

Is it possible to reproduce this finding, or is this specific for my build? Any help is very welcome!

Greetings,

Martin

--

Karl M. Smith, Ph.D.
JISAO Univ. Wash. and PMEL NOAA
"The contents of this message are mine personally and do
not necessarily reflect any position of the Government
or the National Oceanic and Atmospheric Administration."

References:
- [ferret_users] error when saving a .ps figure
  - From: karen guihou
- [ferret_users] memory problems in ferret 7.20 (pyferret)
  - From: Martin Schmidt

Previous by thread: [ferret_users] memory problems in ferret 7.20 (pyferret)
Next by thread: [ferret_users] pH calculation from co2 in water

[Thread Prev][Thread Next][Index] Contact Us

Dept of Commerce / NOAA / OAR / PMEL / Ferret

Privacy Policy | Disclaimer | Accessibility Statement