[ferret_users] memory problems in ferret 7.20 (pyferret)

[Martin Schmidt <martin.schmidt@xxxxxxxxxxxxxxxxx>]

Hi,

I find a strange behaviour with pyferret 7.20

<phy-8:mschmidt>ferret
        NOAA/PMEL TMAP
        PyFerret v7.2 (optimized)
        Linux 4.4.85-22-default - 09/19/17
        19-Sep-17 23:27

the binary is build on openSuse 42.3. netcdf/hdf/szip are build 
separately, all other ingredients come from the system. The compiler is 
gcc-5, python is 2.7.13.

The following example illustrates the problem. Averaging and integrating 
a bigger dataset. Data are model results kept on a public THREDDS, so it 
should be reproducable. I leave things as they are without streamlining. 
Don't ask for the meaning ...

use 
http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/genus_run_88_5day_ocean.nc
set region/t=1-jan-2010:31-dec-2010/x=13.5:14.7/y=-23.5:-22.5

let n2_prod_anamm = P_DET_ANAM_N2 * 13.25
let n2_prod_denit = P_DET_DENIT_NH4 * 5.3/2
let n2_prod_h2s   = P_H2S_OXNO3_SUL * 0.2
let n2_prod_sul   = P_SUL_OXNO3_SO4*3/5
let n2_prod = (n2_prod_anamm + n2_prod_denit + n2_prod_h2s + n2_prod_sul)

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]

yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
             VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + 
N2_PROD_SUL)
             DATA SET : MOM4 Baltic Sea 3 n.m.
             FILENAME : genus_run_88_5day_ocean.nc
             FILEPATH : 
http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/
             LONGITUDE: 13.5E to 14.7E (XY integ.)
             LATITUDE : 23.5S to 22.5S (XY integ.)
             DEPTH (m): -0.0009867 to 5700 (ZT summed)
             TIME     : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT 
summed) JULIAN
          1.263E+09

Now increase the region, the result should be growing significantly

set region/t=1-jan-2010:31-dec-2010/x=13.5:14.7/y=-28:-22.5 ! Note y!

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]

yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
             VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + 
N2_PROD_SUL)
             DATA SET : MOM4 Baltic Sea 3 n.m.
             FILENAME : genus_run_88_5day_ocean.nc
             FILEPATH : 
http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/
             LONGITUDE: 13.5E to 14.7E (XY integ.)
             LATITUDE : 28S to 22.5S (XY integ.)
             DEPTH (m): -0.0009867 to 5700 (ZT summed)
             TIME     : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT 
summed) JULIAN
          8.011E+08

This is strange, since the quantity is positive everywhere. Check with 
stats (not shown).

Now play with the memory size:

set memory/size=30 ---> 3.244E+08

set memory/size=50 ---> 3.244E+08 ! Is this stable? No!

can memory

set memory/size=50 ---> 8.262E+08

can memory

set memory/size=300 ---> 4.973E+09

I think, this is enough to show, that something does not work correctly.

The classical ferret shows similar results. Aggregating withouth THREDDS 
gives 4.973E+09 for the larger region without increasing the memory size!

Now a previous ferret version:

ferret
        NOAA/PMEL TMAP
        PyFERRET v7 (opt)
        Linux 2.6.32-642.1.1.el6.x86_64 - 07/01/16
        20-Sep-17 00:02

CAN MODE LOGO
SET MEM /SIZE=25.6
 Cached data cleared from memory
SET MODE VERIFY
yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
             VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + 
N2_PROD_SUL)
             DATA SET : MOM4 Baltic Sea 3 n.m.
             FILENAME : genus_run_88_5day_ocean.nc
             FILEPATH : 
http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/
             LONGITUDE: 13.5E to 14.7E (XY integ.)
             LATITUDE : 23.5S to 22.5S (XY integ.)
             DEPTH (m): -0.0009867 to 5700 (ZT summed)
             TIME     : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT 
summed) JULIAN
          1.232E+09

Similar, but not the same as before! Make the region bigger now.

set region/t=1-jan-2010:31-dec-2010/x=13.5:14.7/y=-28:-22.5

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]

yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
             VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + 
N2_PROD_SUL)
             DATA SET : MOM4 Baltic Sea 3 n.m.
             FILENAME : genus_run_88_5day_ocean.nc
             FILEPATH : 
http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/
             LONGITUDE: 13.5E to 14.7E (XY integ.)
             LATITUDE : 28S to 22.5S (XY integ.)
             DEPTH (m): -0.0009867 to 5700 (ZT summed)
             TIME     : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT 
summed) JULIAN
          4.760E+09

Not in spectrum of the previous results!

Now with another elder ferret:

ferret
        NOAA/PMEL TMAP
        FERRET v6.96
        Linux 2.6.18-406.el5 64-bit - 12/02/15
        20-Sep-17 00:09

CAN MODE LOGO
SET MEM /SIZE=25.6
 Cached data cleared from memory
SET MODE VERIFY

use 
"http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/genus_run_88_5day_ocean.nc";
....

list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
             VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + 
N2_PROD_SUL)
             DATA SET : MOM4 Baltic Sea 3 n.m.
             FILENAME : genus_run_88_5day_ocean.nc
             FILEPATH : 
http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/
             LONGITUDE: 13.5E to 14.7E (XY integ.)
             LATITUDE : 23.5S to 22.5S (XY integ.)
             DEPTH (m): -0.0009867 to 5700 (ZT summed)
             TIME     : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT 
summed) JULIAN
          1.232E+09

!The same as with 7.0

set region/t=1-jan-2010:31-dec-2010/x=13.5:14.7/y=-28:-22.5

yes? list n2_prod[x=@din,y=@din,k=@sum,t=@sum]
             VARIABLE : (N2_PROD_ANAMM + N2_PROD_DENIT + N2_PROD_H2S + 
N2_PROD_SUL)
             DATA SET : MOM4 Baltic Sea 3 n.m.
             FILENAME : genus_run_88_5day_ocean.nc
             FILEPATH : 
http://phy-50.io-warnemuende.de:8080/thredds/dodsC/genus/
             LONGITUDE: 13.5E to 14.7E (XY integ.)
             LATITUDE : 28S to 22.5S (XY integ.)
             DEPTH (m): -0.0009867 to 5700 (ZT summed)
             TIME     : 01-JAN-2010 00:00 to 31-DEC-2010 00:00 (ZT 
summed) JULIAN
          4.759E+09

Again another result - with higher precision the difference does not 
become small.

Is it possible to reproduce this finding, or is this specific for my 
build? Any help is very welcome!

Greetings,

Martin