[Thread Prev][Thread Next][Index]

Re: [ferret_users] MEMORY/SIZE= changes the result of SAVE



Hi,

We have tracked down the cause of this bug. We appreciate the report, as it appears only in some circumstances. The fix will in the next Ferret/PyFerret release.

This was a case of a bug introduced a couple of years ago, but which shows up only when there's the right combination of the size of the grid and the memory setting and Ferret/PyFerret's estimate of what it needs to do in allocating resources as it evaluates the _expression_.  Reports of odd behavior like this are crucial as our testing, though it is extensive, can certainly miss things!

Ansley


On 12/17/2019 10:07 AM, Ansley C. Manke wrote:

Hi,

I suspect this may have to do with Ferret/PyFerret breaking up the computation when computing a number of transformations including @AVE - that's summarized in point 2, here:

https://ferret.pmel.noaa.gov/Ferret/documentation/release_notes/version-7-2-release-notes

This behavior where the results differ does show up for the first time in v7.2. 

Breaking up the computation is done according to the size of the data to be worked on and the available memory setting.  That would explain why changing the size of the computation changes the behavior.  The verbose output of "SET MODE DIAGNOSTIC" lists the operations called scattering and gathering. And a summary is given with "SHOW MEMORY/DIAGNOSTIC".  I'll work with your example and see if I can tell what's going on.

The core analysis code is identical with Ferret and Pyferret, so we wouldn't expect differences there.

-Ansley


On 12/17/2019 4:52 AM, Ryo Furue wrote:
Hi, again,

Just one more experiment.

On Mon, Dec 16, 2019 at 1:31 PM Ryo Furue <furue@xxxxxxxxxx> wrote:
Unfortunately, I've just reproduced exactly the same problem on Ferret 7.5.0 on Linux.  So far, we haven't got Pyferret on our Linux server.

So, the same problem is reproduced with PyFerret 7.5 on the same Linux machine:

On PyFerret 7.5 on my Mac, exactly the same problem occurs with the same dataset.  So, it's more likely that other people can reproduce the problem.

Regards,
Ryo

 
The presence or absence of a SET MEMORY command changes the result of SAVE without any diagnostics.

I'll try to further reduce my data file.

Regards,
Ryo
-----------
  NOAA/PMEL TMAP
  PyFerret v7.5 (optimized)
  Linux 4.15.0-1061-azure - 11/14/19
  16-Dec-19 13:21

====================
  NOAA/PMEL TMAP
  FERRET v7.5 (optimized)
  Linux 2.6.32-754.11.1.el6.x86_64 64-bit - 04/25/19
  16-Dec-19 00:02
 
!=== my script to reproduce the bug ===
! Delete the SET MEMORY command and the two output variables become identical.
!
set data tmp-usss0500m-L600.nc ! Contains var "usss" .

set memory/size=300 ! <- Delete this line
let u = usss[d=1]
save/clobber/file=tmp-u.nc  u[m=1:10@AVE]
save/clobber/file=tmp-usss.nc  usss[m=1:10@AVE]

!--- plot to see if they are identical ---
cancel var/all
cancel data/all
set data tmp-u.nc
set data tmp-usss.nc
plot/x=101W/y=4S:4N u[d=1], usss[d=2]


[Thread Prev][Thread Next][Index]
Contact Us
Dept of Commerce / NOAA / OAR / PMEL / Ferret

Privacy Policy | Disclaimer | Accessibility Statement