Re: [ferret_users] Set memory command not working

[saurabh rathore <rohitsrb2020@xxxxxxxxx>]

Thanks you all for your quick reply and best usage of memory management. I was unable to figure it out why the set mem/size=nnn command was not working as I was working with pyferret version 7.1. So I did the whole reinstallation from python to pyferret. Now its working fine. One thing I noted is that I was having a huge 4 dimensional data set of multiple files loaded and saving at once so it might created some problem with the memory allocation in processing the data.

But after the reinstallation every thing is working fine and I will break my script into separate saving files so in future it wont happen.

regards, saurabh  

On Wed, Jun 7, 2017 at 4:12 AM, Karl Smith - NOAA Affiliate <karl.smith@xxxxxxxx> wrote:

For Ferret and PyFerret, the SET MEM /SIZE=... command will immediate allocate that amount of system memory to Ferret or PyFerret.  (The updated memory management that will be in the next release will only allocate memory as needed for each variable, so this should be less of an issue in the future.)

Because of this, if you ask Ferret or PyFerret for most (or even more) of the memory on your system, your system may start 'swapping', where it starts using your hard drive to temporary store some of that data that should be in memory.  Swapping greatly slows down your system, or at least any process - which might be Ferret or PyFerret - that needs the memory temporarily swapped to disk.  So only request a huge amount of memory in Ferret/PyFerret if you really need that much memory to do what you need to do.

Note that SET MEM /SIZE=200 allocates 200 * 8 * 1M = 1.6G of memory, so don't put that command in your $HOME/.ferret start-up file unless you have many gigabytes of memory on your system and regularly work with datasets of this size.

Karl

On Tue, Jun 6, 2017 at 9:48 AM, Ansley C. Manke <ansley.b.manke@xxxxxxxx> wrote:

Hi,

What version of PyFerret are you running?

Memory allocation does not affect processing speed, but only the size of grids that can be loaded into memory.  PyFerret and Ferret automatically do some operations, particularly transformations, in "chunks", so that you don't need to allocate larger amounts for those tasks.  It also automatically saves data in memory cache for efficiency, so as not to have to re-read data all the time, but will re-use memory space as needed. The next release has some updates to memory management and does this chunking

What response do you see when you try set memory/size=nnn?  If it is successful, Ferret will say, for instance,

yes? set mem/siz=200
 Cached data cleared from memory
 yes? show memory
 Current size of FERRET memory cache: 200 MegaWords  (1 word = 8 bytes)

If it is unable to allocate the amount of memory, it will say so,

yes?  set mem/siz=200000
 Cached data cleared from memory
Unable to allocate 200000.0 Mwords of memory.
Restoring previous memory of 200.0 Mwords.

You can also issue the "CANCEL MEMORY" command which removes data that is stored in memory but does not affect the amount of memory allocated. 

Ansley

On 6/6/2017 3:12 AM, saurabh rathore wrote:

Dear ferreters

I am facing a problem that in pyferret set memory/size=nnn not working for me. 

It is unable to clear the memory. So the processing speed is too low. It suddenly stop working in latest version of pyferret for me. I did the reinstallation also but it is still not working. 

Any help for this ?

Regards Saurabh

--

REGARDS

Saurabh Rathore

Research Scholar (PhD.)

Centre For Oceans, Rivers, Atmosphere & Land Science Technology

Indian Institute Of Technology, Kharagpur

contact :- 91- 8345984434

--

Karl M. Smith, Ph.D.
JISAO Univ. Wash. and PMEL NOAA
"The contents of this message are mine personally and do
not necessarily reflect any position of the Government
or the National Oceanic and Atmospheric Administration."

--

REGARDS

Saurabh Rathore

Research Scholar (PhD.)

Centre For Oceans, Rivers, Atmosphere & Land Science Technology

Indian Institute Of Technology, Kharagpur

contact :- 91- 8345984434