Re: Fw: [ferret_users] memory usage creeping up

Hi Jean,

Linux also caches data waiting to be written to disk, and so cannot delete this cached memory until the data is written to disk. All sync does to force Linux to write to disk all the cached data waiting to be written to disk. I have never heard of drop_caches; I suspect it is a custom script written by your IT group. Normally none of this is required since Linux is designed to automatically sync when it is not busy with other things, or when it needs the cache memory. And as I mentioned, normally Linux automatically clears out cached memory when processes need memory.

All those save /append commands will require Linux to read each of those files and then write them back out with the added data. Since Ferret is doing this rather rapidly, I can imagine that it could keep Linux quite busy trying to keep up, especially if disk access not very fast (such as with NFS-mounted disks on a busy network). One thing you could do is add the Ferret command "spawn sync" (which just calls the Linux sync command) inside your loop at the end. This will give Linux a chance to catch up and make sure the new data is written to disk. The loops will go slower because the sync command will not return until all write-cached data on the system is written to disk. But this should keep the write-cached data to a minimum. I would be very reluctant to use drop_caches since I assume it removes all cached data, which normally should never be required.

Karl

On Tue, Jul 9, 2013 at 1:17 PM, jean li <xgeast@xxxxxxxxx> wrote:

Hi Karl:

We use "sync" and "drop_caches" to clean up the old cache.

Jean

----- Forwarded Message -----
From: jean li <xgeast@xxxxxxxxx>
To: Karl Smith - NOAA Affiliate <karl.smith@xxxxxxxx>
Sent: Tuesday, July 9, 2013 3:11 PM
Subject: Re: [ferret_users] memory usage creeping up

Hi Karl:

Thanks for your quick reply. Sorry that I haven't explained my problem clearly. I don't have any errors related to memory usage.

I think you are right. The problem has something to do with that the Linux cannot delete the cached data in the memory. And we have to use "sync" to free up all the memory. This problem gets so bad that we have to use a crontab script and run this script on a daily basis.

However, I also run other programs on these machines, e.g. matlab and Fortran, which use a lot of memory as well. But I haven't noticed this problem when running Matlab or Fortran. Matlab will crash the system if there is not enough memory.

FYI. Below is my ferret script that causes the memory to creep up.
-----------------
repeat/range=904:987/name=m (use "/home/liz/Arctic_2012/arctic_ll_`m`.nc";\
save/append/file="/home/liz/Arctic_2012/wind_1996_temp.nc" uwind_ll, vwind_ll;\
save/append/file="/home/liz/Arctic_2012/current_1996_temp.nc" u_ll, v_ll;\
save/append/file="/home/liz/Arctic_2012/ice_1996_temp.nc" uice_ll, vice_ll, aice_ll;\
can data "/home/liz/Arctic_2012/arctic_ll_`m`.nc";\
can data/all;\
can memory)

The wind*.nc etc files are at 2-3 GB. Each input file is at 100MB. After I run this script a few times, the memory is all used up.

Thanks again for your help.

Jean

From: Karl Smith - NOAA Affiliate <karl.smith@xxxxxxxx>
To: jean li <xgeast@xxxxxxxxx>
Cc: "ferret_users@xxxxxxxx" <ferret_users@xxxxxxxx>
Sent: Tuesday, July 9, 2013 1:27 PM
Subject: Re: [ferret_users] memory usage creeping up

Hi Jean,

Once Ferret quits, it is not possible for the program to keep memory reserved. You can use the Linux command "ps -fu <username>" (where <username> is you username on the machine) to display all processes you are running if you want to verify Ferret is no longer around. If there is still a ferret process running when you think you have exited all of them, you can use the Linux kill command to kill the runaway process (using the PID given in the ps command).

Linux (like most operating systems) does cache data read from disk, so what you may be noticing is the data from your data sets that you use being held in cache memory. However, if Linux needs more memory for some process, it will delete the oldest cached data and use that memory. So you will never get an out-of-memory error message just because a large amount of your memory happens to be used for caching data. You did not indicate whether you were getting an errors because of the memory usage.

Another thing to note with Ferret is the cancel data and cancel memory Ferret commands will not return memory to the operating system (one exception: when using string arrays - see below). Only changing the size given with the set memory Ferret command will do that.

If you are using string arrays, the memory for the strings are allocated separately from system memory, so cancelling string arrays will return memory to the system. But use of string arrays is very rare.

Regards,
Karl

On Tue, Jul 9, 2013 at 8:35 AM, jean li <xgeast@xxxxxxxxx> wrote:

Hi All:

I find that the memory usage creeping up after using Ferret. Even if I quit Ferret session, the memory couldn't be released. In my ferret code, I use
cancel data and cancel memory but that doesn't seem to help. I could process the data with "set mem/size=1500", but after a few runs, it filled up the memory of a 65G Linux cluster. The same problem occurred on my Linux (Dell) workstation as well. I am using ferret v6.64 on a 64 bit red hat Linux.

Thanks for your help.

Jean

--
Karl M. Smith, Ph.D.
JISAO Univ. Wash. and TMAP/PMEL NOAA
"The contents of this message are mine personally and do
not necessarily reflect any position of the Government
or the National Oceanic and Atmospheric Administration."

--
Karl M. Smith, Ph.D.
JISAO Univ. Wash. and TMAP/PMEL NOAA
"The contents of this message are mine personally and do
not necessarily reflect any position of the Government
or the National Oceanic and Atmospheric Administration."