Re: [ferret_users] Fwd: ferret vs pyferret performance time....

[Karl Smith - NOAA Affiliate <karl.smith@xxxxxxxx>]

Hi Sudheer,

One thing I noticed is that you are requesting a LOT of memory.  The value you supply is multiplied by 10^6 * size of the floating-point variable.  So 1200 * 10^6 * 4 = 4.5 Gb for single-precision Ferret (6.67), and 1200 * 10^6 * 8 = 8.9 Gb for double-precision Ferret (6.84) and PyFerret.  Do you really need this much memory dedicated to Ferret?  More typical values used are 100 (0.74 Gb for double-precision) or 200 (1.5 Gb for double-precision).

I suspect this huge allocation of memory to Ferret or PyFerret is causing your machine to spend a lot of time swapping out memory to disk.  The fact the elapsed time (the "real" time value - 25.6 sec and 5.31 min) is much larger that the time actually used by the processor ("user" plus "sys" time values - 17.8 sec and 34.5 sec) would support this, assuming there was not much else running on your machine at the same time.  And if your machine became quite unresponsive in other command windows while these were running, swapping is almost certainly the problem.  PyFerret also has Python and other Python modules loaded in memory, so more memory is needed and there would be even more swapping, and the reason for the much larger elapsed time.  The actual processor time ratio for PyFerret vs Ferret (34.5 sec versus 17.8 sec - about 2:1) is about what is expected with the higher-quality graphics in PyFerret.

If there actually is that much data to be examined, it would be more efficient stay with a smaller amount of memory in Ferret and work with the data in pieces to get your desired result.

Karl

On Thu, Jan 3, 2013 at 5:04 PM, Sudheer Joseph <sjo.india@xxxxxxxxx> wrote:

Dear developers/Users,

Between version 6.67 and 6.84 versions of ferret I get considerable delay when using t=@ngd] is this already noticed by some one or are there any solutions, If I use 6.67 I get the result in much less time compared to 6.84

Similarly 

                       I get a poor performance with pyferret when compared with normal ferret for the same set of operations. for the below script normal ferret takes 25 seconds where as pyferret takes 5 minutes. Why is this difference is this due to configuration issue ? are there anything in pyferret to set for making it faster?  I do not get any message or error while starting pyferret as I have cdtime there with my python.

Please share your experience if any one has faced same and also if there a solution?

sjo@sjo-ltp../NC_FILES~$ time ferret -script m_tmi_percen_g21dg.jnl 

set mem/size=1200

 Cached data cleared from memory

use m_minus_tmi_sst.nc

let cls_1= if abs(m_TMI_SST) gt 1 then m_TMI_SST

let ngd=cls_1[t=@var]

set reg/x=35:115/y=-35:26

......

.......

let perc_gud=(ngd/(ngd+nbd))*100

fill/lev=(1,100,5)/nolab/set perc_gud

ppl fill

contour/over/lev=(1,100,10)/nolab/set perc_gud

ppl contour/over

real 0m25.558s

user 0m15.109s

sys 0m2.716s

************************************************************

$ time pyferret -script hyc_tmi_percen_g21dg.jnl 

set mem/size=1200

 Cached data cleared from memory

use hyc_minus_tmi_sst.nc

let cls_1= if abs(m_TMI_SST) gt 1 then m_TMI_SST

let ngd=cls_1[t=@ngd]

set reg/x=35:115/y=-35:26

......

......

fill/lev=(1,100,5)/nolab/set perc_gud

ppl fill

contour/over/lev=(1,100,10)/nolab/set perc_gud

ppl contour/over

real 5m18.710s

user 0m25.894s

sys 0m8.637s

*************************************************************

--
with best regards

Sudheer

**********************************************************************************
Dr. Sudheer Joseph                                                                
Scientist                                                                         
INDIAN NATIONAL CENTRE FOR OCEAN INFORMATION SERVICES (INCOIS)
MINISTRY OF EARTH SCIENCES, GOVERNMENT OF INDIA
"OCEAN VALLEY" PRAGATHI NAGAR (BO)
OPP.JNTU, NIZAMPET SO
Andhra Pradesh, India. PIN- 500 090.
TEl:+91-40-23044600(R),Tel:+91-9440832534(Mobile)
Tel:+91-40-23886047(O),Fax:+91-40-23892910(O)
E-mail: sudheer.joseph@xxxxxxxxx;  sjo@xxxxxxxxxxxxx.
Web- http://oppamthadathil.tripod.com
           --------------* ---------------
"The ultimate measure of a man is
not where he stands in moments of
comfort and convenience, but where
he stands at times of challenge and
controversy."
                        Martin Luther King, Jr.
"What we have done for ourselves alone dies with us.
What we have done for others and the world remains and is immortal."
- Albert Pines

--
with best regards

Sudheer

**********************************************************************************
Dr. Sudheer Joseph                                                                
Scientist                                                                         
INDIAN NATIONAL CENTRE FOR OCEAN INFORMATION SERVICES (INCOIS)
MINISTRY OF EARTH SCIENCES, GOVERNMENT OF INDIA
"OCEAN VALLEY" PRAGATHI NAGAR (BO)
OPP.JNTU, NIZAMPET SO
Andhra Pradesh, India. PIN- 500 090.
TEl:+91-40-23044600(R),Tel:+91-9440832534(Mobile)
Tel:+91-40-23886047(O),Fax:+91-40-23892910(O)
E-mail: sudheer.joseph@xxxxxxxxx;  sjo@xxxxxxxxxxxxx.
Web- http://oppamthadathil.tripod.com
           --------------* ---------------
"The ultimate measure of a man is
not where he stands in moments of
comfort and convenience, but where
he stands at times of challenge and
controversy."
                        Martin Luther King, Jr.
"What we have done for ourselves alone dies with us.
What we have done for others and the world remains and is immortal."
- Albert Pines

--
Karl M. Smith, Ph.D.
JISAO Univ. Wash. and TMAP/PMEL NOAA
"The contents of this message are mine personally and do
not necessarily reflect any position of the Government
or the National Oceanic and Atmospheric Administration."