Re: increasing performance with list/repeat indices

[Russell Fiedler <fiedler@xxxxxxxxxxxxxxx>]

Brent,

I think that this a problem with strided netcdf IO. You are attempting to
read/write out values every 144 locations in your files. The most
efficient method of access is via contiguous blocks. This means that
looping over the time axis should give the best performance (144x71x17
size blocks). Looping over depth will give ok performance (144x71) and
striding over x will give the worst.

I created the following test program to illustrate this

def axis/x=1:144:1 xax
def axis/y=1:71:1 yax
def axis/z=1:17:1 zax
def axis/t=1:365:1 tax
let x1=x[gx=xax]+y[gy=yax]+z[gz=zax]+t[t=tax]
sp rm f1.nc
! fastest "l"
repeat/l=1:365 save/file=f1.nc/llimits=1:365/append x1
sp rm f1.nc

! ok "k"
repeat/k=1:17 save/file=f1.nc/klimits=1:17/append x1

! you might want to kill this one... "i"
sp rm f1.nc
repeat/i=1:144 save/file=f1.nc/ilimits=1:144/append x1 


It might be even faster to 'block' the loop over l so that it reads

repeat/l=1:361:5 save/file=f1.nc/llimits=1:365/append/l=`l:`l+4` x1

On the other hand I notice that you are using the northern hemisphere so
the k loop may be the just as  efficient as the l loop for input at least
since these blocks are each 144x37

Cheers,
Russ Fiedler

On Mon, 12 Mar 2001, Brent A McDaniel wrote:

> 
> Greetings Ferret Users,
> 
> After discussing this with Ansley he asked that I post it to the Users
> Group to see if someone else has experience with this.
> 
> Background: I calculated the climatology for 1957-1999 of air temperature
> using ferret.  The climatology is 144x73x17x365.  I then smoothed it using
> a 91 point window.  I want to subtract the climatology out from the
> original files to get the anomalies. (the original files are
> 144x73x17x365(or6) as well).  I'm also only saving the northern
> hemisphere, hence the j=36:73.
> 
> Here's a sample of what I'm doing:
> 
> yes? use airclimsmooth91.nc
> yes? use air.1987.nc
> yes? let airanom=air[d=2]-dailyclim[d=1,gt=GSJ1@asn]
> repeat/i=1:144 save/ilimits=1:144/file="/data/ncep/aanom.1957.nc"/append
> airanom[j=36:73,l=1:365]
> yes? quit
> 
> 
> !where GSJ1 is the grid name of air.1987.nc
> 
> 
> This works but takes quite a long time (approx 8 hours per 1 annual data
> file).  I'm now thinking that repeating over the z-axis might save some time as it
> would decrease the number of times information was written to disk from
> 144 to 17 per file although I haven't tried it (it might not work due to
> memory issues which is why I'm using the repeat command in the first
> place).
> 
> Does anyone else have experience with this or know of a more efficient
> solution?
> 
> 
> Cheers,
> 
> Brent
> 
> 
> 
>