RE: SAVE in EOF analysis

[Billy Kessler <kessler@xxxxxxxxxxxxx>]

As several people have pointed out, inverting large
matrices is an extremely cpu-intensive task. More
sophisticated techniques try to find (guess) the 
largest eigenvalues first, avoiding the task of 
finding all 3825 eigenvalues, at least 99% of which 
will be of no interest and probably noise. Perhaps 
someone on the list knows a way to do this in fortran,
in which case it could be incorporated into Ferret.
Another possibility is that such techniques may be 
available in Matlab, which is very good with this 
kind of thing. Much as I value Ferret, there are some
calculations, especially matrix manipulation, for which 
Matlab is inherently better. The routines in the Ferret
EOF implementation are based on very old techniques, and
are definitely not the most efficient. Perhaps the way 
to go would be to do the EOFs in Matlab and just use 
Ferret's superior plotting ability to plot the results. 

A quick and dirty way to improve the situation would
be to decimate the data, say by regridding to a coarser
grid, before finding EOFs on a large region. For example
when Boyin wants EOFs over 120E-70W/40S-50N, there is
probably no need to use the complete 2x2 degree resolution
of the original data to recover the large-spatial-scale
EOFs that are of interest. If you are pretty sure the
same patterns would be seen in a 4x4 grid, then regridding
before doing the EOFs would cut the CPU time down by a
factor of 64 (1/4 the number of grid cells, and the cube
root of the number of iterations to do the inversion).
Even just cutting down the zonal resolution would make
a large difference in the needed CPU.

The reason that the slowdown is seen in the SAVE command
(in Boyin's example) is that the definition command

LET eof_xyfcn = eof_space(sst, 0.5)

does nothing except to specify the definition of the 
variable eof_xyfcn. No actual computation is done until 
this variable is needed (for example a PLOT or SAVE). 
Therefore, doing a SAVE immediately is a very good idea, 
ensuring that the computation is done only once. 

Given that most of the EOFs will be useless, there is
no reason to SAVE all 3825 of them. It will just make 
a large file of mostly junk. In this case, for example:

SAVE/file=eof_xyfcn25.cdf eof_xyfcn[l=1:25]   ! first 25 EOFs

This will not save any CPU time, unfortunately, but it will
make a much smaller file. Then:

CAN VAR eof_xyfcn	! be sure not to compute this again
USE eof_xyfcn25.cdf	! bring in the premade file
FILL eof_xyfcn[l=1]	! or other plots as wanted

Billy K
------------------------------------------------------------
> From: Boyin Huang <huang_boyin@yahoo.com>
> Date: Thu, 25 Sep 2003 13:40:16 -0700 (PDT)
> Subject: SAVE in EOF analysis
> 
> Hi, ferreters:
> 
> Is the ferret so slow in doing EOF analysis?
> 
> Using the demo file for eof analysis:
> 
> -------------------------------------------------------
> SET REGION/X=67w:1w/Y=11S:11N
> USE coads_climatology
> LET eof_xyfcn = eof_space(sst, 0.5)
> SAVE/CLOBBER/FILE=sst_clim_eof_space.cdf eof_xyfcn
> 
> USE sst_clim_eof_space.cdf
> SET VIEW ul; fill/l=1/TITLE="eof 1" eof_xyfcn; go land
> -------------------------------------------------------
> 
> Which works fine to me. Now I wish to have EOF in a
> larger domain, say 120E-70W and 40S-50N.
> 
> I replaced set region by
> SET REGION/X=120e:70w/Y=40S:50N
> and added
> SET MEMORY/SIZE=260.(my computer memory is 1.5gb).
> 
> I found that the ferret runs for ever (how long should
> I wait for?).
> 
> I tried to diagnose the problem by typing these
> command lines interactively line by line, and found
> the problem is that the program was stuck by SAVE
> command.
> 
> I then tried NOT to use the save command, but ferret
> says that the variable was not found.
> 
> So, do we have to save EOF first, and the read and
> plot?
> 
> How can I do the EOF analysis within a larger space
> domain and long time series? Say, simulated global SST
> with 200x100 grids for 1200 month, sst(200,100,1200).
> 
> 
> 
> =====
> Boyin Huang
> The Center for Research on the Changing Earth System (CRCES)
> 10211 Wincopin Circle, Suite 240
> Columbia, Maryland 21044
> Tel. 410-992-5300/819 (O), 410-737-4557(H)
> Fax. 410-992-5944, Email. byh@crces.org
> http://byh.crces.org
> 
> __________________________________
> Do you Yahoo!?
> The New Yahoo! Shopping - with improved product search
> http://shopping.yahoo.com
>