RE: SAVE in EOF analysis

[Beckers Jean-Marie <JM.Beckers@xxxxxxxxx>]

>

You might search for the paper of Tomazou in Monthly Weather Review.
He shows how using a Lanczos iterative method (Krylov subspace) allows to
extract the leading eigenvalues (and EOFs) using a Fortran ARPACK library.
Best regards

(I'm on the road and do not have access to my data base but can send you the
exact reference when I'm back)

> As several people have pointed out, inverting large
> matrices is an extremely cpu-intensive task. More
> sophisticated techniques try to find (guess) the
> largest eigenvalues first, avoiding the task of
> finding all 3825 eigenvalues, at least 99% of which
> will be of no interest and probably noise. Perhaps
> someone on the list knows a way to do this in fortran,
> in which case it could be incorporated into Ferret.
> Another possibility is that such techniques may be
> available in Matlab, which is very good with this
> kind of thing. Much as I value Ferret, there are some
> calculations, especially matrix manipulation, for which
> Matlab is inherently better. The routines in the Ferret
> EOF implementation are based on very old techniques, and
> are definitely not the most efficient. Perhaps the way
> to go would be to do the EOFs in Matlab and just use
> Ferret's superior plotting ability to plot the results.
>
> A quick and dirty way to improve the situation would
> be to decimate the data, say by regridding to a coarser
> grid, before finding EOFs on a large region. For example
> when Boyin wants EOFs over 120E-70W/40S-50N, there is
> probably no need to use the complete 2x2 degree resolution
> of the original data to recover the large-spatial-scale
> EOFs that are of interest. If you are pretty sure the
> same patterns would be seen in a 4x4 grid, then regridding
> before doing the EOFs would cut the CPU time down by a
> factor of 64 (1/4 the number of grid cells, and the cube
> root of the number of iterations to do the inversion).
> Even just cutting down the zonal resolution would make
> a large difference in the needed CPU.
>
> The reason that the slowdown is seen in the SAVE command
> (in Boyin's example) is that the definition command
>
> LET eof_xyfcn = eof_space(sst, 0.5)
>
> does nothing except to specify the definition of the
> variable eof_xyfcn. No actual computation is done until
> this variable is needed (for example a PLOT or SAVE).
> Therefore, doing a SAVE immediately is a very good idea,
> ensuring that the computation is done only once.
>
> Given that most of the EOFs will be useless, there is
> no reason to SAVE all 3825 of them. It will just make
> a large file of mostly junk. In this case, for example:
>
> SAVE/file=eof_xyfcn25.cdf eof_xyfcn[l=1:25]   ! first 25 EOFs
>
> This will not save any CPU time, unfortunately, but it will
> make a much smaller file. Then:
>
> CAN VAR eof_xyfcn	! be sure not to compute this again
> USE eof_xyfcn25.cdf	! bring in the premade file
> FILL eof_xyfcn[l=1]	! or other plots as wanted
>
> Billy K
> ------------------------------------------------------------
>> From: Boyin Huang <huang_boyin@yahoo.com>
>> Date: Thu, 25 Sep 2003 13:40:16 -0700 (PDT)
>> Subject: SAVE in EOF analysis
>>
>> Hi, ferreters:
>>
>> Is the ferret so slow in doing EOF analysis?
>>
>> Using the demo file for eof analysis:
>>
>> -------------------------------------------------------
>> SET REGION/X=67w:1w/Y=11S:11N
>> USE coads_climatology
>> LET eof_xyfcn = eof_space(sst, 0.5)
>> SAVE/CLOBBER/FILE=sst_clim_eof_space.cdf eof_xyfcn
>>
>> USE sst_clim_eof_space.cdf
>> SET VIEW ul; fill/l=1/TITLE="eof 1" eof_xyfcn; go land
>> -------------------------------------------------------
>>
>> Which works fine to me. Now I wish to have EOF in a
>> larger domain, say 120E-70W and 40S-50N.
>>
>> I replaced set region by
>> SET REGION/X=120e:70w/Y=40S:50N
>> and added
>> SET MEMORY/SIZE=260.(my computer memory is 1.5gb).
>>
>> I found that the ferret runs for ever (how long should
>> I wait for?).
>>
>> I tried to diagnose the problem by typing these
>> command lines interactively line by line, and found
>> the problem is that the program was stuck by SAVE
>> command.
>>
>> I then tried NOT to use the save command, but ferret
>> says that the variable was not found.
>>
>> So, do we have to save EOF first, and the read and
>> plot?
>>
>> How can I do the EOF analysis within a larger space
>> domain and long time series? Say, simulated global SST
>> with 200x100 grids for 1200 month, sst(200,100,1200).
>>
>>
>>
>> =====
>> Boyin Huang
>> The Center for Research on the Changing Earth System (CRCES)
>> 10211 Wincopin Circle, Suite 240
>> Columbia, Maryland 21044
>> Tel. 410-992-5300/819 (O), 410-737-4557(H)
>> Fax. 410-992-5944, Email. byh@crces.org
>> http://byh.crces.org
>>
>> __________________________________
>> Do you Yahoo!?
>> The New Yahoo! Shopping - with improved product search
>> http://shopping.yahoo.com
>>