[Thread Prev][Thread Next][Index]

RE: eof calculus and missing values



Dear Ansley,

Yes I will appreciate if you send to me the external code for svd
calculation. My O.S is a Linux-Debian Woody 3.0 on a AMD Duron (~ Pentium
III).

Asking for some advice on statistical lists (R project) about svd and
missing values, seems to be a very controversial issue the filling of
missing values in a time series. When few gaps are present the EM algorithm
based on: Schafer, J.L.(1997) Analysis of Incomplete Multivariate Data,
Chapman & Hall, London, will perform a maximum-likelihood estimation on the
matrix of incomplete data using this algorithm. And yes, it is not possible
for svd to handle missing values because is an undefined matrix operation.

Thanks to everybody. It has been very helpful

Cheers

Antonio

> -----Mensaje original-----
> De: Ansley Manke [mailto:Ansley.B.Manke@noaa.gov]
> Enviado el: viernes, 12 de septiembre de 2003 18:10
> Para: antonio rodriguez
> CC: Mick Spilane
> Asunto: Re: eof calculus and missing values
>
>
> Hi Antonio,
> Would you like the EOF functions which use the singular value
> decomposition to test?  We just have not had a chance to fully
> test them.  I'm sure the code itself is fine; it's been in use by
> Billy Kessler for a long time, and the Ferret implementation is
> so similar to what we did with the previous set of functions that
> it's probably fine as well, but we just have not had a chance to
> do much testing.
>
> If you're interested, let me know what operating system you use
> and I'll see if I can provide compiled functions, or I could send the
> source code for you to work with
>
> antonio rodriguez wrote:
>
> > Dear Mick,
> >
> > Yes it helps a lot and is a very good explanation. Wouldn't be more
> > interesting instead of using covariance matrices, the use of
> the singular
> > value decomposition to obtain both the eigenvectors and
> eigenvalues. I've
> > read that it is a more robust and computanionally efficent
> method than the
> > use of huge covariance matrices. The issue is that I don't know
> if the svd
> > method accepts missing values.
> >
> > Cheers,
> >
> > Antonio
> >
> > > -----Mensaje original-----
> > > De: Mick Spillane [mailto:spillane@pmel.noaa.gov]
> > > Enviado el: jueves, 11 de septiembre de 2003 19:15
> > > Para: antonio rodriguez
> > > CC: Ferret
> > > Asunto: Re: eof calculus and missing values
> > >
> > >
> > >
> > > > Is it possible to know how Ferret manages the presence of
> > > missing values in
> > > > the calculation of the EOF.
> > >
> > > Hi Antonio,
> > >       You may get a more detailed explanation of this from one of the
> > > others in the group or by looking at the code for these external
> > > functions.  However here is a brief statement of how it is done.
> > >
> > >       Basically what needs to be done is to get the eigensolution to
> > > a covariance matrix. Such a matrix can be calculated even when some
> > > data values are missing, provided that sufficient cross products are
> > > available to make all elements in the matrix meaningful.  The latter
> > > is the point of the PERCENT parameter in eof_stat, eof_space, and
> > > eof_tfunc - if a grid point has so many missing values that its cross
> > > products are unreliable it is dropped entirely from the analysis.
> > >       So now there is a covariance matrix, slightly imperfect, but
> > > for which a set of eigenvectors can be computed, using a standard
> > > method (such as "jacobi.f" in Numerical Recipes, Press et al.).  The
> > > trick comes in computing the "time functions".  Some of the values
> > > of the individual data series, needed for a normal evaluation are
> > > missing.  The ferret eof routines use an objective method, based on
> > > some unpublished notes of Dudley Chelton and similar to the algebra
> > > in Davis (1976) and Chelton et al. (1982) to find the best, in a least
> > > squares sense, value of the time function at times where there are
> > > missing values.  Not provided in the Ferret version, though available
> > > via the underlying algorithm, is a measure of the "expected" error at
> > > those points in time where the objective estimation is necessary.
> > >
> > > Hope this helps.  The references below should give you the flavor
> > > of the objective method employed.
> > >
> > > Mick
> > >
> > > Davis, R.E., 1976 Journal of Physical Oceanography, 6: 249-266.
> > >
> > > Chelton, D.B., P.A. Bernal, and J.A. McGowan, 1982, Journal of
> > >       Marine Research, 40(4): 1095-1125.
> > >
> > >
>
> --
> Ansley Manke  Pacific Marine Environmental Laboratory  Seattle WA
>
>
> ---
> Incoming mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.512 / Virus Database: 309 - Release Date: 19/08/2003
>
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.512 / Virus Database: 309 - Release Date: 19/08/2003



[Thread Prev][Thread Next][Index]

Dept of Commerce / NOAA / OAR / PMEL / TMAP

Contact Us | Privacy Policy | Disclaimer | Accessibility Statement