Re: more on eof calculus and missing values

[Ansley Manke <Ansley.B.Manke@xxxxxxxx>]

Hi Andonio,
The code looks for any spatial locations where there are no
times at all, and removes them from the calculation entirely;
keeping track of the locations where enough data is present
to include.  At the end of the calculation, the results are put
back in their original locations.

Ansley Manke

antonio rodriguez wrote:

> Hi again,
>
> When working with COADS data, continents represent vast areas of missing
> values, so sufficient cross products are unavailable. How Ferret sidesteps
> this issue?
>
> Cheers
>
> Antonio
>
> > -----Mensaje original-----
> > De: Mick Spillane [mailto:spillane@pmel.noaa.gov]
> > Enviado el: jueves, 11 de septiembre de 2003 19:15
> > Para: antonio rodriguez
> > CC: Ferret
> > Asunto: Re: eof calculus and missing values
> >
> >
> >
> > > Is it possible to know how Ferret manages the presence of
> > missing values in
> > > the calculation of the EOF.
> >
> > Hi Antonio,
> >       You may get a more detailed explanation of this from one of the
> > others in the group or by looking at the code for these external
> > functions.  However here is a brief statement of how it is done.
> >
> >       Basically what needs to be done is to get the eigensolution to
> > a covariance matrix. Such a matrix can be calculated even when some
> > data values are missing, provided that sufficient cross products are
> > available to make all elements in the matrix meaningful.  The latter
> > is the point of the PERCENT parameter in eof_stat, eof_space, and
> > eof_tfunc - if a grid point has so many missing values that its cross
> > products are unreliable it is dropped entirely from the analysis.
> >       So now there is a covariance matrix, slightly imperfect, but
> > for which a set of eigenvectors can be computed, using a standard
> > method (such as "jacobi.f" in Numerical Recipes, Press et al.).  The
> > trick comes in computing the "time functions".  Some of the values
> > of the individual data series, needed for a normal evaluation are
> > missing.  The ferret eof routines use an objective method, based on
> > some unpublished notes of Dudley Chelton and similar to the algebra
> > in Davis (1976) and Chelton et al. (1982) to find the best, in a least
> > squares sense, value of the time function at times where there are
> > missing values.  Not provided in the Ferret version, though available
> > via the underlying algorithm, is a measure of the "expected" error at
> > those points in time where the objective estimation is necessary.
> >
> > Hope this helps.  The references below should give you the flavor
> > of the objective method employed.
> >
> > Mick
> >
> > Davis, R.E., 1976 Journal of Physical Oceanography, 6: 249-266.
> >
> > Chelton, D.B., P.A. Bernal, and J.A. McGowan, 1982, Journal of
> >       Marine Research, 40(4): 1095-1125.
> >
> >
> > ---
> > Incoming mail is certified Virus Free.
> > Checked by AVG anti-virus system (http://www.grisoft.com).
> > Version: 6.0.512 / Virus Database: 309 - Release Date: 19/08/2003
> >
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.512 / Virus Database: 309 - Release Date: 19/08/2003

--
Ansley Manke  Pacific Marine Environmental Laboratory  Seattle WA