Re: [ferret_users] Got "no valid data" with eofsvd

[Ansley Manke <ansley.b.manke@xxxxxxxx>]

Hi,
The old functions did more than that; they took gaps into account and did not require the gaps to be at the same timestep for all locations. (I do not know the details of the algorithm).  If one simply took out gaps, then the timeseries at different locations would not represent the same set of times and the computation would not be correct.  We moved to these other functions, using singular value decomposition, which are faster, and we would like the user to correct for or fill in gaps in a way that makes sense for the particular data set.

The svd functions as we implement them in Ferret gather the data at all spatial locations where there is complete time series data, (so that if there are missing xy locations for instance over land, that is handled).  Then the functions compute the EOF's and for eofsvd_space, places the result back at the correct spatial locations.

If there are no xy locations with complete time series, the functions will be changed so that they simply bail out.  For your data, you would see:

Bailing out of external function "eofsvd_space":
        There are no spatial locations having complete time series.

and then you could use Ferret functions or another method to handle things correctly. For your data, you'd skip that first timestep. Other data it might make sense to use a smoothing or filling transformation to fill missing data.

On 9/25/2013 4:26 PM, Jian Ma wrote:

Hi Ansley,

Could you please update the function so that it can first identify the all missing time steps, take those out, and then do the eofsvd, and after, add back the missing t steps to the tfunc t series? If so, it gains back the ability what the old function has.

Many thanks,

Tony

On Wed, Sep 25, 2013 at 3:12 PM, Ansley Manke <ansley.b.manke@xxxxxxxx> wrote:

Hi Jian,
Thank you for writing back. I had not had a chance to look at your question. The earlier version of the EOF functions (eof_space, etc) called code which allowed missing data.  The eofsvd functions work only if there is NOT missing data within the time series - that is to say, if there is data at some time at a given spatial location, then there must be data at ALL times at that location.

It sounds as if there is a bug which so that the function is not reporting back to you that missing data was found. We'll take a look at why.

Ansley

On 9/25/2013 11:30 AM, Jian Ma wrote:

Hi All,

I figured out the reason. It is because I have t step filled with missing values. While this is not any problem with the old version ferret with eof_*, it is with newer version ferret with eofsvd_*. I guess this should be reported as a bug. Currently the only solution for me may be to move the time steps and remove the missing value t steps from the data.

Tony

On Mon, Sep 23, 2013 at 5:52 PM, Jian Ma <jianma@xxxxxxxxxx> wrote:

Hi All,

I met a strange problem with eofsvd for the attached data file. It turns out to be "No valid data". Here is my commands:

yes? set d sstappcf_ma.nc 

yes? fill/l=1 sstappcf_ma

yes? let exyt=eofsvd_space(sstappcf_ma)

yes? fill/l=1 exyt

However I got good results with a data file with time coordinate dd-mm-yyyy. This file has only 1:22 in t coordinate without actual dd-mm-yyyy. Is this the reason for the problem?

Many thanks for any hint,

Tony