Re: [ferret_users] EOFSVD_STAT error

["William S. Kessler" <william.s.kessler@xxxxxxxx>]

Consider that an EOF calculation is usually used to _simplify_ an interpretation. Namely, you expect the result to represent the variability in a very few functions, often only 1 or 2. 

Thus the small details in any dimension are unlikely to be important, and subsampling suggested by Russ is very appropriate. Subsampled fields should give a very good result, in addition to being much easier to work with.

Some thought is needed about how to subsample. For instance, if your data is daily but you're really interested in low frequencies, then subsampling to monthly might be the right approach. In that case, consider sampling from a running 31-day running mean (i.e. monthly average) to avoid extreme values that might occur on a particular day. And in any subsampling, always check by overplotting the original and subsampled time series at a few points, so easy to do in Ferret. 

Billy K

> On Oct 5, 2016, at 10:44 PM, Russ Fiedler <russell.fiedler@xxxxxxxx> wrote:
> 
> Hi,
> 
> Yes, you are requesting a huge amount of memory by the looks. I had a quick glance at the source code and there is a call
> in routine eofsvd_stat_work_size to allocate an array of size approximately the square of the number of horizontal points (possibly packed
> to avoid all missing values/zeros). Anyway, this means an array of about 20GB is being requested!
> 
> You'll need to subsample in space pretty aggressively or possibly look at doing this outside Ferret. You may need an out of core solution or
> be able to perform the SVD on a distributed memory device (or be lucky enough to have heaps of shared memory).
> 
> Russ
> 
> 
> 
> On 06/10/16 15:06, Manche Shivashankar wrote:
>> Thank you so much Russ for your advice
>> 
>> Here are the details of my data
>> it is daily ssh data covering north Indian ocean having dimensions 320 by 240 by 4866 latitude longitude and time respectively.
>> and the script I am using as follows
>> 
>> yes? use sla.nc
>> yes? let Eof_Grid = sla[x=30:110,y=-30:30,l=1:4886]
>> yes? let Eof_Stat = eofsvd_stat(Eof_Grid)
>> then ferret is asking for memory size to increase after increasing the memory by 
>> set memory/size= command the error is coming like this
>> 
>> **ERROR in efcn_compute() allocating -92862752 bytes of memory
>>     work array 5:  X=1:1604808814, Y=1:2, Z=1:1, T=1:1, E=1:1, F=1:1
>>  **ERROR: error in external function
>> 
>> how I can solve this problem
>> I am great full if someone helps me.
>> 
>> thanks and regards
>> shiva shankar
>> 
>> On Thu, Oct 6, 2016 at 8:14 AM, Russ Fiedler <russell.fiedler@xxxxxxxx> wrote:
>> 
>> Hi,
>> 
>> The problem is that you are requesting too much memory. The work array 5 looks to have 2.3 billion elements which comes to about 10GB if single precision or 20GB if double. The negative in the first error message indicates an integer overflow.
>> 
>> From the documentation
>> 
>> http://ferret.pmel.noaa.gov/Ferret/documentation/users-guide/appendix-a-external-functions/EOF-STAT?searchterm=eof
>> 
>> "The EOFSVD analysis solves a matrix problem where the matrix is dimensioned (NX*NY*NZ) by NT, which can quickly become quite large. The EOFSVD functions use other workspace as well which demands even more memory, and often memory must be increased with the SET MEMORY command. Regridding to a coarser grid or restricting the region may be necessary."
>> 
>> Could you show the size of your variable and the exact commands that you are using to do the eof computation? Somebody may be able to suggest a better way to do this.
>> 
>> Russ
>> 
>> 
>> 
>> On 06/10/16 13:02, Manche Shivashankar wrote:
>>> Dear all,
>>> 
>>> I am trying to see each eigen mode variance by list/i=1:10/j=2 eofsvd_stat  command after eof decomposition of data. While trying it the error comes like this 
>>> 
>>> 
>>> **ERROR in efcn_compute() allocating -92862752 bytes of memory
>>>     work array 5:  X=1:1604808814, Y=1:2, Z=1:1, T=1:1, E=1:1, F=1:1
>>>  **ERROR: error in external function
>>> 
>>> what could be the reason.
>>> 
>>> Please help me.
>>> Thanks in advance.
>>> 
>>> Regards
>>> Shiva shankar
>> 
>> 
> 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
William S. Kessler
NOAA / Pacific Marine Environmental Laboratory
7600 Sand Point Way NE
Seattle WA 98115 USA

william.s.kessler@xxxxxxxx
Tel: +1 206-526-6221
Fax: +1 206-526-6744
Web: http://faculty.washington.edu/kessler/