[Thread Prev][Thread Next][Index]

Re: memsize ferret error



Peter,

I am able to duplicate your error with some of our own data and have submitted a bug to the Ferret developers.  In the interim I have a work around.  Just comment out the following lines in las/jnls/std_list.jnl:

!IF ($VAR_GRID"0") THEN
!   set grid ($VAR_GRID)
!ELSE
!   set grid $2[d=$1]
!ENDIF


You may also find that you need to increase the timeout on your server.  You can do that by including the following lines in your dataset.xml file:

<properties>
 <ferret>
  <timeout>180</timeout>
 </ferret>
</properties>


This will increase the timeout to from the default 60 seconds to 180 seconds for the dataset in question.

Let us know how these solutions work for you.

You will also be interested to know that we are working on 'batch mode' functionality for data subsetting requests like the ones plaguing you at the moment.  This will allow LAS to return immediately with a message describing the FTP (or OPeNDAP) location of the file that is in the process of being created.  This is experimental code at the moment but we could make it available to you for testing if you are interested.


-- Jon


Peter.J.Turner@csiro.au wrote:

Hi Jon,

 

      I have attached a number of files containing (I hope) what you want and added the dataset xml and des file. There are 10 netCDF files, one for each year of the dataset. I have provided an ncdump for the 1994 file. I am using LAS 6.3. I have copied LASinfo.pl into server.  I had previously modified std_initialize.jnl by adding a line “set mem/size=200” at the end.  Unfortunately although this worked at the start it seemed that this request failed after the server had been running for a while. Indeed this was causing the system to fail when it really did not need that much memory. I have now set the memory to 128 using in las.xml

(<memsize>128<</memsize> in the <properties><ferret> group). 

The example that has failed has asked for just over 100 Mwords. The machine itself only has 512Mbytes and if a Ferret word is 4 bytes we are very close to the entire machine memory.

 

I am not sure what is reasonable to expect from the LAS server. Some guidance would be appreciated.

 

The LAS server is at http://www.marine.csiro.au/las/servlets/dataset

 

Regards

 

Peter Turner

 

 

-----Original Message-----
From: Jonathan Callahan [mailto:Jonathan.S.Callahan@noaa.gov]
Sent: Wednesday, 7 July 2004 2:44 AM
To: Turner, Peter (Marine, Hobart)
Cc: pbmoses@ucsd.edu; oar.pmel.las_users@noaa.gov
Subject: Re: memsize ferret error

 

Peter,

It would be helpful to know the following:

·        version of LAS

·        precise LAS request (xml request object)

·        ncdump -h of the data file

With these we should be able to come up with similar tests at our end to try and mimic the problem.

Of course, having access to your LAS is very useful so that we can see the problems.  And putting the las/etc/LASinfo.pl (<6.3) or las/bin/LASinfo.pl (>=6.3)  cgi script in las/server/ will allow  us to do some debugging remotely.


-- Jon


Peter.J.Turner@csiro.au wrote:

Hi Phil,
 
        I have had the same problem with the LAS server I have setup. I
adjusted the ferret memory size and that helped but did not solve the
problem. I am having some difficulty understanding why Ferret seems to
need large amounts of memory. Perhaps some others who have more
experience with  LAS and Ferret could provide some guidance about how to
determine Ferret memory requirements?
 
 
Cheers
 
Peter Turner
 
-----Original Message-----
From: owner-las_users@pmel.noaa.gov
[mailto:owner-las_users@pmel.noaa.gov] On Behalf Of Phil Moses
Sent: Saturday, 3 July 2004 3:48 AM
To: las_users
Subject: memsize ferret error
 
Hello,
 
With some help from others, we have identified some errors that we have
been getting on our LAS server. 
 
It was suggested that we modify the memsize within the ferret scripts
and see if this helps. 
 
We are upgrading the physical memory of the machine, as we have
identified a need to do this anyway. However, I am curious if anyone can
pin-point the following error.
 
 **ERROR: insufficient memory: 27324000 words were requested.
 
Will adjusting the memsize in the ferret scripts help resolve this?
 
I have a few ideas but would like any suggestions from the experts prior
to making the adjustments. 
 
 
 
 
[Tue Jun  1 03:12:53 2004] yes? set
region/x="20.0":"120.0"/y="-20.0":"5.0"/z="5":"5450"/t="16-Jan-1992":"16
-Dec-2002"
[Tue Jun  1 03:12:53 2004] yes? GO "jnls/std_list.jnl" "1" "salt"
"output/4cbe2e1e004de40a4b1ba1a1d69a2c5b.nc" "cdf"
[Tue Jun  1 03:12:53 2004]  **ERROR: insufficient memory: 27324000 words
were requested.
[Tue Jun  1 03:12:53 2004] define symbol data_isize
`salt[d=1],return=isize`
[Tue Jun  1 03:12:53 2004] Command file, command group, or REPEAT
execution aborted
[Tue Jun  1 03:13:45 2004] Adding an acceptable error string: "*** NOTE:
".
 
 
Thanks,
Phil
 
  

DATE: Wed Jul 7 11:18:59 2004 RQST: GET /las-bin/LASserver.pl?xml=%3C%3Fxml+version%3D%221.0%22%3F%3E%3ClasRequest+package%3D%22%22+href%3D%22file%3Alas.xml%22+%3E%3Clink+match%3D%22%2Flasdata%2Foperations%2Fdata%22+%2F%3E%3Cproperties+%3E%3Cferret+%3E%3Csize+%3E.5%3C%2Fsize%3E%3Cformat+%3Ecdf%3C%2Fformat%3E%3C%2Fferret%3E%3C%2Fproperties%3E%3Cargs+%3E%3Clink+match%3D%22%2Flasdata%2Fdatasets%2FSST10d_nc%2Fvariables%2Fsst%22+%2F%3E%3Cregion+%3E%3Crange+low%3D%22154.0%22+type%3D%22x%22+high%3D%22156.0%22+%2F%3E%3Crange+low%3D%22-37.0%22+type%3D%22y%22+high%3D%22-34.0%22+%2F%3E%3Crange+low%3D%2201-Jan-1994%22+type%3D%22t%22+high%3D%2230-Jun-1996%22+%2F%3E%3C%2Fregion%3E%3C%2Fargs%3E%3C%2FlasRequest%3E HOST: 140.79.17.2 CLNT: Java/1.4.1_02 XML: <?xml version="1.0"?><lasRequest package="" href="" class="moz-txt-link-rfc2396E" href="">"file:las.xml" ><link match="/lasdata/operations/data" /><properties ><ferret ><size >.5</size><format >cdf</format></ferret></properties><args ><link match="/lasdata/datasets/SST10d_nc/variables/sst" /><region ><range low="154.0" type="x" high="156.0" /><range low="-37.0" type="y" high="-34.0" /><range low="01-Jan-1994" type="t" high="30-Jun-1996" /></region></args></lasRequest> CACHE: false

<datasets> <SST10d_nc name="SST 10-day composite" url="" class="moz-txt-link-rfc2396E" href="">"file:SST10d" doc="doc/Australasian_sst.html"> <variables> <sst name="sst" units="DEG C"> <link match="/lasdata/grids/SST10d_nc_longitude_latitude_time_grid"/> </sst> </variables> </SST10d_nc> </datasets> <grids> <SST10d_nc_longitude_latitude_time_grid> <link match="/lasdata/axes/SST10d_nc_longitude"/> <link match="/lasdata/axes/SST10d_nc_latitude"/> <link match="/lasdata/axes/SST10d_nc_time"/> </SST10d_nc_longitude_latitude_time_grid> </grids> <axes> <SST10d_nc_longitude type="x"> <arange start="79.987" step="0.042" size="2621"/> </SST10d_nc_longitude> <SST10d_nc_latitude type="y"> <arange start="-64.9979" step="0.036" size="2085"/> </SST10d_nc_latitude> <SST10d_nc_time type="t" units="days"> <arange start="1993-Oct-07" step="2" size="1767"/> </SST10d_nc_time> </axes>

[Thread Prev][Thread Next][Index]

Dept of Commerce / NOAA / OAR / PMEL / TMAP
Contact Us | Privacy Policy | Disclaimer | Accessibility Statement