[Thread Prev][Thread Next][Index]

RE: [las_users] Error on large/high volume data sets - "remote server was unable to deliver data"




When I put in a request for the entire data set (~30yrs) I get the error below.  I don’t get this error at 8yrs of data.  We have set up  our files such that LAS reads them as contiguous in time (start, step, number of entries) – but we do this for all our data sets, and only MUR (high resolution) and AVHRR_OI (30 years) give us issues such as the one below.  Please let me know if I should copy more of the error for you.  Thanks!


Here is the error I receive:


Unfortunately, we cannot provide what you have asked for.

A remote server was unable to deliver the data LAS needs to make your product.

LAS was trying to access these servers:

  1. http://thredds.jpl.nasa.gov/thredds/dodsC/GHRSST_NCDC-L4LRblend-GLOB-AVHRR_OI_TIMEAGG/aggregate__ghrsst_NCDC-L4LRblend-GLOB-AVHRR_OI.ncml

Advanced users may request more technical information

Email to the site administrator.

Server at Revision rNo version information available.

**netCDF error

             NetCDF: Malformed or inaccessible DAP DATADDS (OPeNDAP/netCDF Error code -73)

             Data set: http://thredds.jpl.nasa.gov/thredds/dodsC/GHRSST_NCDC-L4LRblend-GLOB-AVHRR_OI_TIMEAGG/aggregate__ghrsst_NCDC-L4LRblend-GLOB-AVHRR_OI.ncml

LOAD the_plot_var

Command file, command group, or REPEAT execution aborted



go "$1"

!-> go "/export/00/tomcat/6.0.33/webapps/las/WEB-INF/classes/resources/ferret/temp/ferret_operation_1366928893528.jnl"

! Symbols from the server

                        DEFINE SYMBOL data_0_ID = analysed_sst-id-f7ec01a3bb

                                 DEFINE SYMBOL data_0_dataset_ID = PODAAC-GHAAO-4BC01

                                 DEFINE SYMBOL data_0_dataset_doc = http://podaac.jpl.nasa.gov/dataset/NCDC-L4LRblend-GLOB-AVHRR_OI

                                 DEFINE SYMBOL data_0_dataset_name = GHRSST Level 4 AVHRR_OI Global Blended Sea Surface Temperature Analysis

                                 DEFINE SYMBOL data_0_dataset_url = http://thredds.jpl.nasa.gov/thredds/dodsC/GHRSST_NCDC-L4LRblend-GLOB-AVHRR_OI_TIMEAGG/aggregate__ghrsst_NCDC-L4LRblend-GLOB-AVHRR_OI.ncml

                                 DEFINE SYMBOL data_0_dsid = PODAAC-GHAAO-4BC01

                                 DEFINE SYMBOL data_0_dsname = GHRSST Level 4 AVHRR_OI Global Blended Sea Surface Temperature Analysis

                                 DEFINE SYMBOL data_0_ftds_url = http://thredds.jpl.nasa.gov/thredds/dodsC/las/PODAAC-GHAAO-4BC01/data_thredds.jpl.nasa.gov_thredds_dodsC_GHRSST_NCDC-L4LRblend-GLOB-AVHRR_OI_TIMEAGG_aggregate__ghrsst_NCDC-L4LRblend-GLOB-AVHRR_OI.ncml.jnl

                                 DEFINE SYMBOL data_0_grid_type = regular

                                 DEFINE SYMBOL data_0_intervals = xyt

                                 DEFINE SYMBOL data_0_name = analysed sea surface temperature

                                 DEFINE SYMBOL data_0_points = xyt

                                 DEFINE SYMBOL data_0_region = region_0

                                 DEFINE SYMBOL data_0_title = analysed sea surface temperature

                                 DEFINE SYMBOL data_0_units = kelvin

                                 DEFINE SYMBOL data_0_url = http://thredds.jpl.nasa.gov/thredds/dodsC/GHRSST_NCDC-L4LRblend-GLOB-AVHRR_OI_TIMEAGG/aggregate__ghrsst_NCDC-L4LRblend-GLOB-AVHRR_OI.ncml

                                 DEFINE SYMBOL data_0_var = analysed_sst

                                 DEFINE SYMBOL data_0_xpath = /lasdata/datasets/PODAAC-GHAAO-4BC01/variables/analysed_sst-id-f7ec01a3bb

                                 DEFINE SYMBOL data_count = 1

                                 DEFINE SYMBOL ferret_fill_type = fill

                                 DEFINE SYMBOL ferret_interpolate_data = false

                                 DEFINE SYMBOL ferret_land_type = contour

                                 DEFINE SYMBOL ferret_mark_grid = no

                                 DEFINE SYMBOL ferret_service_action = Plot_2D

                                 DEFINE SYMBOL ferret_size = 0.5

                                 DEFINE SYMBOL ferret_view = xt

                                 DEFINE SYMBOL las_debug = true

                                 DEFINE SYMBOL operation_ID = Plot_2D

                                 DEFINE SYMBOL operation_key = FA43D44D5D40623D965EF96A5BB66D31

                                 DEFINE SYMBOL operation_name = Color plot

                                 DEFINE SYMBOL operation_service = ferret

                                 DEFINE SYMBOL operation_service_action = Plot_2D

                                 DEFINE SYMBOL product_server_default_dsid = ocean_atlas_subset

                                 DEFINE SYMBOL product_server_default_operation = Plot_2D_XY_zoom

                                 DEFINE SYMBOL product_server_default_option = Options_2D_image_contour_xy_7

                                 DEFINE SYMBOL product_server_default_varid = TEMP-ocean_atlas_subset

                                 DEFINE SYMBOL product_server_default_view = xy

                                 DEFINE SYMBOL product_server_ps_timeout = 7200

                                 DEFINE SYMBOL product_server_ui_timeout = 20

                                 DEFINE SYMBOL product_server_use_cache = false

                                 DEFINE SYMBOL product_server_version = 7.3

                                 DEFINE SYMBOL region_0_t_hi = 16-Jun-2012

                                 DEFINE SYMBOL region_0_t_lo = 01-Sep-1981

                                 DEFINE SYMBOL region_0_x_hi = 180

                                 DEFINE SYMBOL region_0_x_lo = -180

                                 DEFINE SYMBOL region_0_y_hi = 0

                                 DEFINE SYMBOL region_0_y_lo = 0

                                 DEFINE SYMBOL result_cancel_ID = cancel

                                 DEFINE SYMBOL result_cancel_filename = /usr/local/tomcat/webapps/las/output/FA43D44D5D40623D965EF96A5BB66D31_cancel.txt

                                 DEFINE SYMBOL result_cancel_type = cancel

                                 DEFINE SYMBOL result_count = 7

                                 DEFINE SYMBOL result_debug_ID = debug

                                 DEFINE SYMBOL result_debug_filename = /usr/local/tomcat/webapps/las/output/FA43D44D5D40623D965EF96A5BB66D31_debug.txt

                                 DEFINE SYMBOL result_debug_type = debug

                                 DEFINE SYMBOL result_map_scale_ID = map_scale

                                 DEFINE SYMBOL result_map_scale_filename = /usr/local/tomcat/webapps/las/output/FA43D44D5D40623D965EF96A5BB66D31_map_scale.xml

                                 DEFINE SYMBOL result_map_scale_type = map_scale

                                 DEFINE SYMBOL result_plot_image_ID = plot_image

                                 DEFINE SYMBOL result_plot_image_filename = /usr/local/tomcat/webapps/las/output/FA43D44D5D40623D965EF96A5BB66D31_plot_image.gif

                                 DEFINE SYMBOL result_plot_image_type = image

                                 DEFINE SYMBOL result_ref_map_ID = ref_map

                                 DEFINE SYMBOL result_ref_map_filename = /usr/local/tomcat/webapps/las/output/FA43D44D5D40623D965EF96A5BB66D31_ref_map.gif

                                 DEFINE SYMBOL result_ref_map_type = image

                                 DEFINE SYMBOL result_rss_ID = rss

                                 DEFINE SYMBOL result_rss_filename = /usr/local/tomcat/webapps/las/output/436FEA43FD05A6DEF956F5B127DA46A0_rss.rss

                                 DEFINE SYMBOL result_rss_type = rss

         GO ($operation_service_action)

!-> GO Plot_2D



And it goes from there.











From: Ansley Manke [mailto:ansley.b.manke@xxxxxxxx]
Sent: Thursday, April 25, 2013 3:26 PM
To: Shelton, Kacie E (398C)
Cc: las users [las_users@xxxxxxxx]
Subject: Re: [las_users] Error on large/high volume data sets - "remote server was unable to deliver data"


Hi Kacie,
No, LAS will never load anywhere near a GB of data for a plot. LAS will ask the thredds server where the data is located for a decimated subset of the data, covering the time span and region in x,y, and/or z. On LAS plots, there'll be an annotation "subsampled a in X, b in Y".  That's indicating the decimation that was done. For example instead of loading the 32768 by 16384 grid for an XY plot, it'll ask for only something like 500x300, by requesting every 64th point in X and 56th point in Y. If you zoomed in to smaller XY regions, you'd get more resolution. For a Hovmuller plot, a similar thing is done, asking for every 10th timestep and 40th point in X for instance.

When it says "remote server was unable to deliver data", it means exactly that. The data request from the thredds server returned an error. It's not the size of the request but some other interaction of the data request with the data on that server. 

Possibly the way the data is stored, or the aggregation that is used, is making it difficult for the server to return data. Imagine if all the data for this dataset is stored in separate files, one file per timestep, and the dataset is an aggregation of all of those files. To get a time series or a Hovmuller plot, the data server software is opening up each file, reading the piece of it that it needs, and then going on to the next file. If the individual datasets are compressed, then decompression is also being done.  The thredds server software is meant to be able to do this, but one can imagine the complexities.

We'd be happy to look at the errors you're seeing; click on the link in the "Advanced users may see more technical information" and let us know what it says.  That link returns a big long debug output listing, but the error messages appear at the top of it, so the first couple dozen lines have the information.


On 4/25/2013 12:07 PM, Shelton, Kacie E (398C) wrote:



To confirm that I understand;


High volume data (either enough high resolution data or long enough time series of data) will produce an error, and the only thing to do is to post caveats to users to not select the entire time-series at once.   Is this correct?


We can produce a hovmoller plot for ~1.5 yrs of data for MUR, and about 8 yrs for AVHRR_OI. 


Is there a known limit on how much data can be handled on one request?  Ie, no more than X GB per request, or no more than Y granules in one request?


Thank you,





From: Ansley Manke [mailto:ansley.b.manke@xxxxxxxx]
Sent: Friday, April 19, 2013 5:49 PM
To: Shelton, Kacie E (398C)
Cc: las users [las_users@xxxxxxxx]
Subject: Re: [las_users] Error on large/high volume data sets - "remote server was unable to deliver data"


Hi Kacie,
As a test, try making a short time selection and smaller range in longitude to see if you can get a result.

In LAS we handle large grids like this by asking for a decimated version when the request is really large. A several-hundred by several-hundred pixel image can't show that fine a grid anyway so we make a smaller request, striding the data request so it's a more reasonable size to be shipping around.

I have put the MUR dataset into a test server running LAS v8 and I can get an XT plot for a year's worth of it. See the attached screenshot. However for the entire time range the data that is returned from the NetCDF library reading the datset is all zero's.

Likewise I can make time series plots that are 6 or 8 months long, but requests for more data results in error messages from the NetCDF library reading the data.

Going back to the html view of the dataset, I can get coordinates successfully. Ask for lat 8000:1:8000, lon 32000:1:32000, time 0:10:2791 and the coordinates are returned correctly.  Try:

But, try to get analysed_sst at those same index ranges, and we get an error.  That's this constrained url:

The software behind the url access to the datset is not the same netCDF library that Ferret, running within LAS is using to read this opendap data but it seems to me this is an indication that both pieces of software are having trouble accessing the dataset along the time dimension.


On 4/19/2013 1:55 PM, Shelton, Kacie E (398C) wrote:


We are getting a near instant error when requesting Hovmoller plots of MUR and AVHRR_OI data sets.  (See included images.)

"A remote server was unable to deliver the data LAS needs to make your product."

The links are good; MUR links to here: http://thredds.jpl.nasa.gov/thredds/dodsC/GHRSST_JPL-L4UHfnd-GLOB-MUR_TIMEAGG/aggregate__ghrsst_JPL-L4UHfnd-GLOB-MUR.ncml.html

Likewise the link for AVHRR_OI is good. 

We suspect this error has something to do with the volume of requested data; MUR is high resolution (~1km global), and AVHRR_OI has 30 yrs of global data. 

We do not get an error on loading any one particular granule, nor on a time series plot -- the Hovmoller plot is where we see this error. 

Any ideas on what is going on, or how we can fix this?

Thank you,



[Thread Prev][Thread Next][Index]

Contact Us
Dept of Commerce / NOAA / OAR / PMEL / TMAP

Privacy Policy | Disclaimer | Accessibility Statement