[Thread Prev][Thread Next][Index]

Re: Best strategy for long-time series

Hi Yingshuo,

Thanks for the link. To clarify: GDS has the ability to add metadata attributes, like text descriptions, but not new grid dimensions, which I think is perhaps what you meant. Grids in the un-aggregated datasets must already have a time dimension (even if its just length 1), in order for GrADS to be able to template them by time. In Agg server terminology, GDS supports "JoinExisting" but not "JoinNew".

I think it may be the "JoinNew" capability you need for HDF - at least some of the data we've looked at, has been missing any axes other than X and Y. However, if there is at least some grid dimension present (like "scan") that can be used as a placeholder for time, then you could potentially aggregate HDF with the GDS.

- Joe

On Tuesday, June 10, 2003, at 01:38 PM, yingshuo shen wrote:


well, that's great... i would like to see an example in gds.xml file like
what john caron did in his
http://www.unidata.ucar.edu/projects/THREDDS/tech/aggServer/ AggServerConfig.
it would be nice if you can put some examples of this nice feature for
public downloading.. i would like to see if it is possible to serve hdf
file via GDS (for example quikscat level 3 data from podaac.jpl.nasa.gov).

----- Original Message -----
From: "Joe Wielgosz" <joew@cola.iges.org>
To: "yingshuo shen" <yshen@musashimaru.soest.hawaii.edu>
Cc: "John Caron" <caron@unidata.ucar.edu>; <las_users@ferret.wrc.noaa.gov>;
"Jean-Francois PIOLLE" <Jean.Francois.Piolle@ifremer.fr>; "Jonathan
Callahan" <Jonathan.S.Callahan@noaa.gov>; "jma" <jma@cola.iges.org>
Sent: Tuesday, June 10, 2003 9:58 AM
Subject: Re: Best strategy for long-time series

Hi Yingshuo,

Just for the record, if you want to add your own metadata to GDS
datasets, you can do so by providing a supplementary DAS file:

- Joe

On Tuesday, June 10, 2003, at 11:53 AM, yingshuo shen wrote:


gds is doing a great job with grib and other binary files on
aggregation. caron's aggregation server can do things that gds is not
able to such as adding some metadata (during aggretation
aggType="JoinNew") and also it can aggregate remote data (via DODS
type). These two OPeNDAP servers are becoming apdrc data service
infrastructure (http://apdrc.soest.hawaii.edu:9090). i am looking
forward to seeing the release of caron's NCML and it might be able to
help us to put hdf file on to our OPeDAP service.

----- Original Message -----
From: jma
To: Jonathan Callahan
Cc: Jean-Francois PIOLLE ; las_users@ferret.wrc.noaa.gov ; John Caron
Sent: Tuesday, June 10, 2003 7:56 AM
Subject: Re: Best strategy for long-time series

I agree with Shen Yingshuo. It would work if you put the SST data
behind a GrADS DODS Server (GDS) and then serve the GDS data set via
LAS. Grads can handle the aggregation of the 3-hourly files with ease,
and missing datasets and daily updates are no problem. We have similar
setups here at COLA that are operational; I'd be happy to help you get
started. When you open the GDS data set, only the metadata is read
initially, there is no I/O until you actually make a data request. A
long time series request might be a little bit slow, but perhaps not
more than the LAS time-out. Jon's suggestion of aggregating the 3-hour
files into monthly or daily chunks might speed things up. There are
some handy netcdf operator programs that could do this pretty easily (
http://nco.sourceforge.net ).

Jennifer Miletta Adams
4041 Powder Mill Road, Suite 302
Calverton, MD 20705

On Tuesday, June 10, 2003, at 11:15 AM, Jonathan Callahan wrote:


You have touched on a problem that we in the development group haven't
yet developed much expertise in. I would like to rephrase the
question to the group (and John Caron):

Can the DODS aggregation server efficiently serve up time series that
require opening hundreds or thousands of files?

Here are various potential solutions to the problem, each with its own

. Set up LAS to only allow selection of XY views while still
describing the dataset as XYT so that the time selector appears. Then
use custom code to map requests for a particular time onto a
particular file name to access. You should get a nice usable
interface but will lose the ability to create time series. The
documentation on customizing LAS code has a section that is relevant:


Clearly, I'll need to create an FAQ that addresses your problem more

. Create files that contain more than a single time step and aggregate

I suspect that the DODS aggregation server is slow in part because it
is being asked to do file IO on thousands of files when you ask for a
time series. If this is the case, creating yearly or even monthly
files out of your snapshots and then stringing those together with
aggregation should solve the problem. The drawback here is that you
have to reformat your data.

. Any other suggestions out there?

-- Jon

The advantage of this

Jean-Francois PIOLLE wrote:

I would like to serve with LAS a long time series of SST maps over
atlantic : we have up to 8 maps/a day (every 3 hours but some are
sometimes missing). This time series is daily updated with new maps. So
far, we have more than 4500 netCDF map files to serve.
I tried to serve them through a DODS aggregated server and set up the
LAS (v6.0) to access the data through this DODS server. But it appears
then that the first access to a map (actually the first access to a
dataset) is really slow, exceeding the LAS default time-out. This is
because (I guess) each time the DODS catalog is updated, a DODS dataset
has to be re-aggregated and this first step is very time-consumming.
My question is : is DODS the best strategy to serve a long time-series
through LAS? It seems that a Ferret descriptor file can't be used here
since it is limited to 500 files. What other strategy? I guess many
people have already experienced this problem...


Jean-francois Piolle

Jean-Francois PIOLLE
CERSAT French ERS Processing and Archiving Facility
29280 Plouzane

Tel.: (+33) 2 98 22 46 91 email: jfpiolle@ifremer.fr
Fax: (+33) 2 98 22 45 33 WWW: http://www.ifremer.fr/cersat


Joe Wielgosz / joew@cola.iges.org
Center for Ocean-Land-Atmosphere Studies (COLA)
Institute for Global Environment and Society (IGES)

Joe Wielgosz / joew@cola.iges.org
Center for Ocean-Land-Atmosphere Studies (COLA)
Institute for Global Environment and Society (IGES)

[Thread Prev][Thread Next][Index]

Dept of Commerce / NOAA / OAR / PMEL / TMAP
Contact Us | Privacy Policy | Disclaimer | Accessibility Statement