hi Ying, On 7/14/2010 12:34 PM, Roland Schweitzer wrote:
Ying at NCCS wrote:We (at least I) don't recommend this type of aggregation.Hi Roland and AnsleyI remember many years ago I used ferret jnl to aggregate model run files and configured into LAS. Does new ferret/las still support this kind aggregation ?
You may be thinking of the multi-dataset "descriptor files". There are some Unix command-line tools to create those from a set of netCDF files and which you should be able to find in the Users Guide. But we do not recommend descriptor files for use in LAS. I can't imagine it would be faster. Aggregation via TDS is the preferred method; it allows for "native striding" that is used by LAS for large datasets (see the Ferret Documentation "netCDF and strides" http://ferret.pmel.noaa.gov/Ferret/documentation/users-guide/data-set-basics/NETCDF-DATA#_VPINDEXENTRY_167 ) and also allows for analysis and difference operations done by F-TDS, which I don't believe will work for data that's accessed using a descriptor file.
With all the new caching and other optimizations in TDS I think the performance it quite good even for large data sets. There are always trade-offs, but I think the performance is adequate the the other characteristics like ease of installation and maintenance make TDS a better solution than "aggregating" with a Ferret descriptor file.It seems to me that LAS/Ferret is fast enough but access model data that are aggregated through either TDS/GDS is very slow, in particular for large number of files. It might be better if just let Ferret read the file directly. Also I wonder if this new ferretnc4 supports parallel IO (as netcdf 4 does).I am interested at running LAS backend service on our center's powerful machine with GPFS support, so it may speed up data analysis function on the server side.If your LAS F-TDS on this backend, then you will be able to do server-side analysis on that powerful machine and LAS will take advantage of those server side functions for regridding when doing comparisons.Thanks again for all these years support to LAS usersRoland