[Thread Prev][Thread Next][Index]

Re: Best strategy for long-time series


Jonathan Callahan wrote:

    * Create files that contain more than a single time step and
      aggregate those.

      I suspect that the DODS aggregation server is slow in part because
      it is being asked to do  file IO on thousands of files when you
      ask for a time series.  If this is the case, creating yearly or
      even monthly files out of your snapshots and then stringing those
      together with aggregation should solve the problem.  The drawback
      here is that you have to reformat your data.
The consensus seems to be that the GDS can handle this job (though I'm hoping the Aggregation server folks have some suggestions for tuning and caching that can help it do a better job).

That said, if you want to concatenate a bunch of netCDF files into one file the NCO tools suite does this quite well. I know everybody has their favorite tool for doing such a job and this one is mine. See the User's Guide at http://nco.sourceforge.net/ for more information.


[Thread Prev][Thread Next][Index]

Dept of Commerce / NOAA / OAR / PMEL / TMAP
Contact Us | Privacy Policy | Disclaimer | Accessibility Statement