Re: Large files access time

[Steve Hankin <Steven.C.Hankin@xxxxxxxx>]

Hi Jean-Marie,

Did you try
    yes?  USE/regularT my_file.nc

When that qualifier is used Ferret reads only the first and last time step of
the time axis.  There's a long disk seek between the two time steps, since the
file is large, but it would account for only a fraction of a second in the
worst case.

If you are using /regularT and still getting the slow speed then I cannot
account for it.  Can you send us the output of ncdump -c  ?

    - steve


J-M Epitalon wrote:

> Jon and Steve,
>
> sorry. I mislead you.
> My file is 1.5 Gb (not Mb) !
> and it is accessed through NFS.
> I suppose this is why it is really slow.
>
> Anyway, when accessed localy, it still takes half a minute to open it.
> (3 to 4 mn thru NFS)
>
> Since I want to keep the system architecture with NFS, I solved my
> problem by splitting the file in several parts and using a MC file.
>
> Please, what would you suggest as an alternative solution ?
>
> Jean-Marie Epitalon
> CERFACS
> Toulouse, FR
>
> On Fri, 2005-04-22 at 18:27, Jonathan Callahan wrote:
> > Jean-Marie,
> >
> > This seems inordinately slow for Ferret.  Especially for a file that is
> > only 1.5 Mb!  When this happens it usually means that the time axis is
> > marked as 'irregular' which means that Ferret has to read in the entire
> > time axis before it can begin.  We have found many cases where the axis
> > was actually 'regular' in spite of what the NetCDF attribute says.  If
> > this is the case then you can use ncatted to change the time axis
> > attribute and Ferret should open this dataset much more quickly.
> >
> >
> > -- Jon
> >
> >
> > J-M Epitalon wrote:
> >
> > >hello,
> > >
> > >I have daily simulation output covering the period 01-01-1950 to
> > >31-12-2005. It is in a NetCDF file that is 1.5 Mb large.
> > >
> > >When I access it through Ferret, it takes 2 to 4 mn to open it (execute
> > >instruction "use").
> > >
> > >I read in NCO documentation that "Some random access operations on large
> > >files on certain architecture are slow when using the NetCDF interface".
> > >
> > >That seems to be the problem. I could check that it is slow also with
> > >other tools than ferret (Python, or NCO tools).
> > >
> > >Anyway, what solution would you suggest to work around this ?
> > >Is using MC descriptors a good idea?
> > >
> > >Thanks
> > >
> > >Jean-Marie Epitalon
> > >CERFACS
> > >Toulouse, France
> > >
> > >
> > >
> >
> >