Re: Large files access time

[J-M Epitalon <Jean-Marie.Epitalon@xxxxxxxxxx>]

Steve,

thanks for your answer.
I notice that the option /regulart speeds up quite a bit.
On my personnal workstation, a Red-Hat PC running Ferret 5.51, the
instruction use/regulart executes instantaneously.

However, on my LAS server, which is a quadriprocessor PowerEdge 2650
DELL PC with Red Hat Enterprise Linux, running Ferret 5.8, I cannot
achieve the same result:

When opening a 1.5 Gb NetCDF file, it takes 20 to 30 seconds, even for a
local file!!!
For a NFS remote file, it takes 30 to 45s.

Then, the second time I open it, whether in the same or another Ferret
session, it opens immediately.

I tried different versions of Ferret for Linux Red-Hat 9: v5.53, 5.7 and
5.8 and I cannot see any difference.
I suppose the problem lies in the PC or in the OS...

Please, could you help me understand this ?

Jean-Marie

On Tue, 2005-05-03 at 05:24, Steve Hankin wrote:
> Hi Jean-Marie,
> 
> Did you try
>     yes?  USE/regularT my_file.nc
> 
> When that qualifier is used Ferret reads only the first and last time step of
> the time axis.  There's a long disk seek between the two time steps, since the
> file is large, but it would account for only a fraction of a second in the
> worst case.
> 
> If you are using /regularT and still getting the slow speed then I cannot
> account for it.  Can you send us the output of ncdump -c  ?
> 
>     - steve
> 
> 
> J-M Epitalon wrote:
> 
> > Jon and Steve,
> >
> > sorry. I mislead you.
> > My file is 1.5 Gb (not Mb) !
> > and it is accessed through NFS.
> > I suppose this is why it is really slow.
> >
> > Anyway, when accessed localy, it still takes half a minute to open it.
> > (3 to 4 mn thru NFS)
> >
> > Since I want to keep the system architecture with NFS, I solved my
> > problem by splitting the file in several parts and using a MC file.
> >
> > Please, what would you suggest as an alternative solution ?
> >
> > Jean-Marie Epitalon
> > CERFACS
> > Toulouse, FR
> >
> > On Fri, 2005-04-22 at 18:27, Jonathan Callahan wrote:
> > > Jean-Marie,
> > >
> > > This seems inordinately slow for Ferret.  Especially for a file that is
> > > only 1.5 Mb!  When this happens it usually means that the time axis is
> > > marked as 'irregular' which means that Ferret has to read in the entire
> > > time axis before it can begin.  We have found many cases where the axis
> > > was actually 'regular' in spite of what the NetCDF attribute says.  If
> > > this is the case then you can use ncatted to change the time axis
> > > attribute and Ferret should open this dataset much more quickly.
> > >
> > >
> > > -- Jon
> > >
> > >
> > > J-M Epitalon wrote:
> > >
> > > >hello,
> > > >
> > > >I have daily simulation output covering the period 01-01-1950 to
> > > >31-12-2005. It is in a NetCDF file that is 1.5 Mb large.
> > > >
> > > >When I access it through Ferret, it takes 2 to 4 mn to open it (execute
> > > >instruction "use").
> > > >
> > > >I read in NCO documentation that "Some random access operations on large
> > > >files on certain architecture are slow when using the NetCDF interface".
> > > >
> > > >That seems to be the problem. I could check that it is slow also with
> > > >other tools than ferret (Python, or NCO tools).
> > > >
> > > >Anyway, what solution would you suggest to work around this ?
> > > >Is using MC descriptors a good idea?
> > > >
> > > >Thanks
> > > >
> > > >Jean-Marie Epitalon
> > > >CERFACS
> > > >Toulouse, France
> > > >
> > > >
> > > >
> > >
> > >
> 
>