Re: Large files access time

[J-M Epitalon <Jean-Marie.Epitalon@xxxxxxxxxx>]

Steve,

after further investigation, I noticed the following:

When using option use/regulart for accessing a 1.5Gb file,
access time is immediate when Ferret version 5.51 (for RedHat 7.1) is
used.

It is much longer (20 to 45 seconds depending on local or remote) when
Ferret 5.53 (for RedHat 9) and up (5.7 an 5.8 tested) is used.....

I will continue my investigations, testing Ferret 5.8 for RedHat 7.1

Please, if you have any idea about this, tell me.

Jean-Marie Epitalon

> Hi J-M,
> 
> The usual interpretation of this is disk latency:  the first time you open the file
> the contents are read from disk (30-45 sec).  The second time, the disk blocks are
> still in the Unix disk cache -- "opens immediately".
> 
> This implies that there is a substantial amount of disk seeking going on.  I'm not
> sure why ...   You could check in the netCDF email archives or post a message to the
> netCDF group. If you have a debug ("-g") version of Ferret available you can step
> through the netCDF open sections and probably can see exactly which netCDF call is
> taking so long ...
> 
>     - steve
> 
> J-M Epitalon wrote:
> 
> > Steve,
> >
> > thanks for your answer.
> > I notice that the option /regulart speeds up quite a bit.
> > On my personnal workstation, a Red-Hat PC running Ferret 5.51, the
> > instruction use/regulart executes instantaneously.
> >
> > However, on my LAS server, which is a quadriprocessor PowerEdge 2650
> > DELL PC with Red Hat Enterprise Linux, running Ferret 5.8, I cannot
> > achieve the same result:
> >
> > When opening a 1.5 Gb NetCDF file, it takes 20 to 30 seconds, even for a
> > local file!!!
> > For a NFS remote file, it takes 30 to 45s.
> >
> > Then, the second time I open it, whether in the same or another Ferret
> > session, it opens immediately.
> >
> > I tried different versions of Ferret for Linux Red-Hat 9: v5.53, 5.7 and
> > 5.8 and I cannot see any difference.
> > I suppose the problem lies in the PC or in the OS...
> >
> > Please, could you help me understand this ?
> >
> > Jean-Marie
> >
> > On Tue, 2005-05-03 at 05:24, Steve Hankin wrote:
> > > Hi Jean-Marie,
> > >
> > > Did you try
> > >     yes?  USE/regularT my_file.nc
> > >
> > > When that qualifier is used Ferret reads only the first and last time step of
> > > the time axis.  There's a long disk seek between the two time steps, since the
> > > file is large, but it would account for only a fraction of a second in the
> > > worst case.
> > >
> > > If you are using /regularT and still getting the slow speed then I cannot
> > > account for it.  Can you send us the output of ncdump -c  ?
> > >
> > >     - steve
> > >
> > >
> > > J-M Epitalon wrote:
> > >
> > > > Jon and Steve,
> > > >
> > > > sorry. I mislead you.
> > > > My file is 1.5 Gb (not Mb) !
> > > > and it is accessed through NFS.
> > > > I suppose this is why it is really slow.
> > > >
> > > > Anyway, when accessed localy, it still takes half a minute to open it.
> > > > (3 to 4 mn thru NFS)
> > > >
> > > > Since I want to keep the system architecture with NFS, I solved my
> > > > problem by splitting the file in several parts and using a MC file.
> > > >
> > > > Please, what would you suggest as an alternative solution ?
> > > >
> > > > Jean-Marie Epitalon
> > > > CERFACS
> > > > Toulouse, FR
> > > >
> > > > On Fri, 2005-04-22 at 18:27, Jonathan Callahan wrote:
> > > > > Jean-Marie,
> > > > >
> > > > > This seems inordinately slow for Ferret.  Especially for a file that is
> > > > > only 1.5 Mb!  When this happens it usually means that the time axis is
> > > > > marked as 'irregular' which means that Ferret has to read in the entire
> > > > > time axis before it can begin.  We have found many cases where the axis
> > > > > was actually 'regular' in spite of what the NetCDF attribute says.  If
> > > > > this is the case then you can use ncatted to change the time axis
> > > > > attribute and Ferret should open this dataset much more quickly.
> > > > >
> > > > >
> > > > > -- Jon
> > > > >
> > > > >
> > > > > J-M Epitalon wrote:
> > > > >
> > > > > >hello,
> > > > > >
> > > > > >I have daily simulation output covering the period 01-01-1950 to
> > > > > >31-12-2005. It is in a NetCDF file that is 1.5 Mb large.
> > > > > >
> > > > > >When I access it through Ferret, it takes 2 to 4 mn to open it (execute
> > > > > >instruction "use").
> > > > > >
> > > > > >I read in NCO documentation that "Some random access operations on large
> > > > > >files on certain architecture are slow when using the NetCDF interface".
> > > > > >
> > > > > >That seems to be the problem. I could check that it is slow also with
> > > > > >other tools than ferret (Python, or NCO tools).
> > > > > >
> > > > > >Anyway, what solution would you suggest to work around this ?
> > > > > >Is using MC descriptors a good idea?
> > > > > >
> > > > > >Thanks
> > > > > >
> > > > > >Jean-Marie Epitalon
> > > > > >CERFACS
> > > > > >Toulouse, France
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > >
> > >
> 
>