Re: What is the maximum data size FERRET can handle?

[Steve Hankin <hankin@xxxxxxxxxxxxx>]

Hi Yang,

The error message indicates that this is not a Ferret limitation, per se,
but a netCDF (actually Posix) limitation.

As I understand it the normal limit on file size in netCDF is 2 Gbytes.  On
64 bit systems it is possible to rebuild the netCDF code with special flag
settings ("-ndebug" as I recall ... oddly named) and then *with
restrictions* to work with files in excess of 2 Gbytes.  The restrictions
have to do with the sizes of individual variables.

Note that if you take this approach you will be creating netCDF files that
are not compatible with *most* application programs that use netCDF.  You
would need to build a special version of Ferret using the modified netCDF
libraries.

Normally a much better solution is to divide your data up along its time
axis into multiple (smaller) netCDF files.  The "MC descriptor" framework
in Ferret allows you to work with multiple files as a single data set (see
users Guide: "Ch10 Sec4.CREATING A MULTI-FILE NETCDF DATA SET").

    - steve

==================

Yang Haijun wrote:

> Hi,
>
> I have a data with the size of about 2.8G. I met the following problem
> when I use FERRET to open it,
>
> yes? use ho.TSP.900959.mon.nc
> lnetcdf/lposixio.c:273: failed assertion `offset < X_INT_MAX'
> Abort (core dumped)
>
> Seems because the data is too big. Is there any way to handle big data?
> Thanks a lot.
>
> Haijun
>
>                                            ''~``
>                                           ( o o )
> +------------------------------------.oooO--(_)--Oooo.---+
> |Haijun Yang                                             |
> |Department of Atmospheric and Oceanic Sciences          |
> |University of Wisconsin-Madison                         |
> |1225 West Dayton Street                                 |
> |Madison, WI 53706                                       |
> |Phone: 1-608-262-1956(office)         .oooO             |
> |                                      (   )   Oooo.     |
> +---------------------------------------\ (----(   )-----+
>                                          \_)    ) /
>                                                (_/

--
Steve Hankin
NOAA/PMEL, 7600 Sand Point Way NE, Seattle, WA 98115-0070
ph. (206) 526-6080 -- FAX (206) 526-6744