Re: Big data file and limit on storage for coordinates

[Ansley Manke <ansley.b.manke@xxxxxxxx>]

Hi Mike,
It's the irregular time axis that's causing this message.  For all the 
irregular axes that are in use, Ferret stores all the axis coordinates, 
plus the location of the boundaries between the coordinates.  You're 
running into the limit on that storage. Sometimes because of the way a 
dataset is created, it has an irregular time axis, even though the time 
interval is regular -- if say,  missing data is represented by a gap in 
the time coordinates rather than a regular time axis with missing data 
values. If you can rework your data to be stored on a regularly-spaced 
time axis, that would do the trick.

What Ferret does with this data is to create a regularly-spaced abstract 
L axis, so in fact you can use the data, perhaps using Ferret to change 
the time axis.  When I open the URL you have in  your message, I see the 
error message you quote, but Ferret does open the dataset.  It creates a 
dependent variable out of the time coordinates, called time_hr, and it 
puts the file variables on the abstract L axis of the same length.  Here 
are some ways to look at the data. (the output of the commands is below)

 use 
"http://dods.mbari.org/cgi-bin/nph-nc/data/OASISdata/netcdf/hourlyM1.nc";
 show data
 sh grid AIR_TEMPERATURE_HR

 ! Look at the time axis of a variable; it is an abstract L axis
 list/l=1:5 t[gt=AIR_TEMPERATURE_HR]

 ! Look at the time coordinates in variable time_hr, and delta-time
 list/l=1:5/format=(2f15.1) time_hr, time_hr[L=@ddf]


yes? use 
"http://dods.mbari.org/cgi-bin/nph-nc/data/OASISdata/netcdf/hourlyM1.nc";
**TMAP ERR: limit on storage for coordinates has been reached
            MAX= 250000
...
yes? sho data
    currently SET data sets:
   1> 
http://dods.mbari.org/cgi-bin/nph-nc/data/OASISdata/netcdf/hourlyM1.nc  
(default)
name     title                             I         J         K         L
TIME_HR                                   ...       ...       ...       
1:132981
      (invalid coordinate axis)
AIR_TEMPERATURE_HR
         Hourly air_temperature           1:1       1:1       1:1       
1:132981
RELATIVE_HUMIDITY_HR
         Hourly Relative Humidity         1:1       1:1       1:1       
1:132981
...

yes? show grid AIR_TEMPERATURE_HR
   GRID GLR2
name       axis              # pts   start                end
LONGITUDE_MET_HR LONGITUD      1mr   122.03W              122.03W
LATITUDE_MET_HR LATITUDE       1 r   36.75N               36.75N
DEPTH_MET_HR DEPTH (m)         1 r-  3.5                  3.5
TIME_HR   T               132981 r   1                    132981

yes?   ! Look at the time axis of a variable; it is an abstract L axis
yes? list/l=1:5 t[gt=AIR_TEMPERATURE_HR]
            VARIABLE : T
                       axis TIME_HR
            DATA SET : Hourly Gridded OASIS Mooring M1 Observations
            FILENAME : hourlyM1.nc
            FILEPATH : 
http://dods.mbari.org/cgi-bin/nph-nc/data/OASISdata/netcdf/
            SUBSET   : 5 points (T)
1   / 1:  1.000
2   / 2:  2.000
3   / 3:  3.000
4   / 4:  4.000
5   / 5:  5.000

yes? ! Look at the time coordinates in variable time_hr, and delta-time
yes? list/l=1:5/format=(2f15.1) time_hr, time_hr[L=@ddf]
            DATA SET: 
http://dods.mbari.org/cgi-bin/nph-nc/data/OASISdata/netcdf/hourlyM1.nc
            Hourly Gridded OASIS Mooring M1 Observations
            T: 0.5 to 5.5
Column  1: TIME_HR is TIME_HR (SECONDS since 1970-01-01 
00:00:00)(invalid coordinate axis)
Column  2: TIME_HR[T=@DDF] is TIME_HR (SECONDS since 1970-01-01 
00:00:00)(invalid coordinate axis - forward derivative on T)
   619660800.0         3584.0
   619664384.0         3584.0
   619667968.0         3648.0
   619671616.0         3584.0
   619675200.0         3584.0

The other potential pitfall I see with this data is that Ferret uses 
single precision to store  data variables.  (It uses double precision 
for coordinates, but here we have changed from coordinates to a single 
precision variable.)   You might want to use a time origin closer to the 
time range of the data, or use units other than seconds.


Ansley

McCann, Mike wrote:

> Hello,
>
> I have a data set of mooring data that I have had difficulty creating for Ferret ever since we recovered our last deployment.  I append each deployment after gridding to an hourly time axis.  There are 2 depth axes: a 0:5:500 for the adcp data (DEPTH_HR) and a 12 element irregular axis for the microcat data (DEPTH_TS_HR).  The time series is now 15 years long and I'm wondering if I am now bumping into a limit inside of Ferret.
>
> I get this error when I try and load into Ferret:
>
> yes? use "http://dods.mbari.org/cgi-bin/nph-nc/data/OASISdata/netcdf/hourlyM1.nc";
> **TMAP ERR: limit on storage for coordinates has been reached
>             MAX= 250000
>
> I am able to plot the data using ncBrowse, so I think the data set is O.K. (e.g. no dis-ordered time dimension).  I am planning to use an aggregation server to simplify the generation of this data set, but am concerned that even if I do that Ferret would still not be able to use the data.  Have I reached a reached a limit?  Does anyone have any suggestions?
>
> Thanks in advance,
> Mike
>
> --
> Mike McCann 
> Software Engineer
> Monterey Bay Aquarium Research Institute
> 7700 Sandholdt Road
> Moss Landing, CA 95039-9644
> Voice: 831.775.1769  Fax: 831.775.1736 http://www.mbari.org
>
>