[Thread Prev][Thread Next][Index]

RE: Big data file and limit on storage for coordinates



Ansley,

Thank you for the explanation and suggestions.

To create this data set I first establish an hourly time axis that encompasses all the deployments, e.g.:  

 DEFINE AXIS/T="21-AUG-1989 00:00 ":"21-OCT-2004 20:00":3600/T0="01-JAN-1970 00:00:00"/UNITS=seconds TIME_HR

Then I process data from each deployment (apply QC flags, grid to hourly), and then append to the netcdf file, e.g. for our last deployment (some relevant commands):

 DEFINE AXIS/T="07-NOV-2003 20:00":"21-OCT-2004 20:00":600/T0="01-JAN-1970 00:00:00"/UNITS=seconds TIME_ORG
 DEFINE AXIS/T="07-NOV-2003 20:00":"21-OCT-2004 20:00":3600/T0="01-JAN-1970 00:00:00"/UNITS=seconds TIME_HR
 LET SALINITY_1 = missing(SALINITY_FLAG, -9999)
 LET SALINITY_1F = IF SALINITY_1 EQ 0 THEN 1
 LET SALINITY_2 = SALINITY * SALINITY_1F
 DEFINE VARIABLE SALINITY_ORG = SALINITY_2[gt=TIME_ORG@MAX,gz=DEPTH_TS_HR@LIN,gy=LATITUDE_TS_HR,gx=LONGITUDE_TS_HR]
 DEFINE VARIABLE SALINITY_HR = SALINITY_ORG[gt=TIME_HR@AVE,gz=DEPTH_TS_HR@LIN,gy=LATITUDE_TS_HR,gx=LONGITUDE_TS_HR]
 SET VAR/TITLE="Hourly Salinity"/UNITS="" SALINITY_HR
 LIST/FORMAT=CDF/FILE="hourlyM1.nc"/APPEND/T="07-NOV-2003 18:00":"21-OCT-2004 19:30" SALINITY_HR

Without digging further I suspect that a gap is created such that Ferret removes the 'point_spacing = "even"' attribute on TIME_HR.  Perhaps the 'T=...' option on the LIST command above may be confusing things (these time limits are the limits of the deployment data).  Perhaps it would be best to remove this as the SALINITY_HR variable is already on an hourly grid.  I will try removing it.

Anyway, I used the NCO command ncatted to add the 'point_spacing = "even"' attribute back to TIME_HR:

 ncatted -O -h -a point_spacing,TIME_HR,c,c,"even" hourlyM1.nc

and now Ferret can process the data in the file.

Thanks to everyone who responded.  It's good to know that with my data now properly metadata attributed I am nowhere near any limits within Ferret.

-Mike

> -----Original Message-----
> From: Ansley Manke [mailto:ansley.b.manke@noaa.gov]
> Sent: Wednesday, March 16, 2005 12:16 PM
> To: McCann, Mike
> Cc: ferret_users
> Subject: Re: Big data file and limit on storage for coordinates
> 
> 
> Hi Mike,
> It's the irregular time axis that's causing this message.  
> For all the 
> irregular axes that are in use, Ferret stores all the axis 
> coordinates, 
> plus the location of the boundaries between the coordinates.  You're 
> running into the limit on that storage. Sometimes because of 
> the way a 
> dataset is created, it has an irregular time axis, even 
> though the time 
> interval is regular -- if say,  missing data is represented 
> by a gap in 
> the time coordinates rather than a regular time axis with 
> missing data 
> values. If you can rework your data to be stored on a 
> regularly-spaced 
> time axis, that would do the trick.
> 
> What Ferret does with this data is to create a 
> regularly-spaced abstract 
> L axis, so in fact you can use the data, perhaps using Ferret 
> to change 
> the time axis.  When I open the URL you have in  your 
> message, I see the 
> error message you quote, but Ferret does open the dataset.  
> It creates a 
> dependent variable out of the time coordinates, called 
> time_hr, and it 
> puts the file variables on the abstract L axis of the same 
> length.  Here 
> are some ways to look at the data. (the output of the 
> commands is below)
> 
>   use 
> "http://dods.mbari.org/cgi-bin/nph-nc/data/OASISdata/netcdf/ho
> urlyM1.nc"
>   show data
>   sh grid AIR_TEMPERATURE_HR
> 
>   ! Look at the time axis of a variable; it is an abstract L axis
>   list/l=1:5 t[gt=AIR_TEMPERATURE_HR]
> 
>   ! Look at the time coordinates in variable time_hr, and delta-time
>   list/l=1:5/format=(2f15.1) time_hr, time_hr[L=@ddf]
> 
> 
> yes? use 
> "http://dods.mbari.org/cgi-bin/nph-nc/data/OASISdata/netcdf/ho
> urlyM1.nc"
>  **TMAP ERR: limit on storage for coordinates has been reached
>              MAX= 250000
> ...
> yes? sho data
>      currently SET data sets:
>     1> 
> http://dods.mbari.org/cgi-bin/nph-nc/data/OASISdata/netcdf/hou
> rlyM1.nc  
> (default)
>  name     title                             I         J       
>   K         L
>  TIME_HR                                   ...       ...      
>  ...       
> 1:132981
>        (invalid coordinate axis)
>  AIR_TEMPERATURE_HR
>           Hourly air_temperature           1:1       1:1      
>  1:1       
> 1:132981
>  RELATIVE_HUMIDITY_HR
>           Hourly Relative Humidity         1:1       1:1      
>  1:1       
> 1:132981
> ...
> 
> yes? show grid AIR_TEMPERATURE_HR
>     GRID GLR2
>  name       axis              # pts   start                end
>  LONGITUDE_MET_HR LONGITUD      1mr   122.03W              122.03W
>  LATITUDE_MET_HR LATITUDE       1 r   36.75N               36.75N
>  DEPTH_MET_HR DEPTH (m)         1 r-  3.5                  3.5
>  TIME_HR   T               132981 r   1                    132981
> 
> yes?   ! Look at the time axis of a variable; it is an abstract L axis
> yes? list/l=1:5 t[gt=AIR_TEMPERATURE_HR]
>              VARIABLE : T
>                         axis TIME_HR
>              DATA SET : Hourly Gridded OASIS Mooring M1 Observations
>              FILENAME : hourlyM1.nc
>              FILEPATH : 
> http://dods.mbari.org/cgi-bin/nph-nc/data/OASISdata/netcdf/
>              SUBSET   : 5 points (T)
>  1   / 1:  1.000
>  2   / 2:  2.000
>  3   / 3:  3.000
>  4   / 4:  4.000
>  5   / 5:  5.000
> 
> yes? ! Look at the time coordinates in variable time_hr, and 
> delta-time
> yes? list/l=1:5/format=(2f15.1) time_hr, time_hr[L=@ddf]
>              DATA SET: 
> http://dods.mbari.org/cgi-bin/nph-nc/data/OASISdata/netcdf/hourlyM1.nc
>              Hourly Gridded OASIS Mooring M1 Observations
>              T: 0.5 to 5.5
>  Column  1: TIME_HR is TIME_HR (SECONDS since 1970-01-01 
> 00:00:00)(invalid coordinate axis)
>  Column  2: TIME_HR[T=@DDF] is TIME_HR (SECONDS since 1970-01-01 
> 00:00:00)(invalid coordinate axis - forward derivative on T)
>     619660800.0         3584.0
>     619664384.0         3584.0
>     619667968.0         3648.0
>     619671616.0         3584.0
>     619675200.0         3584.0
> 
> The other potential pitfall I see with this data is that Ferret uses 
> single precision to store  data variables.  (It uses double precision 
> for coordinates, but here we have changed from coordinates to 
> a single 
> precision variable.)   You might want to use a time origin 
> closer to the 
> time range of the data, or use units other than seconds.
> 
> 
> Ansley
> 
> McCann, Mike wrote:
> 
> >Hello,
> >
> >I have a data set of mooring data that I have had difficulty 
> creating for Ferret ever since we recovered our last 
> deployment.  I append each deployment after gridding to an 
> hourly time axis.  There are 2 depth axes: a 0:5:500 for the 
> adcp data (DEPTH_HR) and a 12 element irregular axis for the 
> microcat data (DEPTH_TS_HR).  The time series is now 15 years 
> long and I'm wondering if I am now bumping into a limit 
> inside of Ferret.
> >
> >I get this error when I try and load into Ferret:
> >
> >yes? use 
"http://dods.mbari.org/cgi-bin/nph-nc/data/OASISdata/netcdf/hourlyM1.nc";
> **TMAP ERR: limit on storage for coordinates has been reached
>             MAX= 250000
>
>I am able to plot the data using ncBrowse, so I think the data set is O.K. (e.g. no dis-ordered time dimension).  I am planning to use an aggregation server to simplify the generation of this data set, but am concerned that even if I do that Ferret would still not be able to use the data.  Have I reached a reached a limit?  Does anyone have any suggestions?
>
>Thanks in advance,
>Mike
>
>--
>Mike McCann 
>Software Engineer
>Monterey Bay Aquarium Research Institute
>7700 Sandholdt Road
>Moss Landing, CA 95039-9644
>Voice: 831.775.1769  Fax: 831.775.1736 http://www.mbari.org
>
>  
>

[Thread Prev][Thread Next][Index]

Dept of Commerce / NOAA / OAR / PMEL / TMAP

Contact Us | Privacy Policy | Disclaimer | Accessibility Statement