Re: GDS - LAS interaction

[Joe McLean <joe.mclean@xxxxxxxx>]

Hi All,

Joe Wielgosz wrote:

> Hi folks,
>
> A little metadata question for you: based on the information here:
>         http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html
> in particular, the following:
>
> > The NOAA cooperative standard does not endorse any particular
> > interpretation of the distinction between missing_value and > _FillValue.
>
> it seems that COARDS-compliant client applications should recognize
> both the '_FillValue' and 'missing_value' attributes.  However, it
> sounds like from what Luther is saying as if LAS only handles
> 'missing_value'. Is that correct?
>

This is not correct.  The default back-end application to LAS, Ferret, complies
to the COARDS convention and recognizes either 'missing_value' or '_FillValue'
or both variable attributes.

from
http://ferret.wrc.noaa.gov/Ferret/Documentation/Users_Guide/current/fer_html.htm

[...]
The _FillValue attribute informs Ferret that any data value matching this value
is a missing (invalid) data point. For
example, an ocean data set may label land locations with a value such as 1E34.
By identifying 1E34 as a fill value, Ferret
knows to ignore points matching this  value.  The attribute "missing_value" is
similar to "_FillValue" when reading data;
but "_FillValue" also specifies a value to be inserted into unspecified regions
during file creation. You may specify two
distinct flags for invalid data in the same variable by using "_FillValue" and
"missing_value" together.
[...]

>
> I have a memory of reading at some point that missing_value was being
> deprecated in favor of _FillValue, which is why I chose to use the
> latter form in the GDS metadata. However I can't find that page now.
>
> Also, it seemed that _FillValue could be used to do certain
> optimizations:
>
> > _FillValue - If a scalar attribute with this name is defined for a
> > variable and is of the same type as the variable, it will be
> > subsequently used as the fill value for that variable. The purpose of
> > this attribute is to save the applications programmer the work of
> > prefilling the data and also to eliminate the duplicate writes that
> > result from netCDF filling in missing data with its default fill
> > value, only to be immediately overwritten by the programmer's
> > preferred value. This value is considered to be a special value that
> > indicates missing data, and is returned when reading values that were
> > not written. The missing value should be outside the range specified
> > by valid_range (if used) for a variable. It is not necessary to define
> > your own _FillValue attribute for a variable if the default fill value
> > for the type of the variable is adequate.
>
> Now, re-reading this carefully, it seems like the optimizations may
> only apply when writing to a dataset.   If so, then, for read-only
> OPeNDAP datasets it is pretty much immaterial whether "missing_value"
> or "_FillValue" is used.
>

This is my interpretation also.

>
> So, can someone who is familiar with COARDS confirm whether GDS should
> be using missing_value, or LAS should be supporting _FillValue, or both?
>

LAS (actually Ferret) supports missing_value and _FillValue.  I suggest GDS
support both.

>
> - Joe
>
> On Wednesday, September 3, 2003, at 12:05 PM, Yudong Tian wrote:
>
> > Hi Jennifer,
> >    When we added our GDS datasets to the LAS server, we found a
> > weird thing: the LAS server treats the "UNDEF 9.999E20" as real
> > data values, so the scale of the plots is screwed up. You can see
> > this by looking at the scale bar of any generated plots:
> >
> >
> > http://lis1.sci.gsfc.nasa.gov:8080/las/servlets/dataset
> >

I was not able to access any data at the above LAS.

The info link listed the DODS URL for data set: 'Noah-GDAS 1/4 deg, from LIS
2.0, milestone I' as
http://x1:9090/dods/OUTPUT/milstoneI/clm.geos.25/clm-geos0.25
my browser cannot find the server x1:9090.  I would like to see the DODS DAS or
a NetCDF cdl for this data.

The below error received when trying to get a shade plot tells me that the LAS
Ferret is trying to use the _FillValue attribute values for the variables
mentioned, but cannot read them.  My guess is that the DAS will show the
variable Type as REAL or FLOAT, but the _FillValue value, "UNDEF 9.999E20" is
not REAL or FLOAT because of the "UNDEF" string included.

dset: Noah-GEOS 1/4 deg, from LIS 2.0, milestone I
var: 14 temperature of bare soil k
yes? go std_initialize
"http://x1:9090/dods/OUTPUT/milstoneI/noah.geos.25/noah-geos0.25"; "1" "1"
"baresoilt"
 *** NOTE: too many values in attribute "_FillValue" in netCDF file variable:
snowf
 *** NOTE: too many values in attribute "_FillValue" in netCDF file variable:
rainf
 *** NOTE: too many values in attribute "_FillValue" in netCDF file variable:
soilmoist
 *** NOTE: too many values in attribute "_FillValue" in netCDF file variable:
soilmoist
 *** NOTE: too many values in attribute "_FillValue" in netCDF file variable:
soilmoist
 *** NOTE: too many values in attribute "_FillValue" in netCDF file variable:
soilmoist
 *** NOTE: too many values in attribute "_FillValue" in netCDF file variable:
rainf
 *** NOTE: too many values in attribute "_FillValue" in netCDF file variable:
snowf


>
> >   It seems to us that LAS does not recognize the "_FillValue"
> > tag in the .das files. The LAS people asked us to check out this
> > page:
> >   http://ferret.pmel.noaa.gov/Ferret/LAS/FAQ/ingesting_data.htm
> > where they  used the "missing_value" tag. I do not know if it
> > is relevant or not.

I will get the documentation changed to list '_FillValue' as a valid attribute
also.

>
> >   Could you please take a look?
> >
> > Thanks.
> > Yudong
> >
> >
> >
>
> Joe Wielgosz / joew@cola.iges.org
> ---------------------------------------------------
> Center for Ocean-Land-Atmosphere Studies (COLA)
> Institute for Global Environment and Society (IGES)
> http://www.iges.org

Joe McLean