Hi All,
Joe Wielgosz wrote:
Hi folks,
A little metadata question for you: based on the information here:
http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html
in particular, the following:
The NOAA cooperative standard does not endorse any particular
interpretation of the distinction between missing_value and >
_FillValue.
it seems that COARDS-compliant client applications should recognize
both the '_FillValue' and 'missing_value' attributes. However, it
sounds like from what Luther is saying as if LAS only handles
'missing_value'. Is that correct?
This is not correct. The default back-end application to LAS, Ferret,
complies
to the COARDS convention and recognizes either 'missing_value' or
'_FillValue'
or both variable attributes.
from
http://ferret.wrc.noaa.gov/Ferret/Documentation/Users_Guide/current/
fer_html.htm
[...]
The _FillValue attribute informs Ferret that any data value matching
this value
is a missing (invalid) data point. For
example, an ocean data set may label land locations with a value such
as 1E34.
By identifying 1E34 as a fill value, Ferret
knows to ignore points matching this value. The attribute
"missing_value" is
similar to "_FillValue" when reading data;
but "_FillValue" also specifies a value to be inserted into
unspecified regions
during file creation. You may specify two
distinct flags for invalid data in the same variable by using
"_FillValue" and
"missing_value" together.
[...]
I have a memory of reading at some point that missing_value was being
deprecated in favor of _FillValue, which is why I chose to use the
latter form in the GDS metadata. However I can't find that page now.
Also, it seemed that _FillValue could be used to do certain
optimizations:
_FillValue - If a scalar attribute with this name is defined for a
variable and is of the same type as the variable, it will be
subsequently used as the fill value for that variable. The purpose of
this attribute is to save the applications programmer the work of
prefilling the data and also to eliminate the duplicate writes that
result from netCDF filling in missing data with its default fill
value, only to be immediately overwritten by the programmer's
preferred value. This value is considered to be a special value that
indicates missing data, and is returned when reading values that were
not written. The missing value should be outside the range specified
by valid_range (if used) for a variable. It is not necessary to
define
your own _FillValue attribute for a variable if the default fill
value
for the type of the variable is adequate.
Now, re-reading this carefully, it seems like the optimizations may
only apply when writing to a dataset. If so, then, for read-only
OPeNDAP datasets it is pretty much immaterial whether "missing_value"
or "_FillValue" is used.
This is my interpretation also.
So, can someone who is familiar with COARDS confirm whether GDS should
be using missing_value, or LAS should be supporting _FillValue, or
both?
LAS (actually Ferret) supports missing_value and _FillValue. I
suggest GDS
support both.
- Joe
On Wednesday, September 3, 2003, at 12:05 PM, Yudong Tian wrote:
Hi Jennifer,
When we added our GDS datasets to the LAS server, we found a
weird thing: the LAS server treats the "UNDEF 9.999E20" as real
data values, so the scale of the plots is screwed up. You can see
this by looking at the scale bar of any generated plots:
http://lis1.sci.gsfc.nasa.gov:8080/las/servlets/dataset
I was not able to access any data at the above LAS.
The info link listed the DODS URL for data set: 'Noah-GDAS 1/4 deg,
from LIS
2.0, milestone I' as
http://x1:9090/dods/OUTPUT/milstoneI/clm.geos.25/clm-geos0.25
my browser cannot find the server x1:9090. I would like to see the
DODS DAS or
a NetCDF cdl for this data.
The below error received when trying to get a shade plot tells me that
the LAS
Ferret is trying to use the _FillValue attribute values for the
variables
mentioned, but cannot read them. My guess is that the DAS will show
the
variable Type as REAL or FLOAT, but the _FillValue value, "UNDEF
9.999E20" is
not REAL or FLOAT because of the "UNDEF" string included.
dset: Noah-GEOS 1/4 deg, from LIS 2.0, milestone I
var: 14 temperature of bare soil k
yes? go std_initialize
"http://x1:9090/dods/OUTPUT/milstoneI/noah.geos.25/noah-geos0.25"; "1"
"1"
"baresoilt"
*** NOTE: too many values in attribute "_FillValue" in netCDF file
variable:
snowf
*** NOTE: too many values in attribute "_FillValue" in netCDF file
variable:
rainf
*** NOTE: too many values in attribute "_FillValue" in netCDF file
variable:
soilmoist
*** NOTE: too many values in attribute "_FillValue" in netCDF file
variable:
soilmoist
*** NOTE: too many values in attribute "_FillValue" in netCDF file
variable:
soilmoist
*** NOTE: too many values in attribute "_FillValue" in netCDF file
variable:
soilmoist
*** NOTE: too many values in attribute "_FillValue" in netCDF file
variable:
rainf
*** NOTE: too many values in attribute "_FillValue" in netCDF file
variable:
snowf
It seems to us that LAS does not recognize the "_FillValue"
tag in the .das files. The LAS people asked us to check out this
page:
http://ferret.pmel.noaa.gov/Ferret/LAS/FAQ/ingesting_data.htm
where they used the "missing_value" tag. I do not know if it
is relevant or not.
I will get the documentation changed to list '_FillValue' as a valid
attribute
also.
Could you please take a look?
Thanks.
Yudong
Joe Wielgosz / joew@cola.iges.org
---------------------------------------------------
Center for Ocean-Land-Atmosphere Studies (COLA)
Institute for Global Environment and Society (IGES)
http://www.iges.org
Joe McLean