Re: missing_data handling in ferret

[Joe Wielgosz <joew@xxxxxxxxxxxxx>]

Hi,

In that case, I should ask if you are aware of a quirk of the 
OPeNDAP/DODS-netCDF interface...

In the OPeNDAP protocol, the missing value is encoded as ASCII text, 
while the data values are transmitted as binary values.  Unfortunately, 
as far as I can find, there is no precise standard for representing 
binary floating point values as base-10 ASCII character strings. 
Therefore, there is always a possibility that client and server 
machines may use different algorithms to perform the conversion.

The result if this is the case, is that the binary value of the 
missing_data attribute that is read by the DODS-netCDF client may not 
be bitwise identical to the original binary value on the server. Thus, 
a strict equality comparison may not always correctly detect missing 
values in a DODS-netCDF data variable.

We got bitten by this because GrADS, which dates from a time when the C 
standard routine strtod() was not available on all systems, has its own 
ASCII <==> float conversion algorithm that in a few cases produces very 
slightly different results.

One way we plan to remedy this situation is to have GrADS use strtod() 
when possible, which hopefully will eliminate many of the 
discrepancies. However, in general AFAIK, there is no guarantee that 
strtod() and printf("%f") will produce exactly interchangeable and 
reversible results on all combinations of client and server platforms. 
So it seems the only really safe solution is to implement "fuzzy" 
missing_data checking for DODS data  (something GrADS already does for 
local binary datasets), using inequalities and a delta.

Just thought you might want to be aware of this and consider 
implementing a similar "fuzzy" check in Ferret, particularly since this 
is issue likely to show up when setting up LAS in front of GDS.

I'm cc'ing this to dods-tech as well, to bring it to the attention of 
the OPeNDAP folks if it hasn't been already...

- Joe


On Tuesday, September 23, 2003, at 05:29 PM, Ansley Manke wrote:

> Hi Joe,
> Within Ferret it's generally a strict comparison of the two floating
> point numbers, data-value and missing-flag, a Fortran equality test
> using .EQ.
>
> Ansley Manke
>
>
> Joe Wielgosz wrote:
>
> > Hi folks,
> >
> > I am working on this missing_data-not-recognized problem, which seems
> > to affect both LAS/Ferret and GrADS clients accessing the GDS.
> >
> > I was just wondering, what algorithm does Ferret use to check for
> > missing data when plotting? Does it expect strict binary equality
> > between the netCDF missing data value and the variable data, or is it 
> > a
> > "fuzzy" test with inequalities?
> >
> > Thanks,
> >
> > Joe
> >
> > Joe Wielgosz / joew@cola.iges.org
> > ---------------------------------------------------
> > Center for Ocean-Land-Atmosphere Studies (COLA)
> > Institute for Global Environment and Society (IGES)
> > http://www.iges.org
> --
> Ansley Manke  Pacific Marine Environmental Laboratory  Seattle WA
Joe Wielgosz / joew@cola.iges.org
---------------------------------------------------
Center for Ocean-Land-Atmosphere Studies (COLA)
Institute for Global Environment and Society (IGES)
http://www.iges.org