[Thread Prev][Thread Next][Index]

Re: [ferret_users] missing value not detected in double-precision ferret



Dear Ansley,

thanks for picking up on that so quickly.

Actually, I'd made a blunder: in the file 'ex_float_missing' I tried exactly what you suggested now, but declared the attribute as 'missing' instead of 'missing_value'. I corrected this now, and it doesn't solve the issue. By contrast, adding a '_FillValue' works - good point.

I am curious as to what Patrick has to say about this. He's the main model maintainer and has a better overview over all the projects distributed over several institutes.

He or I will get back to you directly or via the list.

Best wishes and good night (over here),
Hella



On 11/06/13 00:55, Ansley Manke wrote:
Hi Hella,
I don't have a complete answer, but here is what I know, and one
possible solution. We can continue to work with this issue and see if we
can come up with a satisfactory answer.

Is it the case that your files do not have missing_value or _FillValue
attributes?  The CF standard states that if there is a value to be
treated as the missing value, then it should be listed in a _FillValue
attribute.
http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/ch02s05.html#missing-data

For these variables without missing-data attributes, Ferret just assigns
its default bad-value flag value to the variable. When you do this

     yes? use ex_float.nc
     yes? list press
     yes? list `press,return=bad`
       !-> MESSAGE/CONTINUE -1.E+34
     -1.E+34

we are listing out the default bad-value that Ferret uses for all
variables, if none has been assigned by the dataset or by other Ferret
commands.  It is not getting that information from the dataset. If the
internal representation of the data as it is read in, exactly matches
the default bad-value flag as stored internally within Ferret, then the
data would be treated as missing.  This apparently did happen case with
single-precision Ferret.

The NetCDF library gives us calls to read data and attributes  from
their floating-point file values and load them into double-precision
internal storage.  Likewise we can make calls to write data in other
data types than the internal double-precision representation. If we let
the netCDF library do all of the the conversion between float and
double, we can have a netCDF file with floating point data and a
missing-flag of -1.e34 that is recognized,

     yes? use ex_float.nc

     ! Define a new variable that includes missing data, write it out as
     a floating-point variable:

     yes? let pvals = {1.e4,,1.e5}
     yes? let/title=pressure/units=pa pressure = reshape(pvals, press[d=1])

     yes? save/file=test.nc/outtype=float/clobber pressure
       LISTing to file test.nc

     yes? can var/all
     yes? use test.nc
     yes? list pressure
                   VARIABLE : pressure (pa)
                   FILENAME : test.nc
                   SUBSET   : 3 points (TIME)
       01-JAN-2000 00:20:00 / 1:   10000.
       01-JAN-2000 00:40:00 / 2:     ....
       01-JAN-2000 01:00:00 / 3:  100000.

So the netCDF library reading and writing data is consistent within
itself.  It does not seem to work to try to assign the missing_value
attribute to the variable after the file is opened, because we really
need the netCDF library to do the conversions in a way that is self
consistent.

Would it be an option for your group to use NCO to add a _FillValue
attribute to your datasets?  Have a look at this Ferret session.  I made
a copy of your file ex_float.nc,

     yes? use copy_of_ex_float.nc
     yes? sh att press
           attributes for dataset: ./copy_of_ex_float.nc
       press.long_name = pressure
       press.units = Pa

     yes? can dat/all
     yes? let status = nco_attr("copy_of_ex_float.nc", "press",
     "_FillValue", "f", "a", "-1.e+34")
     yes? load status

     yes? list status
                   VARIABLE : NCO_ATTR("copy_of_ex_float.nc", "press",
     "_FillValue", "f", "a", "-1.e+34")
                   X        : 1
                1.000

     yes? ! Now it has the _FillValue attribute

     yes? use copy_of_ex_float.nc
     yes? sh att press
           attributes for dataset: ./copy_of_ex_float.nc
       press.long_name = pressure
       press.units = Pa
       press._FillValue = -1.E+34

     yes? list press
                   VARIABLE : pressure (Pa)
                   FILENAME : copy_of_ex_float.nc
                   SUBSET   : 3 points (TIME)
       01-JAN-2000 00:20:00 / 1:   10000.
       01-JAN-2000 00:40:00 / 2:     ....
       01-JAN-2000 01:00:00 / 3:  100020.


Of course you can call NCO from the Unix command line.

On 6/10/2013 2:13 PM, Hella Riede wrote:
Dear ferret list,

there is a problem reading our climate model netcdf files with the new
ferret double precision version. I boiled it down to a set of small
files in the attachment.

'-1E+34' is not recognized as 'missing value' (see 'ex_float'). To save
space, all variables except dimensional variables in our climate model
output are defined as "float" = "single precision".

When a netcdf file is read into double-precision ferret, the entry
'-1E+34' in a float (single precision) variable is not recognized as
'missing value' anymore, but has a numerical value of
-9.999999790214768E+33 (prec=16). The bad values are thus not masked,
which is inconvenient for plotting or analysis.

yes? list press
                 VARIABLE : pressure (Pa)
                 FILENAME : ex_float.nc
                 SUBSET   : 3 points (TIME)
     01-JAN-2000 00:20:00 / 1:  1.000E+04
     01-JAN-2000 00:40:00 / 2: -1.000E+34
     01-JAN-2000 01:00:00 / 3:  1.000E+05

The workaround 'set var/bad=-9.999999790214768E+33
<variable>' is inconvenient as well.

Interestingly, when listing the bad value of the variable, the bad-value
masking seems to work:

yes? list `press,ret=bad`
     !-> list -1.E+34
                 VARIABLE : constant
            ....

In 'ex_double' (attachment), I exchanged 'float' to 'double' by hand for
all variables. The value '-1E+34' is then recognized correctly:

yes? list press
                 VARIABLE : pressure (Pa)
                 FILENAME : ex_traj2.nc
                 SUBSET   : 3 points (TIME)
     01-JAN-2000 00:20:00 / 1:   10000.
     01-JAN-2000 00:40:00 / 2:     ....
     01-JAN-2000 01:00:00 / 3:  100020.

The explicit definition of a missing value in the netCDF file does not
solve the issue (attachment: ex_float_missing).

Please help us with backwards compatibility as we cannot go into several
terabyte of climate data output to change "float" to "double" in the
variable definitions.


Thanks a lot in advance,
Hella


[Thread Prev][Thread Next][Index]
Contact Us
Dept of Commerce / NOAA / OAR / PMEL / Ferret

Privacy Policy | Disclaimer | Accessibility Statement