Hi Hella,
I don't have a complete answer, but here is what I know, and one
possible solution. We can continue to work with this issue and see if we
can come up with a satisfactory answer.
Is it the case that your files do not have missing_value or _FillValue
attributes? The CF standard states that if there is a value to be
treated as the missing value, then it should be listed in a _FillValue
attribute.
http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/ch02s05.html#missing-data
For these variables without missing-data attributes, Ferret just assigns
its default bad-value flag value to the variable. When you do this
yes? use ex_float.nc
yes? list press
yes? list `press,return=bad`
!-> MESSAGE/CONTINUE -1.E+34
-1.E+34
we are listing out the default bad-value that Ferret uses for all
variables, if none has been assigned by the dataset or by other Ferret
commands. It is not getting that information from the dataset. If the
internal representation of the data as it is read in, exactly matches
the default bad-value flag as stored internally within Ferret, then the
data would be treated as missing. This apparently did happen case with
single-precision Ferret.
The NetCDF library gives us calls to read data and attributes from
their floating-point file values and load them into double-precision
internal storage. Likewise we can make calls to write data in other
data types than the internal double-precision representation. If we let
the netCDF library do all of the the conversion between float and
double, we can have a netCDF file with floating point data and a
missing-flag of -1.e34 that is recognized,
yes? use ex_float.nc
! Define a new variable that includes missing data, write it out as
a floating-point variable:
yes? let pvals = {1.e4,,1.e5}
yes? let/title=pressure/units=pa pressure = reshape(pvals, press[d=1])
yes? save/file=test.nc/outtype=float/clobber pressure
LISTing to file test.nc
yes? can var/all
yes? use test.nc
yes? list pressure
VARIABLE : pressure (pa)
FILENAME : test.nc
SUBSET : 3 points (TIME)
01-JAN-2000 00:20:00 / 1: 10000.
01-JAN-2000 00:40:00 / 2: ....
01-JAN-2000 01:00:00 / 3: 100000.
So the netCDF library reading and writing data is consistent within
itself. It does not seem to work to try to assign the missing_value
attribute to the variable after the file is opened, because we really
need the netCDF library to do the conversions in a way that is self
consistent.
Would it be an option for your group to use NCO to add a _FillValue
attribute to your datasets? Have a look at this Ferret session. I made
a copy of your file ex_float.nc,
yes? use copy_of_ex_float.nc
yes? sh att press
attributes for dataset: ./copy_of_ex_float.nc
press.long_name = pressure
press.units = Pa
yes? can dat/all
yes? let status = nco_attr("copy_of_ex_float.nc", "press",
"_FillValue", "f", "a", "-1.e+34")
yes? load status
yes? list status
VARIABLE : NCO_ATTR("copy_of_ex_float.nc", "press",
"_FillValue", "f", "a", "-1.e+34")
X : 1
1.000
yes? ! Now it has the _FillValue attribute
yes? use copy_of_ex_float.nc
yes? sh att press
attributes for dataset: ./copy_of_ex_float.nc
press.long_name = pressure
press.units = Pa
press._FillValue = -1.E+34
yes? list press
VARIABLE : pressure (Pa)
FILENAME : copy_of_ex_float.nc
SUBSET : 3 points (TIME)
01-JAN-2000 00:20:00 / 1: 10000.
01-JAN-2000 00:40:00 / 2: ....
01-JAN-2000 01:00:00 / 3: 100020.
Of course you can call NCO from the Unix command line.
On 6/10/2013 2:13 PM, Hella Riede wrote:
Dear ferret list,
there is a problem reading our climate model netcdf files with the new
ferret double precision version. I boiled it down to a set of small
files in the attachment.
'-1E+34' is not recognized as 'missing value' (see 'ex_float'). To save
space, all variables except dimensional variables in our climate model
output are defined as "float" = "single precision".
When a netcdf file is read into double-precision ferret, the entry
'-1E+34' in a float (single precision) variable is not recognized as
'missing value' anymore, but has a numerical value of
-9.999999790214768E+33 (prec=16). The bad values are thus not masked,
which is inconvenient for plotting or analysis.
yes? list press
VARIABLE : pressure (Pa)
FILENAME : ex_float.nc
SUBSET : 3 points (TIME)
01-JAN-2000 00:20:00 / 1: 1.000E+04
01-JAN-2000 00:40:00 / 2: -1.000E+34
01-JAN-2000 01:00:00 / 3: 1.000E+05
The workaround 'set var/bad=-9.999999790214768E+33
<variable>' is inconvenient as well.
Interestingly, when listing the bad value of the variable, the bad-value
masking seems to work:
yes? list `press,ret=bad`
!-> list -1.E+34
VARIABLE : constant
....
In 'ex_double' (attachment), I exchanged 'float' to 'double' by hand for
all variables. The value '-1E+34' is then recognized correctly:
yes? list press
VARIABLE : pressure (Pa)
FILENAME : ex_traj2.nc
SUBSET : 3 points (TIME)
01-JAN-2000 00:20:00 / 1: 10000.
01-JAN-2000 00:40:00 / 2: ....
01-JAN-2000 01:00:00 / 3: 100020.
The explicit definition of a missing value in the netCDF file does not
solve the issue (attachment: ex_float_missing).
Please help us with backwards compatibility as we cannot go into several
terabyte of climate data output to change "float" to "double" in the
variable definitions.
Thanks a lot in advance,
Hella