Hi Hella,
I don't have a complete answer, but here is what I know, and one
possible solution. We can continue to work with this issue and see
if we can come up with a satisfactory answer.
Is it the case that your files do not have missing_value or
_FillValue attributes? The CF standard states that if there is a
value to be treated as the missing value, then it should be listed
in a _FillValue attribute.
http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/ch02s05.html#missing-data
For these variables without missing-data attributes, Ferret just
assigns its default bad-value flag value to the variable. When you
do this
yes? use ex_float.nc
yes? list press
yes? list `press,return=bad`
!-> MESSAGE/CONTINUE -1.E+34
-1.E+34
we are listing out the default bad-value that Ferret uses for all
variables, if none has been assigned by the dataset or by other
Ferret commands. It is not getting that information from the
dataset. If the internal representation of the data as it is read
in, exactly matches the default bad-value flag as stored internally
within Ferret, then the data would be treated as missing. This
apparently did happen case with single-precision Ferret.
The NetCDF library gives us calls to read data and attributes from
their floating-point file values and load them into double-precision
internal storage. Likewise we can make calls to write data in other
data types than the internal double-precision representation. If we
let the netCDF library do all of the the conversion between float
and double, we can have a netCDF file with floating point data and a
missing-flag of -1.e34 that is recognized,
yes? use ex_float.nc
! Define a new variable that includes missing data, write
it out as a floating-point variable:
yes? let pvals = {1.e4,,1.e5}
yes? let/title=pressure/units=pa pressure =
reshape(pvals, press[d=1])
yes? save/file=test.nc/outtype=float/clobber pressure
LISTing to file test.nc
yes? can var/all
yes? use test.nc
yes? list pressure
VARIABLE : pressure (pa)
FILENAME : test.nc
SUBSET : 3 points (TIME)
01-JAN-2000 00:20:00 / 1: 10000.
01-JAN-2000 00:40:00 / 2: ....
01-JAN-2000 01:00:00 / 3: 100000.
So the netCDF library reading and writing data is consistent within
itself. It does not seem to work to try to assign the missing_value
attribute to the variable after the file is opened, because we
really need the netCDF library to do the conversions in a way that
is self consistent.
Would it be an option for your group to use NCO to add a _FillValue
attribute to your datasets? Have a look at this Ferret session. I
made a copy of your file ex_float.nc,
yes? use copy_of_ex_float.nc
yes? sh att press
attributes for dataset: ./copy_of_ex_float.nc
press.long_name = pressure
press.units = Pa
yes? can dat/all
yes? let status = nco_attr("copy_of_ex_float.nc", "press",
"_FillValue", "f", "a", "-1.e+34")
yes? load status
yes? list status
VARIABLE : NCO_ATTR("copy_of_ex_float.nc",
"press", "_FillValue", "f", "a", "-1.e+34")
X : 1
1.000
yes? ! Now it has the _FillValue attribute
yes? use copy_of_ex_float.nc
yes? sh att press
attributes for dataset: ./copy_of_ex_float.nc
press.long_name = pressure
press.units = Pa
press._FillValue = -1.E+34
yes? list press
VARIABLE : pressure (Pa)
FILENAME : copy_of_ex_float.nc
SUBSET : 3 points (TIME)
01-JAN-2000 00:20:00 / 1: 10000.
01-JAN-2000 00:40:00 / 2: ....
01-JAN-2000 01:00:00 / 3: 100020.
Of course you can call NCO from the Unix command line.
On 6/10/2013 2:13 PM, Hella Riede
wrote:
Dear
ferret list,
there is a problem reading our climate model netcdf files with the
new
ferret double precision version. I boiled it down to a set of
small
files in the attachment.
'-1E+34' is not recognized as 'missing value' (see 'ex_float'). To
save
space, all variables except dimensional variables in our climate
model
output are defined as "float" = "single precision".
When a netcdf file is read into double-precision ferret, the entry
'-1E+34' in a float (single precision) variable is not recognized
as
'missing value' anymore, but has a numerical value of
-9.999999790214768E+33 (prec=16). The bad values are thus not
masked,
which is inconvenient for plotting or analysis.
yes? list press
VARIABLE : pressure (Pa)
FILENAME : ex_float.nc
SUBSET : 3 points (TIME)
01-JAN-2000 00:20:00 / 1: 1.000E+04
01-JAN-2000 00:40:00 / 2: -1.000E+34
01-JAN-2000 01:00:00 / 3: 1.000E+05
The workaround 'set var/bad=-9.999999790214768E+33
<variable>' is inconvenient as well.
Interestingly, when listing the bad value of the variable, the
bad-value
masking seems to work:
yes? list `press,ret=bad`
!-> list -1.E+34
VARIABLE : constant
....
In 'ex_double' (attachment), I exchanged 'float' to 'double' by
hand for
all variables. The value '-1E+34' is then recognized correctly:
yes? list press
VARIABLE : pressure (Pa)
FILENAME : ex_traj2.nc
SUBSET : 3 points (TIME)
01-JAN-2000 00:20:00 / 1: 10000.
01-JAN-2000 00:40:00 / 2: ....
01-JAN-2000 01:00:00 / 3: 100020.
The explicit definition of a missing value in the netCDF file does
not
solve the issue (attachment: ex_float_missing).
Please help us with backwards compatibility as we cannot go into
several
terabyte of climate data output to change "float" to "double" in
the
variable definitions.
Thanks a lot in advance,
Hella
|