[Thread Prev][Thread Next][Index]
Re: How to compress NetCDF float variables to 2-byte integer usingFerret 5.80?
Hi Gus,
I'll take some time for you to get a feel for how Ferret works, but you
sound like you're well on your way. The manual is a large document, and
it can be hard to find what you need. The Users List is a good resource,
and searching the archives can sometimes turn up useful answers too.
We've had requests for the ability to work with other data types - there
are those of course who would like to do their analysis in double
precision, which would be a larger change than an integer output.
Perhaps writing 2-byte or other data types is a capability could be
added as a Ferret external function.
Ansley
Gus Correa wrote:
Hello Ansley Manke and Ferret users list
Thank you, Ansley, for your help.
Your clarification on the concept and use of the "immediate mode" of a
variable (the value stored on a variable),
by contrast to the variable itself (a reference or pointer to a
structure perhaps?),
was very insightful, and helped me understand how Ferret works.
It looks like that I can't use Ferret for the 2-byte integer
compression of NetCDF files, as I intended to do.
I'll have to get around with some scripting using NCO tools,
or the hard way, writing a Fortran or C program using the NetCDF
libraries directly.
I was trying to minimize the number of programs I need to use to
post-process climate model output.
Since I need to use Ferret to do more complicated things (e.g. convert
from sigma level to pressure coordinates),
I thought it might be possible to use it to do the compression as well.
Ferret has all the right ingredients for this, particularly the easy
way to find the global max and min of a dataset.
The only missing feature is the possibility to output NetCDF data in
different numeric formats.
Would it be appropriate to request this feature for a future release
of Ferret?
(I.e, flexible numeric output format of NetCDF files, allowing floats,
double, short, etc).
Or is it far from the goals of Ferret?
Thanks again.
Gus Correa
Ansley Manke wrote:
Hi Gus,
Welcome to Ferret. (just a note that when you send questions about
using Ferret they should go just to ferret_users; contact_ferret is
more for system-related things, installation questions, or reporting
crashes or bugs and so on). You've written a great question, showing
what you're trying to do, what happens when you try and what you need
to know. It makes it much easier to answer!
First off, Ferret writes data to netCDF files only in the float
datatype, so I'm not sure this whole exercise will save you much on
the size of the output file.
The apparent discrepancy you see in the documentation between SET
VAR/SCALE and SET VAR/SCALEFAC is due to the fact that Ferret
commands and qualifiers can always be abbreviated to 4 characters
(or fewer), so these two specifications are the same. See "commands,
command syntax" in the Users Guide index for a list of notes about
commands syntax.
SET VAR/SCALE and SET VAR/OFFSET are only for changing the scale
and/or offset values which will be used when Ferret reads data from a
netCDF file. Ferret does not write the scale factor or add_offset
attributes when it writes a netCDF File; it writes the variable
after applying all scaling. The RETURN keywords nc_offset and
nc_scale are ways to access the values of these attributes as they
appear in a file that you are reading, and RETURN keywords
user_offset and user_scale are ways to get the value of any settings
the user has made. These keywords are not things you can set, but
just names for things you can get back from Ferret.
The syntax errors you're getting are because you want to use the
variable my_scale as a scalar in the SET VAR/SCALE command. When the
command is being parsed, Ferret does not generally evaluate the value
of expressions. To ensure that the value is viewed as a scalar when
the command is being parsed, enclose it in grave accent marks which
forces the expression my_scale to be evaluated. This point is called
"immediate mode" expressions. (We can add an example to the Users
Guide to show this).
yes? let my_scale = intrange/(my_max - my_min)
yes? set var/scale=`my_scale` my_new_var
-Ansley
Gus Correa wrote:
Hello Ferret support person and users
I am a new Ferret user.
I would like to use Ferret 5.80 to achieve a (modest) compression of
NetCDF files,
by doing this pretty standard procedure:
(1) Read the floating point or double precision variables from
NetCDF files;
(2) scale them (i.e. apply the appropriate scale and offset
factors) to fit the range of two-byte integers;
(3) convert them to two-byte integer variables, and
(4) write them to (hopefully smaller sized) NetCDF files.
Step (1) is just the Ferret command "use".
Step (2) goes more or less like this:
let my_max=my_var{x=@max,y=@max,z=@max,t=@max]
let my_min=my_var{x=@min,y=@min,z=@min,t=@min]
let intrange=65534.0
let new_missing_value=-32768
let my_scale=intrange/(my_max - my_min)
let my_off=(my_max + my_min) / 2.0
let my_new_var = my_scale * (my_var - my_off)
let my_new_var2=missing(my_new_var, my_new_missing)
Regarding steps (3) and (4) I've got really confused and stuck.
The Ferret 5.80 manual suggests using:
set var/scale=my_scale my_new_var
to set the scale attribute of the scaled variable, and similar for
the offset.
However, show commands contains a different syntax:
show commands set/variable
...
SET VARIABLE/TITLE/UNIT/GRID/BAD/DATASET/NAME/SCALEFAC/OFFSET
(Note the "SCALEFAC" spelling.)
Moreover, when I try, no matter which syntax I use, all I get are
errors:
yes? set var/scalefac=myscale my_new_var
**ERROR: command syntax: scalefac=myscale
yes? set var/scale=myscale my_new_var
**ERROR: command syntax: scale=myscale
yes? set var/user_scale=myscale my_new_var
**ERROR: unknown command qualifier: user_scale=myscale
yes? set var/nc_scale=myscale my_new_var
**ERROR: unknown command qualifier: nc_scale=myscale
Whereas, the SAY command with RETURN clause seems to work,
with yet different keywords:
yes? say `my_new_var,return=nc_scale`
!-> MESSAGE/CONTINUE 1
1
yes? say `my_new_var,return=user_scale`
!-> MESSAGE/CONTINUE 1
1
yes? say `my_new_var,return=nc_offset`
!-> MESSAGE/CONTINUE 0
0
yes? say `my_new_var,return=user_offset`
!-> MESSAGE/CONTINUE 0
0
Questions:
1. How can I set the scale and offset of the new (scaled) variable?
2. How can I convert it from floating point or double to 2-byte
integers?
3. Is this really implemented in Ferret 5.80, i.e. there a way to do
what I want using Ferret 5.80?
4. Is there any special function to convert a variable from float to
two-byte integer?
5. Is there an example script of how to do all of this?
6. Would it be possible to clarify the meaning, purpose, and
differences of all these scale/offset keywords?
(SCALE, SCALEFAC, NC_SCALE, USER_SCALE, and similar for offset)
Thank you.
Gus Correa
[Thread Prev][Thread Next][Index]
Dept of Commerce /
NOAA /
OAR /
PMEL /
TMAP
Contact Us | Privacy Policy | Disclaimer | Accessibility Statement