Re: precision of ferret calculation

[Ansley Manke <ansley@xxxxxxxxxxxxx>]

Hi Dongxiao,
the `precision=#digits` controls the OUTPUT precision, that is to
say, the formatting, of numbers that are displayed in an expression.
It does not influence the precision of the calculation.

Ferret does its calculations in single-precision floating point
representation see the index entry "precision, of floating-point
variables" in the Users Guide.  This is a discussion of NetCDF
files and data types, but it also discusses the precision of Ferret
internal calculations.

> The declaration "float" indicates that the variable is to be
> stored
> as single precision, floating point (32-bit IEEE representation).
> ...   Note that although (other) data types may result in
> smaller files,
> they will not affect Ferret's memory usage, as all variables are
> converted to "float" internally as they are read by Ferret.
Ansley Manke

Dongxiao Zhang wrote:

Hi,

What is the floating point precision of ferret calculations?
I understand that 'precision=#digits' can control this?
But how to explain the following results?

yes? list/p=9 0.5*365.2425
             VARIABLE : 0.5*365.2425
          182.621246

We can easily get the correct answer, 182.62125, without bothering the
computer.

When searching the mail archive I found the following related question
without getting answered?

??????????????????
http://ferret.wrc.noaa.gov/Ferret/Mail_Archives/fu_98/msg00254.html

Hello,

I find that my spatial averages are not very accurate. In searching for
the problem, I've wanted to consider round off error. What is the
floating point precision of ferret calculations? Can it be varied to test
the sensitivity of the results? How accurately does it calculated the area
of a grid box?

Does anyone have experience with checking the accuracy of the spatial
average transform that would be informative?

Thanks for your help!
Rob Scott
??????????????????????

Thanks for any insights.

Dongxiao
--------------------------------------------------------------------
Dr. Dongxiao Zhang
NOAA/PMEL Bld#3
7600 Sand Point Way NE
Seattle, WA 98115-0070
Phone  : (206)526-4184
--------------------------------------------------------------------

--
Ansley Manke  Pacific Marine Environmental Laboratory  Seattle WA  (206)526-6246