[Thread Prev][Thread Next][Index]

Re: precision of ferret calculation

Hi Dongxiao,
the `precision=#digits` controls the OUTPUT precision, that is to
say, the formatting, of numbers that are displayed in an expression.
It does not influence the precision of the calculation.

Ferret does its calculations in single-precision floating point
representation see the index entry "precision, of floating-point
variables" in the Users Guide.  This is a discussion of NetCDF
files and data types, but it also discusses the precision of Ferret
internal calculations.

The declaration "float" indicates that the variable is to be stored
as single precision, floating point (32-bit IEEE representation).
...   Note that although (other) data types may result in smaller files,
they will not affect Ferret's memory usage, as all variables are
converted to "float" internally as they are read by Ferret.
Ansley Manke

Dongxiao Zhang wrote:


What is the floating point precision of ferret calculations?
I understand that 'precision=#digits' can control this?
But how to explain the following results?

yes? list/p=9 0.5*365.2425
             VARIABLE : 0.5*365.2425

We can easily get the correct answer, 182.62125, without bothering the

When searching the mail archive I found the following related question
without getting answered?



I find that my spatial averages are not very accurate. In searching for
the problem, I've wanted to consider round off error. What is the
floating point precision of ferret calculations? Can it be varied to test
the sensitivity of the results? How accurately does it calculated the area
of a grid box?

Does anyone have experience with checking the accuracy of the spatial
average transform that would be informative?

Thanks for your help!
Rob Scott

Thanks for any insights.

Dr. Dongxiao Zhang
7600 Sand Point Way NE
Seattle, WA 98115-0070
Phone  : (206)526-4184

Ansley Manke  Pacific Marine Environmental Laboratory  Seattle WA  (206)526-6246

[Thread Prev][Thread Next][Index]

Dept of Commerce / NOAA / OAR / PMEL / TMAP

Contact Us | Privacy Policy | Disclaimer | Accessibility Statement