[Thread Prev][Thread Next][Index]

precision of ferret calculation



Hi,

What is the floating point precision of ferret calculations?
I understand that 'precision=#digits' can control this?
But how to explain the following results?

yes? list/p=9 0.5*365.2425
             VARIABLE : 0.5*365.2425
          182.621246

We can easily get the correct answer, 182.62125, without bothering the
computer.

When searching the mail archive I found the following related question
without getting answered?

??????????????????
http://ferret.wrc.noaa.gov/Ferret/Mail_Archives/fu_98/msg00254.html 

Hello,

I find that my spatial averages are not very accurate. In searching for
the problem, I've wanted to consider round off error. What is the
floating point precision of ferret calculations? Can it be varied to test
the sensitivity of the results? How accurately does it calculated the area
of a grid box? 

Does anyone have experience with checking the accuracy of the spatial
average transform that would be informative?

Thanks for your help!
Rob Scott
??????????????????????


Thanks for any insights.

Dongxiao
--------------------------------------------------------------------
Dr. Dongxiao Zhang
NOAA/PMEL Bld#3
7600 Sand Point Way NE
Seattle, WA 98115-0070
Phone  : (206)526-4184
--------------------------------------------------------------------



[Thread Prev][Thread Next][Index]

Dept of Commerce / NOAA / OAR / PMEL / TMAP

Contact Us | Privacy Policy | Disclaimer | Accessibility Statement