Re: [ferret_users] precision issue ?

[Ansley Manke <ansley.b.manke@xxxxxxxx>]

Hi Alexa,
It may well be due to precision but before you get too worried, what 
precision do you have in the data you're starting with?  Single 
precision calculations keep about 6 or 7 significant figures and 128 is 
definitely below that for data of magnitude 1.458e+9.   The comparison 
is 128 / 1.458E+09  = 8.7e-8. Are your data accurate to 8 decimal 
places?  Another way to think of this is, if your numbers were values 
like 146, would you be concerned about a difference of 0.0000128?

When thinking about questions like this, take a look at the data using 
the LIST/PRECISION= qualifier

list/prec=7 UTEDDYC[k=1,j=500,i=205]*UTEDDYA[k=1,j=500,i=205]
...

Alexa Griesel wrote:

> Dear ferret users,
> I have variables defined on a lon,lat,depth grid.  I am calculating 
> cross (uteddyc, vteddyc) and along (uteddya, vteddya) temperature 
> contours components of eddy heat fluxes. As a test,  
> uteddyc*uteddya+vteddyc*vteddya  should be zero.
> but what I am getting is 2^n in a lot of places (i.e. 2, 8, 16, 32, 
> 64,128..etc.), e.g. see below:
>
> yes? list UTEDDYC[k=1,j=500,i=205]*UTEDDYA[k=1,j=500,i=205]
>                    -1.458E+09
> yes? list VTEDDYC[k=1,j=500,i=205]*VTEDDYA[k=1,j=500,i=205]
>                     1.458E+09
> yes? list 
> UTEDDYC[k=1,j=500,i=205]*UTEDDYA[k=1,j=500,i=205]+VTEDDYC[k=1,j=500,i=205]*VTEDDYA[k=1,j=500,i=205] 
>                    -128.0
>
> so listing the terms individually, they should add up to zero. Is this 
> a consequence of ferret operating in single precision ? This makes
> me not trust my ferret calculations any more..
> thanks for any hints! Alexa