[Thread Prev][Thread Next][Index]

Re: [ferret_users] Wrong result in division



Fabio,

On Sun, Jul 19, 2020 at 6:21 AM 'Fabio' via _OAR PMEL Ferret Users <ferret_users@xxxxxxxx> wrote:

let result = myvar1/myvar2

               MYVAR1  MYVAR2  RESULT
  ---- J:30
25E   / 103:  0.0054  0.0049   1.09

I guess you mean that 0.0054/0.0049 = 1.102 , not 1.09 .

I guess that it's not that the calculation is incorrect but that the values listed on the screen are misleading.  Below I reproduce your result:

yes? let myvar1 = 0.00536
yes? let myvar2 = 0.0049
yes? let result = myvar1/myvar2
yes? list/format="(2(1X,F6.4),1X,F4.2)" myvar1, myvar2, result
 Column  1: MYVAR1 is 0.00536
 Column  2: MYVAR2 is 0.0049
 Column  3: RESULT is MYVAR1/MYVAR2
 0.0054 0.0049 1.09

Note how the value 0.00536 is printed as 0.0054 .

So, you want to see what happens if you increase the numbers of digits when you print the values on your screen.

Ryo


[Thread Prev][Thread Next][Index]
Contact Us
Dept of Commerce / NOAA / OAR / PMEL / Ferret

Privacy Policy | Disclaimer | Accessibility Statement