[Thread Prev][Thread Next][Index]

Re: Not losing precision on computations...

Hi Gary,

If 64-bit precision is required in the input values, themselves, in order to
perform a valid calculation, you should not use Ferret for that calculation. If the
precision issues are with the calculation, itself, rather than with the input
values, there may be solutions.

    - steve


Gary Strand wrote:

> I'd like to avoid writing a program by using FERRET to compute some RMS dif-
> ferences. The variables I'm using are defined as netCDF "double", yet accord-
> ing to the manual:
>   "The declaration "float" indicates that the variable is to be stored as
>    single precision, floating point (32-bit IEEE representation). The
>    declarations "long" (32-bit integer), "short" (16-bit integer), "byte"
>    (8-bit integer) and "double" (64-bit IEEE floating point) are also supported
>    by Ferret. Note that although these data types may result in smaller files,
>    they will not affect Ferret's memory usage, as all variables are converted
>    to "float" internally as they are read by Ferret."
> Is there a way around this truncation?


                |  NOAA/PMEL               |  ph. (206) 526-6080
Steve Hankin    |  7600 Sand Point Way NE  |  FAX (206) 526-6744
                |  Seattle, WA 98115-0070  |  hankin@pmel.noaa.gov

[Thread Prev][Thread Next][Index]

Dept of Commerce / NOAA / OAR / PMEL / TMAP

Contact Us | Privacy Policy | Disclaimer | Accessibility Statement