[Thread Prev][Thread Next][Index]
Re: Not losing precision on computations...
Hi Gary,
If 64-bit precision is required in the input values, themselves, in order to
perform a valid calculation, you should not use Ferret for that calculation. If the
precision issues are with the calculation, itself, rather than with the input
values, there may be solutions.
- steve
=======================
Gary Strand wrote:
> I'd like to avoid writing a program by using FERRET to compute some RMS dif-
> ferences. The variables I'm using are defined as netCDF "double", yet accord-
> ing to the manual:
>
> "The declaration "float" indicates that the variable is to be stored as
> single precision, floating point (32-bit IEEE representation). The
> declarations "long" (32-bit integer), "short" (16-bit integer), "byte"
> (8-bit integer) and "double" (64-bit IEEE floating point) are also supported
> by Ferret. Note that although these data types may result in smaller files,
> they will not affect Ferret's memory usage, as all variables are converted
> to "float" internally as they are read by Ferret."
>
> Is there a way around this truncation?
--
| NOAA/PMEL | ph. (206) 526-6080
Steve Hankin | 7600 Sand Point Way NE | FAX (206) 526-6744
| Seattle, WA 98115-0070 | hankin@pmel.noaa.gov
[Thread Prev][Thread Next][Index]
Dept of Commerce /
NOAA /
OAR /
PMEL /
TMAP
Contact Us | Privacy Policy | Disclaimer | Accessibility Statement