Hi Martin,
Ferret/Pyferret makes a call to the netCDF library to read data,
and gets a status return flag, and is set to stop with an error if
that return is not a "success" value. Ferret writes the line
**netCDF error
Then it makes a call to a netCDF routine to translate the error
code. That translation returns a string, in this case,
NetCDF: I/O failure (OPeNDAP/netCDF Error code
-68)
Ferret writes that string, and then the URL or file name that was
being used, and the command it was running, then returns to the
command line. Ferret does not directly write any messages that
contain "CURL Error".
The "listing to file" message is issued AFTER Ferret has read the
data, and received a "success" return flag.
This sounds as if the netCDF library (or some other library that
it is calling, maybe CURL?), is writing the "CURL Error: Failure
when receiving data from the peer" message itself, before the read
call is finished, and that the return flag from the netCDF call is
coming back as a successful read. I have not seen this issue, but
searching the web for the message, it certainly seems to be
something people run into.
You want to at least detect this and exit, before writing bad
data to your result file. You could use SET REDIRECT to send the
output of your script to a file, and separate the commands to LOAD
data and compute the climatology. (Note the SET REDIRECT/STDERR
option described in the documentation is not implemented.)
set redirect/file=log.txt/clobber
use "http:......"
set region
define the climatology variable
LOAD data for climatology
spawn to check the log for CURL error, exit if it is found **
save to file
cancel redirect ! restart your log
set redirect/file=log.txt/clobber
...
** something like this:
yes? let varc = {SPAWN:"grep CURL log.txt"}; let bad_read =
STRINDEX(varc[i=1], "CURL"); if `bad_read` then exit/script
On 7/4/2016 3:44 AM, Martin Schmidt
wrote:
Hi,
I am not really sure where to send this question to. From time to
time I get the following error when reading data from a THREDDS
server:
CURL Error: Failure when receiving data from the peer
but ferret continues ...
LISTing to file ....
Anyway, if this happens, the resulting data are shifted within the
grid and the result is useless.
It is hopeless to find the "rule" how data are changed.
Ferret is calculating a climatology from a huge model output. That
requires to split the model domain into subdomains.
After defining a region, the results for the climatology are
written into separate files and joined later. The scripts for
finding the climatology are simple, just like in the tutorial. The
data flow is
use "http:......"
set region
find the climatology
save to file
set region
find the climatology
save to file
and so on
The error shows up only intermittently. I have checked that the
data set on the THREDDS is sane.
The THREDDS is pretty old, Version 4.2.10 - 20120417.2151. The
access log of the THREDDS looks as it should. I discovered from
the logs, that some of my tiles should be smaller, since ferret
sends two requests to the THREDDS subdividing also the vertical
coordinate. This works mostly well, but it seems, that the error
shows up only in this case.
As a test - with smaller tiles I did not see the error yet but the
calculation takes much longer.
Ferret is pyferret 6.96 (PyFerret 1.2) running on OpenSuse 13.2.
It is py_ferret_120_ub_py_27.
The same for ferret/6.951. This was a prerelease of 6.96 (not
pyferret!)
With older ferret/6.85 some files are written. Later a similar
error shows up but I get more details and ferret stops:
curl error details:
**netCDF error
NetCDF: I/O failure (OPeNDAP/netCDF Error code -68)
Data set:
http://phy-4:8080/thredds/dodsC/genus/genus_era_exp13_run_40_5day_ocean.nc
This is a "Generic IO error". What does this mean?
I have no idea, how to fix this. Did anybody observe something
similar?
Best,
Martin Schmidt