Hi Martin,
Ferret/Pyferret makes a call to the netCDF library to read data, and gets a status return flag, and is set to stop with an error if that return is not a "success" value. Ferret writes the line
**netCDF error
Then it makes a call to a netCDF routine to translate the error code. That translation returns a string, in this case,
NetCDF: I/O failure (OPeNDAP/netCDF Error code -68)
Ferret writes that string, and then the URL or file name that was being used, and the command it was running, then returns to the command line. Ferret does not directly write any messages that contain "CURL Error".
The "listing to file" message is issued AFTER Ferret has read the data, and received a "success" return flag.
This sounds as if the netCDF library (or some other library that it is calling, maybe CURL?), is writing the "CURL Error: Failure when receiving data from the peer" message itself, before the read call is finished, and that the return flag from the netCDF call is coming back as a successful read. I have not seen this issue, but searching the web for the message, it certainly seems to be something people run into.
You want to at least detect this and exit, before writing bad data to your result file. You could use SET REDIRECT to send the output of your script to a file, and separate the commands to LOAD data and compute the climatology. (Note the SET REDIRECT/STDERR option described in the documentation is not implemented.)
set redirect/file=log.txt/clobber
use "http:......"
set region
define the climatology variable
LOAD data for climatology
spawn to check the log for CURL error, exit if it is found **
save to filecancel redirect ! restart your log
set redirect/file=log.txt/clobber
...
** something like this:
yes? let varc = {SPAWN:"grep CURL log.txt"}; let bad_read = STRINDEX(varc[i=1], "CURL"); if `bad_read` then exit/script
On 7/4/2016 3:44 AM, Martin Schmidt wrote:
Hi,
I am not really sure where to send this question to. From time to time I get the following error when reading data from a THREDDS server:
CURL Error: Failure when receiving data from the peer
but ferret continues ...
LISTing to file ....
Anyway, if this happens, the resulting data are shifted within the grid and the result is useless.
It is hopeless to find the "rule" how data are changed.
Ferret is calculating a climatology from a huge model output. That requires to split the model domain into subdomains.
After defining a region, the results for the climatology are written into separate files and joined later. The scripts for
finding the climatology are simple, just like in the tutorial. The data flow is
use "http:......"
set region
find the climatology
save to file
set region
find the climatology
save to file
and so on
The error shows up only intermittently. I have checked that the data set on the THREDDS is sane.
The THREDDS is pretty old, Version 4.2.10 - 20120417.2151. The access log of the THREDDS looks as it should. I discovered from the logs, that some of my tiles should be smaller, since ferret sends two requests to the THREDDS subdividing also the vertical coordinate. This works mostly well, but it seems, that the error shows up only in this case.
As a test - with smaller tiles I did not see the error yet but the calculation takes much longer.
Ferret is pyferret 6.96 (PyFerret 1.2) running on OpenSuse 13.2. It is py_ferret_120_ub_py_27.
The same for ferret/6.951. This was a prerelease of 6.96 (not pyferret!)
With older ferret/6.85 some files are written. Later a similar error shows up but I get more details and ferret stops:
curl error details:
**netCDF error
NetCDF: I/O failure (OPeNDAP/netCDF Error code -68)
Data set: http://phy-4:8080/thredds/dodsC/genus/genus_era_exp13_run_40_5day_ocean.nc
This is a "Generic IO error". What does this mean?
I have no idea, how to fix this. Did anybody observe something similar?
Best,
Martin Schmidt