[ferret_users] error message NetCDF: One or more variable sizes violate format constraints (OPeNDAP/netCDF Error code -62)

[Ansley Manke <a.c.manke@xxxxxxxxx>]

Hi all,

Here's an error message that I didn't recognize, and which may come up for people working with large datasets. I thought I'd write here, so the information is here in the Users List archive.

Somebody asked me about seeing this error message, when trying to write to a file:

NetCDF: One or more variable sizes violate format constraints (OPeNDAP/netCDF Error code -62)

 could not change CDF mode

list/format=cdf/file="data/Hi_freq_data/xx_20hz_data.nc.nc"/append  variable_38

Command file, command group, or REPEAT execution aborted

The dataset being written is high-frequncy data, with a lot of large variables being added to the file. What is happening, is that the file becomes too large for the default netcdf-3 type data file.  Now, the modern netCDF library has large-file support, as described here: https://docs.unidata.ucar.edu/netcdf-c/current/faq.html#Large-File-Support

PyFerret is linked with a version of netCDF which supports large files. We have the option to write files with netcdf-4 formatting, and also the 64-bit offset format mentioned on that page. Because many linux systems come with only older netCDF-3 libraries, the PyFerret developers decided to write netCDF files using netcdf-3 formatting by default.  That way things like the ncdump and ncgen utilities on people's systems will be able to work with the files. But PyFerret also has command qualifiers available to request netCDF-4 formatted files or 64-bit offset formatted files.  In addition to large-file support, netCDF-4 offers some other options. The syntax for all of this is described here:

https://ferret.pmel.noaa.gov/Ferret/documentation/users-guide/data-set-basics/NETCDF-DATA#_VPINDEXENTRY_netcdf4

A solution for the question above was to use SAVE/NCFORMAT=netcdf4

(Another good solution would be to use Union Aggregation, writing each variable to its own file and then aggregating them in the PyFerret session into a virtual dataset.  See "Aggretation" in the documentation for how to do that.)

Cheers!

Ansley Manke

The contents of this message are mine personally. I am not affiliated with the National Oceanic and Atmospheric Administration.