Re: [ferret_users] maximum size of variable (again)

[Ansley Manke <a.c.manke@xxxxxxxxx>]

Hi Ryo,

I don't think this is a variable-size error on writing to the file; that limitation has long-since been handled with updates to the netCDF libraries.  Maybe this has something to with the grids of the variables as defined by pyferret as it initializes the file. When you open your files and check SHOW GRID var1 etc, does the grid have the dimension of length 36 in the m direction? 

If the grid does have the shape you expect, then I think that the repeat loop should not have the/mlimits inside it.  Assuming var1 and var2 have an m axis of length 36, I would do this:

  save/clobber/mlimits=36/m=1 var1[d=1], var2[d=1]
  repeat/m=2:36 (save/append var1[d=1], var2[d=2])

which SHOULD automatically write first the data at m=2, then m=3 etc.  Or try,

  save/clobber/mlimits=36/m=1 var1[d=1], var2[d=1]

  repeat/m=2:36 (save/append var1[d=1,m=`_m`], var2[d=2,m=`_m`])

The syntax here is _m when we are evaluating the pseudo-variable, index on an axis in the e direction.  The command qualifier /m=  uses just m, as does the m= in var2[d=2,m= ].  Those uses are not the values of pseudo-variables.

-Ansley

    8/2021 3:22 AM, Ryo Furue wrote:

Dear Ferreters,

I found a discussion in the mailing list (from 2001), which seems to be saying that the maximum size of a single variable which doesn't include a record dimension is 2 GiB.

Then I'm puzzled.  My variables all have I, J, K, and M dimensions with I=1:500, J=1:400, K=1:67, and M=1:36.  All double precision.  So each variable is 3.6 GiB.

I managed to create netCDF files that each include a single variable via

repeat/k=1:67 (save/append/klimits=1:67/k=`k` myvar)

The k-size of myvar is 1.

ncdump -h says

dimensions:
XU = 500 ;
YV = 400 ;
ZT = 67 ;
bnds = 2 ;
FREQ_AX = 36 ;

(freq_ax is the name of my ensemble dimension.)  As you can see, there is no record dimension.  If the last dimension were a record dimension, the listing would say

FREQ_AX = UNLIMITED ; // (36 currently)

So, somehow I managed to create such datasets as this.

But then, I wanted to put two such variables into one file:

set data myvar1.nc

set data myvar2.nc

SPAWN rm result.nc

repeat/m=1:36 (save/append/mlimits=1:36 var1[d=1], var2[d=2])

This program fails with

NetCDF: One or more variable sizes violate format constraints (OPeNDAP/netCDF Error code -62)

When I limit the size by var1[d=1,k=1:37], var2[d=2,k=1:37], the above program works.  If you calculate the size, then you'll find that k=1:37 just puts each of var1 and var2 just below 2 GiB.

So, I guess, in the REPEAT loop above, the fact m takes a single value is somehow missed.

Also, I wasn't able to add m=`m` to the SAVE statement because that results in "**ERROR: variable unknown or not in data set: M".  I also tried to use _m , but repeat/_m=1:36 results in a syntax error.

So, I'm stuck.

Ryo