[Thread Prev][Thread Next][Index]

Re: [ferret_users] limits on CSV field and record sizes



Hi Marco,
The 20480 points is the default "abstract" axis.  This behavior changed with Ferret v6.93. Ferret now opens the file and determines the number of records, then defines an axis of the right length. However for earlier versions you can define a longer axis and read your data onto that:
yes? DEFINE AXIS/X/X=1:50000:1 longax
yes? DEFINE GRID/X=longax biggrid
yes? FILE/VAR=height/GRID=biggrid plant.dat
yes? PLOT height
I'm not sure what's going on with the behavior you're seeing  that's setting a length of 1530 , but try defining your own axis that's the same length or longer than the file length.

The FILE or SET DATA/EZ command is limited to 20 variables but the "delimited" read is not. This feature is intended to read spreadsheet files. You can define one or more delimiters, and tell Ferret the expected data type of the variables.
! define 50 variables, save them to an ascii file.

yes? let frac = {.1, .2, .3, .4, .5, .6, .7, .8, .9}
yes? repeat/j=1:50 (let a`j` = frac + `j`)

yes? list/clobber/nohead/norow/file=asc50.dat \
...? a1,a2,a3,a4,a5,a6,a7,a8,a9,a10,a11,a12,a13,a14,a15,a16,a17,a18,a19,a20,\
...? a21,a22,a23,a24,a25,a26,a27,a28,a29,a30,a31,a32,a33,a34,a35,a36,a37,a38,\
...? a39,a40,a41,a42,a43,a44,a45,a46,a47,a48,a49,a50


! now read them in
yes? cancel variable/all

yes? file/form=delimited/delimiter" "/var=\
...? "a1,a2,a3,a4,a5,a6,a7,a8,a9,a10,a11,a12,a13,a14,a15,\
...? a16,a17,a18,a19,a20,a21,a22,a23,a24,a25,a26,a27,a28,\
...? a29,a30,a31,a32,a33,a34,a35,a36,a37,a38,a39,a40,a41,\
...? a42,a43,a44,a45,a46,a47,a48,a49,a50" asc50.dat


yes? list a1, a30,a50
             DATA SET: ./asc50.dat
             X: 0.5 to 9.5
 Column  1: A1
 Column  2: A30
 Column  3: A50
            A1    A30    A50
1   / 1:  1.100  30.10  50.10
2   / 2:  1.200  30.20  50.20
3   / 3:  1.300  30.30  50.30
4   / 4:  1.400  30.40  50.40
5   / 5:  1.500  30.50  50.50
6   / 6:  1.600  30.60  50.60
7   / 7:  1.700  30.70  50.70
8   / 8:  1.800  30.80  50.80
9   / 9:  1.900  30.90  50.90

I see this information about delimited reads is not in the documentation - we'll get it added.

-Ansley


On 5/26/2015 4:39 AM, Marco van Hulten wrote:
Hi,

Apparently, there is a limitation of 20 fields (per record) in a CSV
file.  Furthermore, the number of records appears to be limited to 1530.

At least, when I load the file in Ferret 6.9 (file/var=...), it first
shows 20480 (for any csv file).  When reading the values, it sets the
number correctly, as long as it is not more than 1530.  If it is more,
it sets the number to 1530 and ignores the rest of the records.

It is normal to use CSV files for observations, because observations
are usually point data, not gridded.  Many observational data sets are
much larger than the above stated limits.

Is it possible to increase these limits in Ferret significantly?

Cheers,

Marco



[Thread Prev][Thread Next][Index]
Contact Us
Dept of Commerce / NOAA / OAR / PMEL / Ferret

Privacy Policy | Disclaimer | Accessibility Statement