Re: [ferret_users] Re: How to manipulate data ( irregular ) after reading from file to make grid

On Tue, Dec 6, 2016 at 7:03 PM, 'Don DNA' via _OAR PMEL Ferret Users <ferret_users@xxxxxxxx> wrote:

Thank you so much for last answer it really helped me to start .

I have one more query as you can see below, I can easily fill data in numeric variable in grid, I am trying to fill string inside 2d grid, I could not succeed, please look into it.

yes? define axis/x=1:391:1 xdata
yes? define axis/y=1:6:1 ystation
yes? define grid/x=xdata/y=ystation xygrid;
yes? sh g xygrid
    GRID XYGRID
name       axis              # pts   start                end
XDATA     X                  391 r   1                    391
YSTATION Y                    6 r   1                    6
normal    Z
normal    T
normal    E
normal    F

Fill numeric value 1000 inside grid - success

yes? let numeric_var = 1000 + 0* x[g=xygrid] * y[g=xygrid];
yes? list numeric_var
             VARIABLE : 1000 + 0* X[G=XYGRID] * Y[G=XYGRID]
             SUBSET   : 391 by 6 points (X-Y)
      ... listing every 23th point
             1     24     47     70     93     116    139    162    185    208    231    254    277    300    323    346    369
             1     24     47     70     93    116    139    162    185    208    231    254    277    300    323    346    369
1   / 1: 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000.
2   / 2: 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000.
3   / 3: 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000.
4   / 4: 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000.
5   / 5: 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000.
6   / 6: 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000. 1000.

Fill string value 'test' inside grid - failed

yes? let fill_data = "test"; let string_var = fill_data[g=xygrid@asn];
yes? list string_var
             VARIABLE : FILL_DATA[G=XYGRID@ASN]
             SUBSET   : 391 by 6 points (X-Y)
      ... listing every 14th point
           1 15294357718599* * * * * * * * * * * * * * * * * * * *
          115294357718599****************************************
1   / 1:""""""""""""""""""""""""""""""""""""""""""""""""""""""""
2   / 2:""""""""""""""""""""""""""""""""""""""""""""""""""""""""
3   / 3:""""""""""""""""""""""""""""""""""""""""""""""""""""""""
4   / 4:""""""""""""""""""""""""""""""""""""""""""""""""""""""""
5   / 5:""""""""""""""""""""""""""""""""""""""""""""""""""""""""
6   / 6:""""""""""""""""""""""""""""""""""""""""""""""""""""""""

How it can be done ? where I am doing wrong ?, Please guide me.

Thank you.

On Tuesday, December 6, 2016 4:06 AM, Ansley C. Manke <ansley.b.manke@xxxxxxxx> wrote:

Hi,

Ok, now I understand. If you start out to read the data as follows,

yes? columns/skip=1/format=delim/delim="\b,\,"\...? /type="numeric,numeric,numeric,numeric,eurodate,time,numeric,numeric"\...? /var="pkey1,pkey2,latitude,longitude,st_date,st_time,temp,salt" test.csv

yes? list/i=1:20 pkey1,pkey2              DATA SET: ./test.csv              X: 0.5 to 20.5 Column 1: PKEY1 is pkey1 Column 2: PKEY2 is pkey2            PKEY1   PKEY2 1    / 1: 869.0 1.000 2    / 2: 869.0 1.000 3    / 3: 869.0 1.000 4    / 4: 869.0 1.000 5    / 5: 869.0 1.000 6    / 6: 869.0 1.000 7    / 7: 869.0 1.000 8    / 8: 869.0 1.000 9    / 9: 869.0 1.000 10   / 10: 869.0 2.000 11   / 11: 869.0 2.000 12   / 12: 869.0 2.000 13   / 13: 869.0 2.000 14   / 14: 869.0 2.000 15   / 15: 869.0 2.000 16   / 16: 869.0 2.000 17   / 17: 869.0 2.000 18   / 18: 869.0 2.000 19   / 19: 869.0 3.000 20   / 20: 869.0 3.000

It looks as if the pkey1,pkey2 repeat every 9 records. Define a grid that's X-Y where X will be the data at each station, since it is varying fastest, and Y is the stations.yes? define axis/x=1:9:1 xdata

yes? let nrec = `pkey1,return=isize`yes? define axis/y=1:`nrec/9`:1 ystation

yes? define grid/x=xdata/y=ystation xygrid

! now read the data onto this 2-d grid,

yes? cancel data/all

yes? columns/grid=xygrid/skip=1/format=delim/delim="\b,\,"\ ...? /type="numeric,numeric,numeric,numeric,eurodate,time,numeric,numeric"\ ...? /var="pkey1,pkey2,latitude,longitude,st_date,st_time,temp,salt" test.csv

yes? show data     currently SET data sets:     1> ./test.csv (default) name     title                             I         J         K         L PKEY1    pkey1                            1:9       1:81      ...       ... PKEY2    pkey2                            1:9       1:81      ...       ... LATITUDE latitude                         1:9       1:81      ...       ... LONGITUDE           longitude                        1:9       1:81      ...       ... ST_DATE st_date                          1:9       1:81      ...       ...        (Julian days since 1-Jan-1900) ST_TIME st_time                          1:9       1:81      ...       ...        (Time of day) TEMP     temp                             1:9       1:81      ...       ... SALT     salt                             1:9       1:81      ...       ...

yes? sh grid salt     GRID XYGRID name       axis              # pts   start                end                 subset XDATA     X                    9 r   1                    9                   full YSTATION Y               81 r   1                    81                  full normal    Z normal    T

The dates and time units are shown, so you could define a single "time" variable which is days and fraction of a day,

yes? let/units="days since 1-Jan-1900" time = st_date + st_time/24.

you could also define a single station-id variable by combining pkey1 and pkey2 in some way. Finally, give the variables units and titles with SET VAR/UNITS= /TITLE= and save them to a netCDF file.

If you want to, when working with the data, you can even define a time axis from each station's time data (but there is no single time axis for the whole set).

yes? define axis/t/t0=1-Jan-1900/units=days time_3 = time[y=3] yes? show axis time_3 name       axis              # pts   start                end TIME_3    TIME                 9 i   14-FEB-2000 00:00    14-FEB-2000 08:00 T0 = 1-JAN-1900    Axis span (to cell edges) = 0.375

On 12/3/2016 4:54 AM, Don DNA wrote:

Yes you are right, its station data, I understood your point, defining time axis is difficult with this data, but as you can see I got 82 stations, considering first 2 columns, pkey1,pkey2, here pkey1 is cruise code, and pkey2 is station code, because of this, I wanted to create continuous 1D (of size 82 , i=1:82) grid for meta data information columns (pkey1, pkey2, latitude, longitude) , and for st_date, temp,salt 2D grid ( of I x M, 82 stations on "I" and 8 records per station on "M" as you can see 8 records is maximum so, m=1:8 )

I hope you will understand my problem. after saving file as netcdf, in case if I have to see 1st station data, then following command should give me output

! for station 1data

list/i=1 st_date,temp,salt

! for station 1 metadata

list/i=1 pkey1,pkey2,latitude,longitude

Is it possible to do like which I expected ?

On Saturday, December 3, 2016 2:55 AM, Ansley C. Manke <ansley.b.manke@xxxxxxxx> wrote:

Hello,

This looks like it is a set of data at various stations, but are not organized as a grid. I'll show a few Ferret commands to let you look at what is in the file.

If the station locations were chosen so that they form a grid, then you could define a grid that corresponds to the station locations (or in fact define any grid you want to), and use one of the SCAT2GRID*XY functions to put the temp, salt variables into the grid. That would mean that you would be ignoring the fact that the measurements represent several year's worth of data. You could perhaps define an X-Y-T grid and use a SCAT2GRID*XYT function to put the data onto that. Likely this data would be fairly sparse in such a grid.

Instead you might want to think about the data as non-gridded data, as sets of time series at each station.

Note that if you edit the file to remove the quotes around the date-time field, then you can read the dates and times, by telling Ferret that a blank is another delimiter to use. (Future versions of Ferret will have options to read time records like this using a single date-time data type, for now read two variables date and time).

yes? columns/skip=1/format=delim\ /delim="\b\,"/type="numeric,numeric,numeric,numeric,eurodate,time,numeric,numeric"\ /var="pkey1,pkey2,latitude,longitude,st_date,st_time,temp,salt" test.csv
yes? sh dat     currently SET data sets:    1> ./test.csv (default) name     title                             I         J         K         L PKEY1    pkey1                            1:729     ...       ...       ... PKEY2    pkey2                            1:729     ...       ...       ... LATITUDE latitude                         1:729     ...       ...       ... LONGITUDE          longitude                        1:729     ...       ...       ... ST_DATE st_date                          1:729     ...       ...       ...       (Julian days since 1-Jan-1900) ST_TIME st_time                          1:729     ...       ...       ...       (Time of day) TEMP     temp                             1:729     ...       ...       ... SALT     salt                             1:729     ...       ...       ...

! time in days and fractions of daysyes? let date_and_time = st_date + st_time/24
Now you can investigate the data by making some plots, for instance:

yes? plot longitudeyes? plot latitudeyes? plot/vs/line/sym longitude,latitudeyes? plot date_and_timeyes? plot/i=1:50 date_and_time ! to zoom in on the details of the time information

You'll see that the longitude/latitudes as they appear in the file not represent a grid; if they did then the one varying fastest would repeat. You'll also see that the times are sequences of times at each station, but they do not represent a single time axis.

-Ansley

On 12/2/2016 4:49 AM, Don DNA wrote:

Hey All,

I got csv file, which I can read easily on ferret with 1 line command, but I don't know how to manipulate further to make grid which I am looking forward, please someone help me, it will be very useful.

Here is my problem

$ wc -l test.csv
730 test.csv

$ head -1 test.csv
"pkey1","pkey2","LATITUDE","LONGITUDE","ST_DATE","TEMP","SALT"

$ cut -d',' -f1,2,3,4 test.csv | sort -un | wc -l
82

$ cut -d',' -f1,2,3,4 test.csv | sort -un | head -10
"pkey1","pkey2","LATITUDE","LONGITUDE"
869,1,0,93
869,2,0,93
869,3,0,93
869,4,0,93
869,5,0,93
869,6,0,93
884,1,0,83
884,2,0,83
884,3,0,83

$ ferret
     NOAA/PMEL TMAP
     FERRET v6.96
     Linux 2.6.32-573.7.1.el6.x86_64 64-bit - 12/02/15
     2-Dec-16 17:52

yes? columns/skip=1/format=delim/delim="\,"/type="numeric,numeric,numeric,numeric,text,numeric,numeric"/var="pkey1,pkey2,latitude,longitude,st_date,temp,salt" test.csv
yes? sh d
     currently SET data sets:
    1> ./test.csv (default)
name     title                             I         J         K         L         M         N
PKEY1    PKEY1                            1:730     ...       ...       ...       ...       ...
PKEY2    PKEY2                            1:730     ...       ...       ...       ...       ...
LATITUDE LATITUDE                         1:730     ...       ...       ...       ...       ...
LONGITUDE
          LONGITUDE                        1:730     ...       ...       ...       ...       ...
ST_DATE ST_DATE                          1:730     ...       ...       ...       ...       ...
TEMP     TEMP                             1:730     ...       ...       ...       ...       ...
SALT     SALT                             1:730     ...       ...       ...       ...       ...

yes?

I would like to make grid such a way that, when I type "sh d", I want to see like this, since my data is not uniformly spaced, I want to keep in 'M' as string only

yes? sh d
     currently SET data sets:
    1> ./test.csv (default)
name     title                             I         J         K         L         M                 N
PKEY1    PKEY1                            1:82     ...       ...       ...       ...               ...
PKEY2    PKEY2                            1:82     ...       ...       ...       ...               ...
LATITUDE LATITUDE                         1:82     ...       ...       ...       ...               ...
LONGITUDE
          LONGITUDE                        1:82     ...       ...       ...       ...               ...
ST_DATE ST_DATE                          1:82     ...       ...       ...       1:maxlength       ...
TEMP     TEMP                             1:82     ...       ...       ...       1:maxlength       ...
SALT     SALT                             1:82     ...       ...       ...       1:maxlength       ...

If I list like this, it should show no of records = maxlength, something like below

list/i=1 st_date,temp,salt
             DATA SET: ./test.csv
             X: 0.5 to 8.5
Column 1: ST_DATE
Column 2: TEMP
Column 3: SALT
                      ST_DATE   TEMP   SALT
1           / 1: "2000-02-13 17:00:00" 14.37 35.14
2           / 2: "2000-02-13 18:00:00" 15.25 35.09
3           / 3: "2000-02-13 19:00:00" 14.42 35.10
4           / 4: "2000-02-13 20:00:00" 14.37 35.14
5           / 5: "2000-02-13 21:00:00" 14.25 35.12
6           / 6: "2000-02-13 22:00:00" 14.32 35.19
7           / 7: "2000-02-13 23:00:00" 14.37 35.14
maxlength   / 8: "2000-02-14 00:00:00" 14.30 35.14

Please someone help me.

Thanks in advance.