[Thread Prev][Thread Next][Index]

Re: [ferret_users] Whether this is bug in Ferret ? or I am doing something wrong ?



Thank you once again,

I just implemented one more possible way to do so without using regridding transformation and define axis, but using already available function  compressi_by(arg1,arg2). Here I used main file without splitting into parts (merge1 merge2.....merge n),

So here I am sharing the method I implemented, and it seems flexible too, but to retain the same variable name, and their units, etc I need to work on set var/name=dummy main_var as you suggested yesterday. I hope if following is right way, then I will be able to do remaining task.

Have a look on what I have done, and please let me know, if its wrong.

[akshay@aix tmp]$ f
     NOAA/PMEL TMAP
     FERRET v6.85 
     Linux 2.6.18-371.1.2.el5 64-bit - 11/12/13
      7-Dec-13 17:32    

yes? use test_1.nc
 
yes? sh d
     currently SET data sets:
    1> ./test_1.nc  (default)
 name     title                             I         J         K         L         M         N
 TEMP     Temperature                      1:5       ...       1:2026    ...       ...       ...
 
yes? sh g temp
    GRID GBN1
 name       axis              # pts   start                end
 PROFILE   X                    5 r   1                    5
 normal    Y
 DEPTH_LEVEL DEPTH (m)       2026 r-  0                    2025
 normal    T
 normal    E
 normal    F


! Index I want to write
yes? let myindx = {1,,1,,1}

! Count of valid index of array myindx
yes? let n_valid_points = myindx[i=@ngd]

yes? list n_valid_points
             VARIABLE : MYINDX[I=@NGD]
             X        : 0.5 to 5.5
          3.000

yes? list x[gx=temp],myindx
 WARNING: Listed variables have ambiguous coordinates on axes: X
             DATA SET: ./test_1.nc
 Column  1: X is X (axis PROFILE)
 Column  2: MYINDX is {1,,1,,1}
           X  MYINDX
I / 1:  1.000  1.000
I / 2:  2.000   ....
I / 3:  3.000  1.000
I / 4:  4.000   ....
I / 5:  5.000  1.000

yes? list/k=1:25 temp
             VARIABLE : Temperature (Degree C)
             FILENAME : test_1.nc
             SUBSET   : 5 by 25 points (X-DEPTH (m))
                1      2      3      4      5   
                1      2      3      4      5
 0     /  1:   ....   ....   ....   ....   ....
 1     /  2:   ....   ....   ....   ....   ....
 2     /  3:   ....   ....   ....   ....   ....
 3     /  4:   ....   ....   ....   ....   ....
 4     /  5:   ....   ....   ....   ....   ....
 5     /  6:   ....   ....   ....   ....   ....
 6     /  7:   ....   ....   ....   ....   ....
 7     /  8:   ....   ....   ....   ....   ....
 8     /  9:   ....   ....   ....   ....   ....
 9     / 10:   ....   ....   ....   ....   ....
 10    / 11:   ....   ....   ....   ....   ....
 11    / 12:   ....   ....   ....   ....   ....
 12    / 13:   ....   ....   ....   ....   ....
 13    / 14:   ....   ....   ....   ....   ....
 14    / 15:   ....   ....   ....   ....   ....
 15    / 16:   ....   ....   ....   ....   ....
 16    / 17:   ....   ....   ....   ....  27.88
 17    / 18:  29.53  29.22  29.72  29.69  27.81
 18    / 19:  29.53  29.22  29.65  29.65  27.74
 19    / 20:  29.53  29.22  29.59  29.60  27.67
 20    / 21:  29.53  29.22  29.53  29.56  27.59
 21    / 22:  29.53  29.22  29.47  29.52  27.52
 22    / 23:  29.54  29.22  29.40  29.47  27.45
 23    / 24:  29.54  29.22  29.34  29.43  27.38
 24    / 25:  29.54  29.22  29.28  29.39  27.31

yes? list/k=1:25 compressi_by(temp,myindx)
             VARIABLE : COMPRESSI_BY(TEMP,MYINDX)
             FILENAME : test_1.nc
             SUBSET   : 5 by 25 points (X-DEPTH (m))
                1      2      3      4      5   
                1      2      3      4      5
 0     /  1:   ....   ....   ....   ....   ....
 1     /  2:   ....   ....   ....   ....   ....
 2     /  3:   ....   ....   ....   ....   ....
 3     /  4:   ....   ....   ....   ....   ....
 4     /  5:   ....   ....   ....   ....   ....
 5     /  6:   ....   ....   ....   ....   ....
 6     /  7:   ....   ....   ....   ....   ....
 7     /  8:   ....   ....   ....   ....   ....
 8     /  9:   ....   ....   ....   ....   ....
 9     / 10:   ....   ....   ....   ....   ....
 10    / 11:   ....   ....   ....   ....   ....
 11    / 12:   ....   ....   ....   ....   ....
 12    / 13:   ....   ....   ....   ....   ....
 13    / 14:   ....   ....   ....   ....   ....
 14    / 15:   ....   ....   ....   ....   ....
 15    / 16:   ....   ....   ....   ....   ....
 16    / 17:   ....   ....  27.88   ....   ....
 17    / 18:  29.53  29.72  27.81   ....   ....
 18    / 19:  29.53  29.65  27.74   ....   ....
 19    / 20:  29.53  29.59  27.67   ....   ....
 20    / 21:  29.53  29.53  27.59   ....   ....
 21    / 22:  29.53  29.47  27.52   ....   ....
 22    / 23:  29.54  29.40  27.45   ....   ....
 23    / 24:  29.54  29.34  27.38   ....   ....
 24    / 25:  29.54  29.28  27.31   ....   ....

yes? let tester = compressi_by(temp,myindx)

yes? list/file=out.nc/format=cdf/clobber/i=1:`n_valid_points` tester
 !-> list/file=out.nc/format=cdf/clobber/i=1:3 tester
 LISTing to file out.nc

yes? canc data/all
yes? canc var/all

yes? use out.nc

yes? sh d
     currently SET data sets:
    1> ./out.nc  (default)
 name     title                             I         J         K         L         M         N
 TESTER   COMPRESSI_BY(TEMP,MYINDX)        1:3       ...       1:2026    ...       ...       ...
       (Z=-0.5:2026)
 
yes? sh g tester
    GRID GJJ1
 name       axis              # pts   start                end
 XAX1_3    X                    3 r   1                    3
 normal    Y
 DEPTH_LEVEL DEPTH (m)       2026 r-  0                    2025
 normal    T
 normal    E
 normal    F

yes? list/k=1:25 tester
             VARIABLE : COMPRESSI_BY(TEMP,MYINDX)
                        Z=-0.5:2026
             FILENAME : out.nc
             SUBSET   : 3 by 25 points (X-DEPTH (m))
                1      2      3   
                1      2      3
 0     /  1:   ....   ....   ....
 1     /  2:   ....   ....   ....
 2     /  3:   ....   ....   ....
 3     /  4:   ....   ....   ....
 4     /  5:   ....   ....   ....
 5     /  6:   ....   ....   ....
 6     /  7:   ....   ....   ....
 7     /  8:   ....   ....   ....
 8     /  9:   ....   ....   ....
 9     / 10:   ....   ....   ....
 10    / 11:   ....   ....   ....
 11    / 12:   ....   ....   ....
 12    / 13:   ....   ....   ....
 13    / 14:   ....   ....   ....
 14    / 15:   ....   ....   ....
 15    / 16:   ....   ....   ....
 16    / 17:   ....   ....  27.88
 17    / 18:  29.53  29.72  27.81
 18    / 19:  29.53  29.65  27.74
 19    / 20:  29.53  29.59  27.67
 20    / 21:  29.53  29.53  27.59
 21    / 22:  29.53  29.47  27.52
 22    / 23:  29.54  29.40  27.45
 23    / 24:  29.54  29.34  27.38
 24    / 25:  29.54  29.28  27.31
yes?

I am just worried about following warning message
"
WARNING: Listed variables have ambiguous coordinates on axes: X"

Thanks and Regards,
Akshay Hegde
CSIR-National Institute of Oceanography,
Dona Paula, Panjim, Goa -403004



 


On Sat, Dec 7, 2013 at 4:15 AM, Ansley Manke <ansley.b.manke@xxxxxxxx> wrote:
Well, first I'll briefly try to answer the Ferret questions.  To handle the variable naming, you can rename the variable in the input dataset to some dummy name, then define the variable to write using the original name.
yes? use input_file.nc
yes? set var/name=temp_in  temp

yes? let temp = [some _expression_ involving temp_in]
You could use a repeat loop to do the SET VAR/NAME for all the variables in the input file, similar to the loop in this message :
http://www.pmel.noaa.gov/maillists/tmap/ferret_users/fu_2007/msg00621.html  (where there should be a definition "let nvars = ..nvars")

For managing the sequence of events to get all the data written, usually writing one or more smaller scripts to do the actual operations is the way to go, passing filenames, indices, variable names or whatever else, into the scripts as arguments.


But, I'm not sure that writing all the data into one giant file is the right way to manage a collection of data like this. What happens when more data needs to be added?  Or removed? Re-write the whole things? This is not flexible. Managing and using large files has problems.

One alternative that comes to my mind is ERDDAP, http://coastwatch.pfeg.noaa.gov/erddap. This is the product of one of our sister NOAA labs, which lets you present a set of data files as a single dataset.  It sounds as if your data is a collection of time series so you might look at the "tabledap" protocol.  I have not installed ERDDAP myself but my understanding is that it is not difficult to install or to set up with data.  It lets one access data from a browser page, look at tables, download data subsets, and it has has an interface that draws graphics. Specially constructed URL's can also be used to access an ERDDAP dataset, specifying a subset of variables and constraints, so that one can choose some of the variables, over, say a range of latitude and longitude, or perhaps at a specified set of station names, and then save that in a file. Please have a look and see what you think.

Ansley


On 12/6/2013 12:00 PM, Akshay Hegde wrote:
Thank you so much for your time mam, but I have  29 variables in each file, more than 10,000 files like merge1 merge2 ....merge10000, which was extracted from single file with i index 1:24000

so if I use individual file define axis would be

define axis/x=1:10001:10000 outaxis

and then again creating new 29 variable, regridding them to new xaxis, will be tedious job, and I suspect even appending would result crash signal, and again renaming those variable I have to use nco I think.


whether any solution using repeat to fetch some index in file from test_1.nc and write to new file ?

something like this I expect

yes? use test_1.nc

! write index 1,3, and 5 of i to new file
yes? let my_index_to_write = {1,3,5}

yes? list my_index_to_write
             VARIABLE : {1,3,5}
             SUBSET   : 3 points (X)
 1   / 1:  1.000
 2   / 2:  3.000
 3   / 3:  5.000

yes? repeat/i=1:`temp,return=iend` (

! check whether i is available in  my_index_to_write
! if i is available ? write to new file and append
.........
)

so finally I expect output axis should contain 1:3 i index , with x 1,4, I really don't want to use nco to change variable name


Akshay









On Sat, Dec 7, 2013 at 12:47 AM, Ansley Manke <ansley.b.manke@xxxxxxxx> wrote:
Hi,
Two things are going on here. I think you may be confusing index values with coordinate values a bit.  The main issue though is that in order to append in directions other than time, you need to do some special set-up first.  I'll answer in-line, below.


On 12/6/2013 3:43 AM, Akshay Hegde wrote:
I am interested to merge file,  this is what I have done, let me know if I am doing  wrong here...

[akshay@aix ~]$ f
     NOAA/PMEL TMAP
     FERRET v6.85  
     Linux 2.6.18-371.1.2.el5 64-bit - 11/12/13
      6-Dec-13 16:49    

yes? use test_1.nc

yes? sh d
     currently SET data sets:
    1> ./test_1.nc  (default)
 name     title                             I         J         K         L         M         N
 TEMP     Temperature                      1:5       ...       1:2026    ...       ...       ...

yes? sh g temp
    GRID GBN1
 name       axis              # pts   start                end
 PROFILE   X                    5 r                      5
 normal    Y
 DEPTH_LEVEL DEPTH (m)       2026 r-  0                    2025
 normal    T
 normal    E
 normal    F

yes? let c1=`temp,return=istart` ! start is same as thqat of sh g temp start

 !-> DEFINE VARIABLE c1=1

yes? let c2=`temp,return=iend` ! end is same as that of sh g temp end
 !-> DEFINE VARIABLE c2=5

In test_1.nc, both the index values and the coordinate values run from 1 through 5. The coordinate values happen to match the indices. So i=5 is the index value for coordinate x=5.

Think of  this example. SHOW DATA lists the number of points. SHOW GRID shows the number of coordinates on the axis, then the start and end

yes? use levitus_climatology
yes? sh dat
     currently SET data sets:
    1> /home/users/tmap/ferret/linux/fer_dsets/data/levitus_climatology.cdf  (default)

 name     title                             I         J         K         L         M         N
 TEMP     TEMPERATURE                      1:360     1:180     1:20      ...       ...       ...
 SALT     SALINITY                         1:360     1:180     1:20      ...       ...       ...

yes? sh grid salt
    GRID GMS1

 name       axis              # pts   start                end
 XAXLEVITR LONGITUDE          360mr   20.5E                19.5E(379.5)
 YAXLEVITR LATITUDE           180 r   89.5S                89.5N
 ZAXLEVITR DEPTH (m)           20 i-  0                    5000

 normal    T
 normal    E
 normal    F

yes? say `salt,return=jstart`
 !-> MESSAGE/CONTINUE 1
1
yes? say `salt,return=ystart`
 !-> MESSAGE/CONTINUE 89.5S
89.5S

yes? list/file=test_2.nc/format=cdf/clobber/keep_axisname temp[i=5]
 LISTing to file test_2.nc

yes? use test_2.nc
yes? sh d 2
     currently SET data sets:
    2> ./test_2.nc  (default)
 name     title                             I         J         K         L         M         N
 TEMP     Temperature                      1:1       ...       1:2026    ...       ...       ...
 
yes? sh g temp[d=2]
    GRID GAY1
 name       axis              # pts   start                end
 PROFILE1  X                    1 r   5                    5
 normal    Y
 DEPTH_LEVEL DEPTH (m)       2026 r-  0                    2025
 normal    T
 normal    E
 normal    F

yes? let c1=`temp[d=2],return=istart` ! whereas this is not
 !-> DEFINE VARIABLE c1=1

yes? let c2=`temp[d=2],return=iend`  ! whereas this is not
 !-> DEFINE VARIABLE c2=1
yes?

In test_2.nc, there is one coordinate value in the X axis, X=5. istart and iend give the index values (as seen in the SHOW DATA output); SHOW GRID lists the coordinate values in the file. To return the coordinates:

yes? let v2 =  `temp[d=2],return=xstart`
 !-> DEFINE VARIABLE v2 =  5
yes? let v2 =  `temp[d=2],return=xend`
 !-> DEFINE VARIABLE v2 =  5




and then merging

yes? use test_1.nc
yes? sh d
     currently SET data sets:
    1> ./test_1.nc  (default)
 name     title                             I         J         K         L         M         N
 TEMP     Temperature                      1:5       ...       1:2026    ...       ...       ...
 
yes? list/file=merge1.nc/format=cdf/clobber/keep_axisname temp[i=3]
 LISTing to file merge1.nc

yes? list/file=merge2.nc/format=cdf/clobber/keep_axisname temp[i=5]
 LISTing to file merge2.nc

yes? canc data/all

yes? use merge1.nc ; use merge2.nc

yes? sh d
     currently SET data sets:
    1> ./merge1.nc
 name     title                             I         J         K         L         M         N
 TEMP     Temperature                      1:1       ...       1:2026    ...       ...       ...
 
    2> ./merge2.nc  (default)
 name     title                             I         J         K         L         M         N
 TEMP     Temperature                      1:1       ...       1:2026    ...       ...       ...
 
yes? sh g temp[d=1]; sh g temp[d=2]
    GRID GBM1
 name       axis              # pts   start                end
 PROFILE   X                    1 r   3                    3
 normal    Y
 DEPTH_LEVEL DEPTH (m)       2026 r-  0                    2025
 normal    T
 normal    E
 normal    F
    GRID GAV1
 name       axis              # pts   start                end
 PROFILE1  X                    1 r   5                    5
 normal    Y
 DEPTH_LEVEL DEPTH (m)       2026 r-  0                    2025
 normal    T
 normal    E
 normal    F

yes? list/file=merged_file.nc/format=cdf/clobber temp[d=1]
 LISTing to file merged_file.nc

yes? list/file=merged_file.nc/format=cdf/append temp[d=2]
 LISTing to file merged_file.nc
 **TMAP ERR: error in line definition
             file coords dont match variable coords on axis PROFILE
yes? q



You can use /append without doing anything special, if you are appending in TIME.  If you want to add more data along another direction, you have to first define the full output axis, and save with the limits on that axis, in this case with /ilimits, on the first SAVE command, and then you can fill in the data in those directions with /append.  This is discussed in the documentation - See the documenation for LIST/APPEND, and follow the link in that section about "Appending Multiple Slabs".

To save data at X=1 and X=3 to a single file (as usual if needed, assign units etc to the variable)
yes? use merge1, merge2

yes? define axis/x=1:3:2 out_axis

yes? let out_temp = temp[d=1,gx=out_axis]
yes? save/file=merged_file.nc/clobber/ilimits=1:2/i=1 out_temp
 LISTing to file merged_file.nc

yes? let out_temp = temp[d=2,gx=out_axis]
yes? save/file=merged_file.nc/append/i=2out_temp
 LISTing to file merged_file.nc

When you open up that output file, see how the output axis has 2 points, i=1:2, with x=1, 3.


If I open file one after another individually

[akshay@aix ]$ f
     NOAA/PMEL TMAP
     FERRET v6.85 
     Linux 2.6.18-371.1.2.el5 64-bit - 11/12/13
      6-Dec-13 16:55    

yes? use merge1.nc
yes? sh d
     currently SET data sets:
    1> ./merge1.nc  (default)
 name     title                             I         J         K         L         M         N
 TEMP     Temperature                      1:1       ...       1:2026    ...       ...       ...
 
yes? sh g temp
    GRID GBM1
 name       axis              # pts   start                end
 PROFILE   X                    1 r   3                    3
 normal    Y
 DEPTH_LEVEL DEPTH (m)       2026 r-  0                    2025
 normal    T
 normal    E
 normal    F
yes? q
[akshay@aix ]$ f
     NOAA/PMEL TMAP
     FERRET v6.85 
     Linux 2.6.18-371.1.2.el5 64-bit - 11/12/13
      6-Dec-13 16:55    

yes? use merge2.nc
yes? sh d
     currently SET data sets:
    1> ./merge2.nc  (default)
 name     title                             I         J         K         L         M         N
 TEMP     Temperature                      1:1       ...       1:2026    ...       ...       ...
 
yes? sh g temp
    GRID GAV1
 name       axis              # pts   start                end
 PROFILE   X                    1 r   5                    5
 normal    Y
 DEPTH_LEVEL DEPTH (m)       2026 r-  0                    2025
 normal    T
 normal    E
 normal    F
yes?

Any solution/comment/compliments will be appreciated .

Akshay












[Thread Prev][Thread Next][Index]
Contact Us
Dept of Commerce / NOAA / OAR / PMEL / Ferret

Privacy Policy | Disclaimer | Accessibility Statement