Well, first I'll briefly try to answer the Ferret questions. To handle the variable naming, you can rename the variable in the input dataset to some dummy name, then define the variable to write using the original name.
yes? use input_file.ncYou could use a repeat loop to do the SET VAR/NAME for all the variables in the input file, similar to the loop in this message :
yes? set var/name=temp_in temp
yes? let temp = [some _expression_ involving temp_in]
http://www.pmel.noaa.gov/maillists/tmap/ferret_users/fu_2007/msg00621.html (where there should be a definition "let nvars = ..nvars")
For managing the sequence of events to get all the data written, usually writing one or more smaller scripts to do the actual operations is the way to go, passing filenames, indices, variable names or whatever else, into the scripts as arguments.
But, I'm not sure that writing all the data into one giant file is the right way to manage a collection of data like this. What happens when more data needs to be added? Or removed? Re-write the whole things? This is not flexible. Managing and using large files has problems.
One alternative that comes to my mind is ERDDAP, http://coastwatch.pfeg.noaa.gov/erddap. This is the product of one of our sister NOAA labs, which lets you present a set of data files as a single dataset. It sounds as if your data is a collection of time series so you might look at the "tabledap" protocol. I have not installed ERDDAP myself but my understanding is that it is not difficult to install or to set up with data. It lets one access data from a browser page, look at tables, download data subsets, and it has has an interface that draws graphics. Specially constructed URL's can also be used to access an ERDDAP dataset, specifying a subset of variables and constraints, so that one can choose some of the variables, over, say a range of latitude and longitude, or perhaps at a specified set of station names, and then save that in a file. Please have a look and see what you think.
Ansley
On 12/6/2013 12:00 PM, Akshay Hegde wrote:
and then again creating new 29 variable, regridding them to new xaxis, will be tedious job, and I suspect even appending would result crash signal, and again renaming those variable I have to use nco I think.Thank you so much for your time mam, but I have 29 variables in each file, more than 10,000 files like merge1 merge2 ....merge10000, which was extracted from single file with i index 1:24000
so if I use individual file define axis would be
define axis/x=1:10001:10000 outaxis
whether any solution using repeat to fetch some index in file from test_1.nc and write to new file ?
! write index 1,3, and 5 of i to new file
yes? let my_index_to_write = {1,3,5}
yes? list my_index_to_write
VARIABLE : {1,3,5}
SUBSET : 3 points (X)
1 / 1: 1.000
2 / 2: 3.000
3 / 3: 5.000
yes? repeat/i=1:`temp,return=iend` (
! check whether i is available in my_index_to_write
! if i is available ? write to new file and append
.........
)
so finally I expect output axis should contain 1:3 i index , with x 1,4, I really don't want to use nco to change variable name
Akshay
On Sat, Dec 7, 2013 at 12:47 AM, Ansley Manke <ansley.b.manke@xxxxxxxx> wrote:
Hi,
Two things are going on here. I think you may be confusing index values with coordinate values a bit. The main issue though is that in order to append in directions other than time, you need to do some special set-up first. I'll answer in-line, below.
On 12/6/2013 3:43 AM, Akshay Hegde wrote:
In test_1.nc, both the index values and the coordinate values run from 1 through 5. The coordinate values happen to match the indices. So i=5 is the index value for coordinate x=5.I am interested to merge file, this is what I have done, let me know if I am doing wrong here...
yes? let c1=`temp,return=istart` ! start is same as thqat of sh g temp start
[akshay@aix ~]$ f
NOAA/PMEL TMAP
FERRET v6.85
Linux 2.6.18-371.1.2.el5 64-bit - 11/12/13
6-Dec-13 16:49
yes? use test_1.nc
yes? sh d
currently SET data sets:
1> ./test_1.nc (default)
name title I J K L M N
TEMP Temperature 1:5 ... 1:2026 ... ... ...
yes? sh g temp
GRID GBN1
name axis # pts start end
PROFILE X 5 r 1 5
normal Y
DEPTH_LEVEL DEPTH (m) 2026 r- 0 2025
normal T
normal E
normal F
!-> DEFINE VARIABLE c1=1
yes? let c2=`temp,return=iend` ! end is same as that of sh g temp end
!-> DEFINE VARIABLE c2=5
Think of this example. SHOW DATA lists the number of points. SHOW GRID shows the number of coordinates on the axis, then the start and end
yes? use levitus_climatology
yes? sh dat
currently SET data sets:
1> /home/users/tmap/ferret/linux/fer_dsets/data/levitus_climatology.cdf (default)TEMP TEMPERATURE 1:360 1:180 1:20 ... ... ...
name title I J K L M N
SALT SALINITY 1:360 1:180 1:20 ... ... ...
yes? sh grid salt
GRID GMS1XAXLEVITR LONGITUDE 360mr 20.5E 19.5E(379.5)
name axis # pts start end
YAXLEVITR LATITUDE 180 r 89.5S 89.5N
ZAXLEVITR DEPTH (m) 20 i- 0 5000yes? say `salt,return=jstart`
normal T
normal E
normal F
!-> MESSAGE/CONTINUE 1
1
yes? say `salt,return=ystart`
!-> MESSAGE/CONTINUE 89.5S
89.5S
In test_2.nc, there is one coordinate value in the X axis, X=5. istart and iend give the index values (as seen in the SHOW DATA output); SHOW GRID lists the coordinate values in the file. To return the coordinates:
yes? list/file=test_2.nc/format=cdf/clobber/keep_axisname temp[i=5]
LISTing to file test_2.nc
yes? use test_2.nc
yes? sh d 2
currently SET data sets:
2> ./test_2.nc (default)
name title I J K L M N
TEMP Temperature 1:1 ... 1:2026 ... ... ...
yes? sh g temp[d=2]
GRID GAY1
name axis # pts start end
PROFILE1 X 1 r 5 5
normal Y
DEPTH_LEVEL DEPTH (m) 2026 r- 0 2025
normal T
normal E
normal F
yes? let c1=`temp[d=2],return=istart` ! whereas this is not
!-> DEFINE VARIABLE c1=1
yes? let c2=`temp[d=2],return=iend` ! whereas this is not
!-> DEFINE VARIABLE c2=1
yes?
yes? let v2 = `temp[d=2],return=xstart`
!-> DEFINE VARIABLE v2 = 5
yes? let v2 = `temp[d=2],return=xend`
!-> DEFINE VARIABLE v2 = 5You can use /append without doing anything special, if you are appending in TIME. If you want to add more data along another direction, you have to first define the full output axis, and save with the limits on that axis, in this case with /ilimits, on the first SAVE command, and then you can fill in the data in those directions with /append. This is discussed in the documentation - See the documenation for LIST/APPEND, and follow the link in that section about "Appending Multiple Slabs".
and then merging
yes? use test_1.nc
yes? sh d
currently SET data sets:
1> ./test_1.nc (default)
name title I J K L M N
TEMP Temperature 1:5 ... 1:2026 ... ... ...
yes? list/file=merge1.nc/format=cdf/clobber/keep_axisname temp[i=3]
LISTing to file merge1.nc
yes? list/file=merge2.nc/format=cdf/clobber/keep_axisname temp[i=5]
LISTing to file merge2.nc
yes? canc data/all
yes? use merge1.nc ; use merge2.nc
yes? sh d
currently SET data sets:
1> ./merge1.nc
name title I J K L M N
TEMP Temperature 1:1 ... 1:2026 ... ... ...
2> ./merge2.nc (default)
name title I J K L M N
TEMP Temperature 1:1 ... 1:2026 ... ... ...
yes? sh g temp[d=1]; sh g temp[d=2]
GRID GBM1
name axis # pts start end
PROFILE X 1 r 3 3
normal Y
DEPTH_LEVEL DEPTH (m) 2026 r- 0 2025
normal T
normal E
normal F
GRID GAV1
name axis # pts start end
PROFILE1 X 1 r 5 5
normal Y
DEPTH_LEVEL DEPTH (m) 2026 r- 0 2025
normal T
normal E
normal F
yes? list/file=merged_file.nc/format=cdf/clobber temp[d=1]
LISTing to file merged_file.nc
yes? list/file=merged_file.nc/format=cdf/append temp[d=2]
LISTing to file merged_file.nc
**TMAP ERR: error in line definition
file coords dont match variable coords on axis PROFILE
yes? q
To save data at X=1 and X=3 to a single file (as usual if needed, assign units etc to the variable)
yes? use merge1, merge2
yes? define axis/x=1:3:2 out_axis
yes? let out_temp = temp[d=1,gx=out_axis]
yes? save/file=merged_file.nc/clobber/ilimits=1:2/i=1 out_temp
LISTing to file merged_file.nc
yes? let out_temp = temp[d=2,gx=out_axis]
yes? save/file=merged_file.nc/append/i=2out_temp
LISTing to file merged_file.nc
When you open up that output file, see how the output axis has 2 points, i=1:2, with x=1, 3.
If I open file one after another individually
[akshay@aix ]$ f
NOAA/PMEL TMAP
FERRET v6.85
Linux 2.6.18-371.1.2.el5 64-bit - 11/12/13
6-Dec-13 16:55
yes? use merge1.nc
yes? sh d
currently SET data sets:
1> ./merge1.nc (default)
name title I J K L M N
TEMP Temperature 1:1 ... 1:2026 ... ... ...
yes? sh g temp
GRID GBM1
name axis # pts start end
PROFILE X 1 r 3 3
normal Y
DEPTH_LEVEL DEPTH (m) 2026 r- 0 2025
normal T
normal E
normal F
yes? q
[akshay@aix ]$ f
NOAA/PMEL TMAP
FERRET v6.85
Linux 2.6.18-371.1.2.el5 64-bit - 11/12/13
6-Dec-13 16:55
yes? use merge2.nc
yes? sh d
currently SET data sets:
1> ./merge2.nc (default)
name title I J K L M N
TEMP Temperature 1:1 ... 1:2026 ... ... ...
yes? sh g temp
GRID GAV1
name axis # pts start end
PROFILE X 1 r 5 5
normal Y
DEPTH_LEVEL DEPTH (m) 2026 r- 0 2025
normal T
normal E
normal F
yes?
Any solution/comment/compliments will be appreciated .
Akshay