Re: [ferret_users] Re: handling data

[Ansley Manke <ansley.b.manke@xxxxxxxx>]

Hi,
The question is how to take an ID number and turn it into a sequence number.  See just below for an example of the ascii data.

We can use transformations to define a new ID, that is a variable "new_id" with values of 1, 2, 3, ... representing the sequence number of each profile in the dataset. For the short file example below, it would look like this,  

! initialize the dataset
define axis/x=1:15:1 xaxis
define grid/x=xaxis xgrid
file/skip=1/var="id_in, dep, temp, sal"/grid=xgrid  file.dat

! variable marking the first element of each profile with 1
let pstart = if i eq 1 or id_in[x=@ddf] GT 0 then 1

! Now usethe @EVNT transformation, increase the count every time
! a value of 1 is encountered

let new_id = pstart[i=@evnt:1]

list id_in, pstart, new_id
             DATA SET: ./id_z.dat
             X: 0.5 to 15.5
 Column  1: ID_IN
 Column  2: PSTART is IF I EQ 1 OR ID_IN[X=@DDF] GT 0 THEN 1
 Column  3: NEW_ID is PSTART[I=@EVNT:1]
           ID_IN  PSTART NEW_ID
1    /  1:  30.00  1.000  1.000
2    /  2:  30.00   ....  1.000
3    /  3:  30.00   ....  1.000
4    /  4:  30.00  1.000  2.000
5    /  5:  50.00   ....  2.000
6    /  6:  50.00   ....  2.000
7    /  7:  50.00   ....  2.000
8    /  8:  50.00  1.000  3.000
9    /  9:  60.00   ....  3.000
10   / 10:  60.00   ....  3.000
11   / 11:  60.00   ....  3.000
12   / 12:  60.00   ....  3.000
13   / 13:  60.00   ....  3.000
14   / 14:  60.00   ....  3.000
15   / 15:  60.00   ....  3.000

On 4/20/2013 3:05 AM, golla nageswararao wrote:

Hi,

 The profile id are not 1,2,3,4,...

Now the profile ids are random like

id  dep temp sal

30, 0   temp1 sal1

30, 20 temp1 sal1
30, 50 temp1 sal1
30, 50 temp1 sal1
50, 0
50, 10
50, 40
50, 45
60, 0
60, 25
60, 30
60, 35
60, 40
60, 45
60, 90
.........

If this is the scenario how can the above command work? My problem is to change the id no. like it should be

id  dep temp sal

1, 0   temp1 sal1

1, 20 temp1 sal1
1, 50 temp1 sal1
1, 50 temp1 sal1
2, 0
2, 10
2, 40
2, 45
3, 0
3, 25
3, 30
3, 35
3, 40
3, 45
3, 90
.........

After that I will do the scat2gridgauss_xz command to make it regular depths of choice. Before that I want to make the profile id no. in sequence.

Thanks in advance

On Fri, Apr 19, 2013 at 9:32 PM, Ansley Manke <ansley.b.manke@xxxxxxxx> wrote:

Hi,
You could create a new ID variable which is just a sequence number of the remaining points.  What is the grid of the variables you are working with?  If it is XZ, then you could use

   let  id = i[gx=var]

I would be interested to hear what you did with to handle the depths.

Ansley

On 4/19/2013 12:14 AM, golla nageswararao wrote:

Now my problem is not with the depths. Now my problem is to assign the profile id to regular number instead of random numbers which came because of the extraction. Can u help me in this regard.

Thanks in advance.

On Thu, Apr 18, 2013 at 11:47 PM, Ansley Manke <ansley.b.manke@xxxxxxxx> wrote:

Hi,
Yes, I understand the problem. More and more people seem to be trying to deal with data like this. We don't really have tools to work with sets of scattered profiles taken at different depths. This data is not on any kind of grid and Ferret's design is to handle data that is already on a grid, with some tools for putting it onto grids.

I do not have an answer for you right now.

Ansley

On 4/18/2013 5:24 AM, golla nageswararao wrote:

Hi Ansley,

 Thank you. I could solve the problem. But I got new problem. Now I had extracted the data only for Indian Ocean with l=1:16000. But only 15300 profiles are there. I want map the data over profile numbers and depth values. But here the problem is the profile numbers are random. If they are continuous, I could have used that scat2gridgauss_xy command. How can bring the profile number continuous? The show data command of my extracted data set is some thing like below.

profid   1:16000

dep      1:16000

temp    1:16000

salt       1:16000

profid values are 25,25,25,25,252,252,252,252,252,500,500,500,500,500,500,500,500,500, etc....

How can i fix this problem?

Thanks in advance.

On Thu, Apr 18, 2013 at 12:09 AM, Ansley Manke <ansley.b.manke@xxxxxxxx> wrote:

Hi,
Is this ascii data? I think you can do the sampling step first using a mask.  Let's see if we can work it out, then write back to the list.

I think it's best to read the data in sections, and save just the data at the locations you want.

The idea is:

Read in 26000 points. See which are in the region you want, using a mask based on the lon and lat variables. Apply the mask to all the variables including temp and salt, and write out only the Indian Ocean ones to a file.  Now read the next 26000 points.  You should define the variable "mask" the way you want to, and you may need to change the "file" command to match what is in your input data.

The observations are just a list. They don't have to be organized in any particular direction. Let's write in the T direction because then we can append to the file more easily.

! read_all.jnl
set mem/siz=500

! Define variables to loop over the observations, reading it in 100 sections
let nbig = 2600000
let read1 = 1
let npts = `nbig/100`

! Read and write along the T axis, because we can append to a netCDF file in T.

def axis/t=`read1`:`read2`:1 taxis
def grid/t=taxis tgrid

! This should be the number of lines to skip at the start of your file.
let iskip=1

! Will write data only if it lies in this latitude-longitude box
let mask = if lat ge -75 and lat le 30 and lon ge 20 and lon le 120 then 1

! This variable will control writing
let out1 = 1

sp rm indian_ocean_profiles.nc
repeat/range=1:100 (go read_section)

! end of read_all.jnl

which calls this script,

! read_section.jnl
! call from read_all, which defines the mask and index variables.

file/grid=tgrid/skip=`iskip+read1-1`/var="index, dep, lat, lon, tmp, slt " "file.txt"

! How many points in the masked region?
let n_out = `mask[t=@ngd]`

! Set up to read the next  section
let read1 = `read1 + npts`

! If no points in the masked region, just exit
IF `n_out EQ 0` THEN exit/script

! Compress the data in the L direction to write
! just the data in the masked region.

let mask_lon = compressL(lon*mask)
let mask_lat = compressL(lat*mask)
let mask_dep = compressL(id*depth)
let mask_temp = compressL(id*tmp)
let mask_salt = compressL(id*slt)

! The output axis with the next n_out points
! Set up output variables with units and title

def axis/t=`out1`:`out1+n_out-1`:1 t_out_ax

let/title="longitude"/units="degrees_east" longitude = mask_lon[gt=t_out_ax@asn]
let/title="longitude"/units="degrees_east" latitude = mask_lat[gt=t_out_ax@asn]
let/title="Depth"/units="m" depth= mask_dep[gt=t_out_ax@asn]
let/title="salinity"/units="  " salt= mask_salt[gt=t_out_ax@asn]
let/title="temperature"/units="degrees C" temp= mask_temp[gt=t_out_ax@asn]

! Write all variables as the next n_out points in our output file
save/append/t=`out1`:`out1+n_out-1`/file=indian_ocean_profiles.nc latitude,longitude, id_ocean

! The start of the next output axis

let out1 = `out1+n_out`

! end of read_section.jnl

Now, in a new Ferret session, we can open up the dataset.  Because of the way we wrote out the data on a time axis, Ferret doesn't know if it's a regularly-spaced axis, so it's a good idea to set the axis that way.

yes? use/regulart indian_ocean_profiles.nc

and finally, this is scattered data in XY with profiles at various levels.  We do not have a function to put this data onto an XYZ grid.

On 4/17/2013 4:36 AM, golla nageswararao wrote:

Hi all,

I could solve this problem to some extent by doing gaussian interpolation technique

I used scat2gridgauss_xy function to map the data on profile number and depths.

But now the problem is data size.

I going to get dimension size of 202670x293.

I tried to split the data also, it is giving the following error.
**ERROR in efcn_compute() allocating 475058480 bytes of memory
    work array 3:  X=1:202670, Y=1:293, Z=1:1, T=1:1, E=1:1, F=1:1
 **ERROR: error in external function

The only way to decrease the data size is to extract the data for Indian Ocean domain (original is world oceans)

Can anybody suggest me how to extract for my domain interest (data set is also having longitude and latitude as variables), and compress the data into the reasonable size.

Thanks in advance

On Tue, Apr 16, 2013 at 8:59 PM, golla nageswararao <ezeenag4u@xxxxxxxxx> wrote:

Hi all,

I am having a dataset with variables depth, latitude, longitude, temp, salt in only one dimension (i.e., i=1:2600000). Actually the data consists of profiles at different locations.

i=1 z=0 temp1 salt1 lat1 lon1
i=2 z=20 temp1 salt1 lat1 lon1
i=3 z=50 temp1 salt1 lat1 lon1
i=4 z=100 temp1 salt1 lat1 lon1
i=5 z=250 temp1 salt1 lat1 lon1

i=6 z=0 temp2 salt2 lat2 lon2

i=7 z=15 temp2 salt2 lat2 lon2
i=8 z=25 temp2 salt2 lat2 lon2
i=9 z=45 temp2 salt2 lat2 lon2
i=10 z=65 temp2 salt2 lat2 lon2
.
.
.
.

How to handle this type of data? I want to extract profiles in Indian Ocean. I tried to do with "if" command. But the program is coming out of ferret after i=53. I dont know whats the problem. Can anybody help me in this regard.

Thanks in advance.

--
With Best regards,
G.NageswaraRao,

--
With Best regards,
G.NageswaraRao,

--
With Best regards,
G.NageswaraRao,

--
With Best regards,
G.NageswaraRao,

--
With Best regards,
G.NageswaraRao,