[Thread Prev][Thread Next][Index]

Re: [ferret_users] scattered ASCII data to NetCDF



Hi Paulo,
When I was making a demo file to look into your problem I initially tried a delimited file with ";" as the delimiter. It gave what may be a similar problem to yours -- "show data" looked good but when I actually tried to use the data it wasn't right! Not having the time to figure it out I fell back on what I know works -- "file/form=free" where either blank space or the "," character work to separate values. The "," works too to isolate missing values as for example

   spawn '1,2,,4,6,7' > temp.file
   file/form=free/col=6/var=v temp.file
   list/nohead v
1   / 1:  1.000
2   / 2:  2.000
3   / 3:   ....
4   / 4:  4.000
5   / 5:  6.000
6   / 6:  7.000

So what I'd suggest is (unless I'm missing something obvious in using "/delim" in this application) that you replace all ";" in your data file with "," so as to hold missing values and use "file/form=free".
   sed 's/;/,/g'  chuva08.txt > chuva08.csv
Then with
   file/form=free/col=176/g=grd2/ord=tx/var=vin chuva08.csv
as the only change in what you wrote below, I'd hope you might have success.
Mick
-------
Paulo Henrique wrote:
Hi Mick

I would like to thank immensely for your elegant solution and well explained answer.

My file is 628 rows (points) by 176 columns (times) with (lon,lat) in the first two columns. I left separated by ";" because there's no data for some times. So, I done:

def axis/x=1:628:1 xax
def axis/t=1:174:1 tax
def grid/x=xax/t=tax grd

def axis/t=-1:174:1 tax2
def grid/x=xax/t=tax2 grd2

file/form=delim/delim=";"/col=176/g=grd2/ord=tx/var=vin chuva08.txt

let lon = vin[l=1]
let lat = vin[l=2]
let pcp = vin[g=grd]

Then, when I 'show data', I get:

yes? show data
     currently SET data sets:
    1> ./chuva08.txt  (default)
name title I J K L VIN VIN 1:628 ... ... 1:176

Everything seems Ok. But, When I try to list variables, I get

yes? list PCP
 **ERROR: illegal limits: T axis of VIN

And then, when I 'show data' again, I get

yes? show data
     currently SET data sets:
    1> ./chuva08.txt  (default)
name title I J K L VIN VIN 1:628 ... ... 1:1

Look, now I have only one "time step". It contains LONs. I tried spaces " " and commas "," using /form=free and bars "|" and semicolons ";" replacing /form=free by respective /form=delim/delim=, without success.

I think it should be working now. Any idea?


2008/6/26 mick spillane <Mick.Spillane@xxxxxxxx <mailto:Mick.Spillane@xxxxxxxx>>:

    Hi Paulo,
      The "/skip=" is intended to skip a number of records at the
    start of a file.  For problems like yours, which has a mixture of
    types of variables in a record (here lon,lat followed by a series
    of data values) I usually use regridding to separate them.
      To make the following work you should first replace the
    semicolons with blanks or commas as the delimiter so that you can
    use "file/form=free" to access the file.

    Suppose you have 50 records, each of which has the LON, LAT,
    [V(L),L=1,170], in your file.
    Then define a grid to hold ALL the V values:

      def axis/x=1:50:1 xax ; def axis/t=1:170:1 tax ; def
    grid/x=xax/t=tax grd

    Now the data file has two extra columns at the beginning which we
    treat as "fake" times in another grid:

      def axis/t=-1:50:1 tax2 ; def grid/x=xax/t=tax2 grd2

    (If the LON,LAT values were at the end of the record you would use
    "def axis/t=1:172:1" )
    Now read in everything on the grid grd2:

      file/form=free/col=172/g=grd2/ord=tx/var=vin data.file

    The variable "vin" has LON in L=1, LAT in L=2, and the V data in
    the remaining 170 columns. So

      let LON=vin[L=1] ; let LAT=vin[L=2]
      let V=vin[g=grd]

    Here is a quick demo where there are only 6 "times" and 4 records
    in the file test.data
    123.4 34.5 1 2 3 4 5 6
    124.5 35.6 9 8 7 6 5 4
    125.6 36.7 2 4 6 8 6 4
    126.7 37.8 1 3 5 7 5 3

    def axis/x=1:4:1 xax ; def axis/t=1:6:1 tax ; def grid/x=xax/t=tax grd
    def axis/t=-1:6:1 tax2 ; def grid/x=xax/t=tax2 grd2
    file/form=free/var=vin/g=grd2/col=8/ord=tx test.data
    list/nohead lon,lat

    1   / 1:  123.4  34.50
    2   / 2:  124.5  35.60
    3   / 3:  125.6  36.70
    4   / 4:  126.7  37.80

    list/nohead/ord=tx v
    1   / 1:  1.000  2.000  3.000  4.000  5.000  6.000
    2   / 2:  9.000  8.000  7.000  6.000  5.000  4.000
    3   / 3:  2.000  4.000  6.000  8.000  6.000  4.000
    4   / 4:  1.000  3.000  5.000  7.000  5.000  3.000

    Good luck,
      Mick
    --------------
    Paulo Henrique wrote:

        Hi Hanh,

        It seems to be skipping initial records in my file.

        I'll try to separate data by months. Maybe using bash/awk.

        Thanks!

        2008/6/26 <nguyen@xxxxxxxxxx <mailto:nguyen@xxxxxxxxxx>
        <mailto:nguyen@xxxxxxxxxx <mailto:nguyen@xxxxxxxxxx>>>:


           Paulo

           indeed I believe that it's limited to 100 columns so you
        may have
           to read
           your file twice. First you read the first 100 columns and
        then you
           use the
           option "/skip=100" to skip the first 100 columns to read the 76
           remaining
           columns.

           Hanh





           > Hi Hanh, it will be useful.
           >
           > My file has 176 rows where the two first are coordinates.
        I used
           >
           > file/form=delim/delim=";"/col=176 my_ascii_data.txt
           >
           > When I do 'show data' I got 100 variables sequentially
        named v1,
           v2, ...,
           > v100. Shall it be v1, v2, ..., v176, not? What's
        happening? Is
           it a Ferret
           > limitation or am I doing something wrong?
           >
           > Bill, your comment is surely valuable. I've used scat2grid
           taking care
           > about
           > errors produced by interpolation, but I hadn't thought
        any kind of
           > evaluation.
           >
           > Thanks for all attention!
           >
           > 2008/6/26 <nguyen@xxxxxxxxxx <mailto:nguyen@xxxxxxxxxx>
        <mailto:nguyen@xxxxxxxxxx <mailto:nguyen@xxxxxxxxxx>>>:

           >
           >> oups sory I didn't get your question
           >> well in this case you can check the section "2.5. ASCII
        data on
           line":
           >>
           >>
http://ferret.pmel.noaa.gov/Ferret/documentation/users-guide/data-set-basics/ASCII-DATA
           >> especially the option /column=ncol to define the number of
           columns you
           >> have.
           >> Hanh
           >>
           >>
           >> > Hanh,
           >> >
           >> > Thanks for your help.
           >> >
           >> > scat2grid is the following step of my job. This time,
        I just
           would
           >> like
           >> to
           >> > store my data without any interpolation.
           >> >
           >> > Or, at least, I would like to read these data without
           specifying 170
           >> > variables in my 'file/var="..."' command.
           >> >
           >> > 2008/6/26 <nguyen@xxxxxxxxxx
        <mailto:nguyen@xxxxxxxxxx> <mailto:nguyen@xxxxxxxxxx
        <mailto:nguyen@xxxxxxxxxx>>>:

           >> >
           >> >> Hi Paulo
           >> >>
           >> >> there is a function "scat2grid" that grids scattered
        data.
           Check the
           >> >> documentation for detailed explanations Ch3 Sec2.3:
           >> >>
           >>
http://ferret.pmel.noaa.gov/Ferret/documentation/users-guide/index/index-s
           >> >>
           >> >> Hanh
           >> >>
           >> >> > Hi Ferreters,
           >> >> >
           >> >> > I have some scattered precipitation data stored in
        a ASCII
           file
           >> with
           >> >> LON
           >> >> > in
           >> >> > first row, LAT in second row and with third to last
        rows being
           >> daily
           >> >> > accumulated rainfall relative to that (lon,lat)
        beginning at
           >> >> 01-jan-2008
           >> >> > separated by semicolons, e.g.:
           >> >> >
           >> >> > -39.05;-7.37;0;0;0;0;0;0;0;0;0;0;0;23;10;5;8 ;...
           >> >> >
        -40.12;-2.88;0;9.8;0;12.6;17.2;0;0;11;0;0;0;0;0;4.2;0 ;...
           >> >> > -39.58;-6.08;0;0;0;0;0;0;0;0;0;0;0;0;5;20;0 ;...
           >> >> > -40.12;-6.57;0;0;0;3;0;0;0;0;0;0;0;0;0;0;13 ;...
           >> >> > -40.55;-3.58;0;0;0;0;0;0;0;6;0;0;0;0;0;0;4 ;...
           >> >> >
        -39.73;-7;0;0;0;25.9;0;0;0;0.8;0;0;0;0;2.8;9.4;9.6;0 ;...
           >> >> > -38.25;-5.52;0;0;0;0;0;0;0;0;0;0;0;0;2;0;2.2;...
           >> >> > .
           >> >> > .
           >> >> > .
           >> >> >
           >> >> > The full dataset covers about 170 days and, hence,
        we may
           expect
           >> about
           >> >> 170
           >> >> > rows. I know this procedure to read ASCII data:
           >> >> >
           >> >> >
file/form=delim/delim="<delim>"/var="lon,lat,day1,day2,day3,...,dayN"
           >> >> > my_ascii_data.txt
           >> >> >
           >> >> > But it seems to be not really useful in my case.
           >> >> >
           >> >> > Is there a way to produce a NC file with one variable
           ('precip') in
           >> a
           >> >> > regular time axis and irregular XY axis?
           >> >> >
           >> >> > If not, should be useful a solution to read this
        data without
           >> >> specifying
           >> >> > one
           >> >> > variable for each day (say, using file/var="<list
        of vars>"
           >> >> my_data.txt).
           >> >> >
           >> >> > Well... to finish, I would assure a nice reward for
        that
           who first
           >> >> help
           >> >> to
           >> >> > solve this problem!
           >> >> >
           >> >> > Thanks
           >> >> >
           >> >> >
           >> >> > --
           >> >> > Ms. Paulo Henrique Santiago de Maria
           >> >> > Grupo de Modelagem Atmosférica
           >> >> > Departamento de Meteorologia e Oceanografia
           >> >> > Fundação Cearense de Meteorologia e Recursos Hídricos
           >> >> > Av. Rui Barbosa 1246 - CEP 60115-221
           >> >> > Fortaleza, Ceará
           >> >> > Fone: (85) 3101-1106 / 3101-1126
           >> >> >
           >> >>
           >> >>
           >> >
           >> >
           >> > --
           >> > Ms. Paulo Henrique Santiago de Maria
           >> > Grupo de Modelagem Atmosférica
           >> > Departamento de Meteorologia e Oceanografia
           >> > Fundação Cearense de Meteorologia e Recursos Hídricos
           >> > Av. Rui Barbosa 1246 - CEP 60115-221
           >> > Fortaleza, Ceará
           >> > Fone: (85) 3101-1106 / 3101-1126
           >> >
           >>
           >>
           >
           >
           > --
           > Ms. Paulo Henrique Santiago de Maria
           > Grupo de Modelagem Atmosférica
           > Departamento de Meteorologia e Oceanografia
           > Fundação Cearense de Meteorologia e Recursos Hídricos
           > Av. Rui Barbosa 1246 - CEP 60115-221
           > Fortaleza, Ceará
           > Fone: (85) 3101-1106 / 3101-1126
           >




-- Ms. Paulo Henrique Santiago de Maria
        Grupo de Modelagem Atmosférica
        Departamento de Meteorologia e Oceanografia
        Fundação Cearense de Meteorologia e Recursos Hídricos
        Av. Rui Barbosa 1246 - CEP 60115-221
        Fortaleza, Ceará
        Fone: (85) 3101-1106 / 3101-1126





--
Ms. Paulo Henrique Santiago de Maria
Grupo de Modelagem Atmosférica
Departamento de Meteorologia e Oceanografia
Fundação Cearense de Meteorologia e Recursos Hídricos
Av. Rui Barbosa 1246 - CEP 60115-221
Fortaleza, Ceará
Fone: (85) 3101-1106 / 3101-1126



[Thread Prev][Thread Next][Index]

Contact Us
Dept of Commerce / NOAA / OAR / PMEL / TMAP

Privacy Policy | Disclaimer | Accessibility Statement