Re: how to add the many netcdf files

[Jaison Kurian <jaison@xxxxxxxxxxxxxxxxxx>]

Hi, 
    
    Here is an easy way to append different files(more than 32) to a netcdf 
file (provided toal size is not > 2GB) using REPEAT command. 

  As a note of caution ; if the ferret version is 5.6, one can directly go for
                         REPEAT/RANGE=1:50 (i am not yet started using 5.6)

         for previous versions we need to take special care about the 
repeat "index".  In Tiwari's case the abstract axis "k" is "free".
So in this case we can use "k" as repeat index.
 
    ==> var(i,j,l) ==> repeat/k=1:50 (do_something_on var) 
 
        If all the axes are occupied; then we have to explicitly
mension the exact range we interested in for each variable along the 
abstract axis which is being used as the repeat "index".

    ==> var(i,j,k,l) ==> repeat/k=1:10 (do_something_on var[k=30:50]) 

  Suppose the name of data files are Data_Jan_1987.nc, Data_Feb_1987.nc 
and so on(file names from Tiwari's mail) and we want to combine 50 such files 
to a single file. This can be achieved very easily in a single ferret 
section.

    steps are....1. create file names 
                 2. open data set
                 3. save to output file
                 4. cancel data & memory (to overcome the limitation with 32
                                                 files)
                 5. repeat the steps 1-4 as many times we want.

-------------------------------------------------
!    sample file name ==>         Data_Jan_1987.nc

        let mnths = TSEQUENCE({"Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"}) 
! method described by Steve Hankin

  repeat/k=1:50 ( ;\
      let mon = IF `(k)/12 GT 1` THEN `k-12*int((k-1)/12)` ELSE `k` ;\
      let yr  = IF `(k)/12 GT 1` THEN `1987 + int((k-1)/12)` ELSE 1987 ;\
      repeat/l=`mon`:`mon` ( ;\
             define symbol mn = `mnths`  ;\ 
             ) ;\
      define symbol fname = Data_($mn)_`yr`.nc  ;\
      set data ($fname) ;\
      save/file=output.nc/CLOBBER/append var1, var2, var3  ;\   
      cancel data ($fname)  ;\
      cancel memory ;\
      )
-------------------------------------------------

hope this helps........

With Regards 

Jaison


On Tue, 18 May 2004, Ansley Manke wrote:

> Hi all,
> We determined that Yogesh needed to do this after appending
> each time step :
> 
>     cancel data input_file.nc
>     cancel memory
> 
> and then he was successful using all 50 files.
> 
> 
> Ansley Manke
> 
> "Yogesh K. Tiwari" wrote:
> 
> > Hello Ansley,
> >
> > I used the second option, as you suggested, SAVE/APPEND command to add
> > 50 netcdf files in to a single 'concat.cdf' file.
> >
> > My each file size is 762K.
> >
> > After adding only first 32 files ferret crashes on the following error:
> >
> > !--------------------------------------------------------------
> > add_connect: too many connects, cannot add another
> >   ** netCDF error: Attribute not found
> >   could not open CDF output file:
> >   concat.cdf
> > LIST/FORMAT=CDF/append/file=concat.cdf NBOX,CO2,STD_CO2
> > Command file, command group, or REPEAT execution aborted
> > !-----------------------------------------------------------
> >
> > Please help,
> >
> > Thanks,
> >
> > Yogesh
> >
> > > Hello Yogesh,
> > > This FAQ addresses a very similar question, and gives two different
> > > options; one is the descriptor file suggested by Steve Cousins in his
> > > earlier reply, and the other is to use Ferret's SAVE/APPEND command
> > > to make a single NetCDF file.
> > >
> > >   http://www.ferret.noaa.gov/Ferret/FAQ/data_management/multi_dataset.html
> > >
> > > Ansley Manke
> > >
> > > Yogesh K. Tiwari wrote:
> > >
> > >> Hello Dear Friends,
> > >>
> > >> I want to add  many  netcdf files in a one file inside ferret.
> > >>
> > >> My data files looks like :
> > >>
> > >> 1>./Data_Jan_1987.nc
> > >>
> > >> name     title                   I         J         K         L
> > >> CO2      CO2 concentration      1:360     1:180     ...       1:1
> > >>
> > >>
> > >> 2>./Data_Feb_1987.nc
> > >>
> > >> name     title                   I         J         K         L
> > >> CO2      CO2 concentration      1:360     1:180     ...       1:1
> > >>
> > >>
> > >>
> > >> 3>./Data_March_1987.nc
> > >>
> > >> name     title                   I         J         K         L
> > >> CO2      CO2 concentration      1:360     1:180     ...       1:1
> > >>
> > >>
> > >> -------------------------
> > >> -----------------
> > >> -------------
> > >>
> > >> There are many months for many years, total 50 files ----------------
> > >> --------------------------------------
> > >> --------------
> > >>
> > >>
> > >>
> > >> I want to add all the files in a single file so it should look like :
> > >>
> > >> >./Data_added.nc
> > >>
> > >> name     title                 I         J         K         L
> > >> CO2      CO2 concentration    1:360     1:180     ...       1:50
> > >>
> > >> Is it possible ??
> > >>
> > >> Or is there any other way that I can use all the files and plot
> > >> timeseries - seasonal cycle and other features.
> > >>
> > >> Thanks for your help,
> > >>
> > >> Regards,
> > >> Yogesh
> > >>
> > >>
> > >
> > >
> >
> > --
> >
> > =================================================================
> > Yogesh K. Tiwari,
> > Max-Planck Institute for Biogeochemistry,
> > Hans-Knoell-Strasse 10, D-07745 Jena,         Postal Address:
> > Germany                                       Postfach 10 01 64
> >                                                D-07701 Jena,
> > Office   : +49 3641 576376                    Germany
> > Fax      : +49 3641 577300
> > Home     : +49 3641 672232
> > Mobile   : +49 1736988789
> > e-mail   : yogesh.tiwari@bgc-jena.mpg.de
> >            __0                __0
> >           -\<,               -\<,
> >          (_)/(_)            (-)/(_)
> > =================================================================
> 

-- 
___________________________________________________

    Jaison Kurian                           
    Centre for Atmospheric and Oceanic Sciences
    Indian Institute of Science
    B A N G A L O R E   560 012
    Ph: +91-80-3942505
        +91-80-3600450
    Fax:+91-80-3600865
___________________________________________________