Re: Reading Fortran data files

[Mark Verschell <markverschell@xxxxxxxxxxxxxx>]

(I put this back on the mailing list because I thought the general community would find the RESHAPE function useful - method 2 below, and also because Aiguo might need specific help using it since I am not that versed in it. Side note: maybe we need to build a little collection of Fortran file formats and how they were read in ferret <- Ferret Crew?)

Aiguo,

I have run across a lot of "creative" output formats in my career. What I have done is to convert almost everything to NetCDF (right from the beginning) unless I will only look at the data a few times. The speed up in accessing a large data base is incredible.

Your particular case will not be pretty. Two ways come to mind....

1) Simplest, but painfully boring
You need to define 300 Unix soft links to each file, e.g.:
ln -s data1.dat data1.001
ln -s data1.dat data1.002
to
ln -s data1.dat data1.300

Then, in ferret you would open a file for each obs:
FILE/SKIP=1/VAR=v001/FORMAT=STREAM data1.001
FILE/SKIP=64883/VAR=v002/FORMAT=STREAM data1.002
FILE/SKIP=129765/VAR=v003/FORMAT=STREAM data1.003
ad nauseum........
The skip comes from (360*180+80+2)*(obs-1)+1

And of course for data2.dat you would have to define 1 large variable that is 360*180*2 with the 2 being depth and z=1 being map1 and z=2 being map2 and the skip being (2*360*180+80+2)*(obs-1)+1

2) Tricky, but should work
I can't help you with the details in this one, but it should work
For data1.dat define your variable in ferret as 2D map1(x,t) where the x axis is
360*180+80 (80 for the title)
DEFINE AXIS/X=1:64880:1 xjunk

Then, you remap the good data into a 2d variable using the RESHAPE function (see the Ferret manual Ch3 Sec2.3.23) Something like:

SET REGION/x=81:64880
RESHAPE(v1d,v2d)
Where v1d is the 64880 long variable, and v2d is the 360*180 variable

For the data2.dat, again try doing the 1 big variable trick.

Good Luck, Mark


On Friday, July 20, 2001, at 01:06 PM, Aiguo Dai wrote:

> Thanks, Mark. The reason I brought this issue up is because that many
> large climate datasets have been put out in Fortran binary format
> ("write(1) title, dataMap" is a fairly common format). NetCDF
> format has become popular in just last several years.
>
> My question is how to read map1 and map2 the following Fortran binary files:
>
> parameter(mi=360,mj=180,nobs=300)
> character*80 title
> real  year, month, map1(mi,mj),map2(mi,mj)
>
> open(1,form='unformatted',file='data1.dat')
> open(2,form='unformatted',file='data2.dat')
>
>
> do 10 n=1,nobs
> write(1)title,map1
> write(2)title,year,month,map1,map2
> 10   continue
>
>
> Thanks again.
>
>
> Aiguo Dai                                    email: adai@ucar.edu
> Climate & Global Dynamics Division           phone: 303-497-1357
> National Center for Atmospheric Research     FAX  : 303-497-2699
> P.O. Box 3000, 1850 Table Mesa Drive
> Boulder, CO  80307
> homepage: http://www.cgd.ucar.edu/cas/adai/
>
>
>
> On Fri, 20 Jul 2001, Mark Verschell wrote:
>
> > Well, your letter is exactly why the ferret manual can't get into more
> > detail. There are an infinite number of convoluted ways of writing files
> > in ForTran (and many other programming languages). And thus, putting in
> > more than a few examples of the most straightforward examples would be a
> > wasted effort. Show one example and then a little change makes the
> > example invalid. Much better to write to NetCDF which can be done from a
> > ForTran program fairly easily and which would be a self-documenting file
> > in addition to being easily read by ferret.
> >
> > As to your particular question, I need more info, are you telling me
> > that you have a file that is really something like:
> >
> > <80 character title>
> > <optional numbers>
> > map1,map2,map3
> > <80 character title>
> > <optional numbers>
> > <optional numbers>
> > map1,map2,map3
> > (repeat)
> >
> > If so, this will take a lot of work to read in ferret and I would
> > suggest that you write a very small ForTran program that reformats the
> > file.
> >
> > Mark
> >
> > p.s. I replied just to you as I don't know if this would be of any
> > interest to the ferret_list in general. Maybe I need to write a little
> > FAQ on output choices.
> >
> >
> > On Friday, July 20, 2001, at 12:26 PM, Aiguo Dai wrote:
> >
> > > Mark,
> > >
> > > I hate to bother you again, but it seems that you may be the right
> > > person to
> > > ask about this:
> > >
> > > If there is a title of character*80 ahead of each of the record in my
> > > previous example, how could I skip the title (and sometimes a few
> > > additional
> > > real numbers in other cases)? Thank you very much in advance for your
> > > help.
> > >
> > > I think the Ferret mannual should add more info and examples on reading
> > > Fortran
> > > binary files.
> > >
> > >
> > > Aiguo Dai                                    email: adai@ucar.edu
> > > Climate & Global Dynamics Division           phone: 303-497-1357
> > > National Center for Atmospheric Research     FAX  : 303-497-2699
> > > P.O. Box 3000, 1850 Table Mesa Drive
> > > Boulder, CO  80307
> > > homepage: http://www.cgd.ucar.edu/cas/adai/
> > > ----- Original Message -----
> > > From: "Aiguo Dai" <adai@ucar.edu>
> > > To: <ferret_users@ferret.wrc.noaa.gov>
> > > Sent: Friday, July 20, 2001 2:08 AM
> > > Subject: Reading Fortran data files
> > >
> > >
> > > Hello,
> > >
> > > I was trying to read Fortran 77 files similar to the following one:
> > >
> > > parameter(mi=72,mj=45,msea=5)
> > > real map1(mi,mj),map2(mi,mj),map3(mi,mj)
> > >
> > > open(1,form='unformatted',file='data.dat')
> > >
> > > do is=1,msea
> > > write(1)map1,map2,map3
> > > enddo
> > >
> > > using Ferret scripts:
> > >
> > > def axis/x=180W:180E/npoint=72/unit="degrees_east" lon
> > > def axis/y=90S:90N/npoint=45/unit="degrees_north" lat
> > > def axis/t=1:5:1 time
> > > def grid/x=lon/y=lat/t=time g5x4
> > > file/format=unformatted/var=amp,pha,avg/col=9720/grid=g5x4 data.dat
> > > !
> > > set var/bad=-99999. amp
> > > set var/bad=-99999. pha
> > >
> > > However, the maps for amp[l=1], ..., amp[l=5], and pha are incorrect.
> > >
> > > Any suggestions would be appreciated for reading this types of data
> > > files (also for cases with only map1 and map2 written). Thanks lot.
> > >
> > > -Aiguo Dai, NCAR, Boulder, CO