[Thread Prev][Thread Next][Index]

RE: [ferret_users] Data reading



Hi Karl,

 

You’re right the crash is due to the output, thank you!

However… I can’t get ferret to read the data right.

When I try to write to netcdf is crashes with a signal 11 – I guess again an output issue…

But:

I got a little fortran program to read the data (gfortran compiled)

I give this below and what I think the ferret set data should look like.

The stats command works, but the numbers are not right.

 

Now I am really confused. Maybe this is an issue with our installation.

Our machine was recently upgraded. Let me check on this before anything else.

 

Maybe you could just give me your opinion whether I got the right ferret “set data…” which does the same as the fortran code.

There should not be any count numbers for the records as the file size fits exactly with the data.

 

Thank you very much!

Kind regards,

Joerg

 

 

This bit of fortran reads the data fine (the numbers are between 0 and 300):

 

program test

real*4,dimension(227964) :: x

integer i

 

open(10,file="GB_1km_197101.bin",ACCESS="DIRECT",RECL=911856,CONVERT="BIG_ENDIAN" ) !  4 bytes reals

 

do i=1,6*31

print*,i,i*911856

read(10,REC=i) x

print*,minval(x),maxval(x)

enddo

 

end

 

 

In ferret I did (avoiding any issues with the 6 variables and 31 days I just use 186 “time” steps):

 

yes? define axis/x=1:227964:1 xa

yes? define axis/t=1:186:1 xt

yes? define grid/x=xa/t=xt xgr

yes? file/var=xx/grid=xgr/format=stream/type=r4/swap GB_1km_197101.bin

yes? show data

     currently SET data sets:

    1> ./GB_1km_197101.bin  (default)

name     title                             I         J         K         L         M         N

XX       XX                               1:227964  ...       ...       1:186     ...       ...

 

yes? stats xx[l=1]

             XX

             X: NaN to NaN

             Y:  N/A

             Z:  N/A

             T: NaN

             E:  N/A

             F:  N/A

             DATA SET: ./GB_1km_197101.bin

 Total # of data points: 227964 (227964*1*1*1*1*1)

# flagged as bad  data: 0

Minimum value: -3.6893E+19

Maximum value: 3.6893E+19

Mean    value: 2.3629E+16 (unweighted average)

Standard deviation: 1.0572E+19

 

Min and Max should be:

   4.38666296       51.8558731   

 

 

 

 

From: Karl Smith - NOAA Affiliate [mailto:karl.smith@xxxxxxxx]
Sent: 02 September 2013 08:52
To: Kaduk, Joerg (Dr.)
Cc: ferret_users@xxxxxxxx
Subject: Re: [ferret_users] Data reading

 

Hi Joerg,

 

The code at the line given in the error message is encountered when Ferret is trying the determine the best format for listing out your data as character (string) values to the console.  I am not sure why you are encountering this error (assuming that error message is accurate); maybe the magnitude of the data values, maybe the data was not interpreted correctly when read in.  (Some Fortran binary data file formats have a count value at the beginning that was either 4 bytes or 8 bytes, depending on version of gfortran used.)

 

But you might be able to work around this problem by not listing any values to the console and instead just saving all the data in a netcdf file after you read it into Ferret.  So after reading in the data, issue the Ferret command:

LIST /FORMAT=CDF /FILE=GB_1km_197101.nc SD, LD, TA, Q, W, P

Since this writes the data using a completely different method, you should not encounter the error you mentioned.  You can then cancel the data read in from the binary stream file, and use the netcdf file you just created for the data.  You probably want to use the Ferret "statistics" command on the dataset variables to make sure values appear to be interpreted correctly.  Then maybe listing some of the values.  If the error shows up again, you might try the ncdump command (from the shell prompt - not a ferret command).  Probably first just list the headers "ncdump -h", then maybe a full dump, to see if the data values appear to be correct.

 

Karl

 

 

On Sun, Sep 1, 2013 at 4:50 PM, Kaduk, Joerg (Dr.) <jk61@xxxxxxxxxxxxxxx> wrote:

Hello,

I have a reading problem:

I have a stream big endian r*4 file with 6 variables on 227964 land points for 31 days.

The data are ordered such that there are first all variables of day 1, then of day 2, …

I tried the below, but ferret crashed (see below) – with or without /swap.

This is:

                FERRET v6.84 

                Linux 2.6.32-220.23.1.el6.x86_64 64-bit - 08/13/13

 

I checked the file size and it fits with 4*6*227964*31

 

I could write fortran code to convert to netcdf, but it would be nice if I could do this via ferret….

Any help greatly appreciated.

 

Cheers,

Joerg

 

 

 

yes? define axis/x=1:227964:1 xa

yes? define axis/t=1:31:1 xt

yes? define grid/x=xa/t=xt xgr

yes? file/var=sd,ld,ta,q,w,p/grid=xgr/format=stream/type=r4/swap GB_1km_197101.bin

yes? show data

     currently SET data sets:

    1> ./GB_1km_197101.bin  (default)

name     title                             I         J         K         L         M         N

SD       SD                               1:227964  ...       ...       1:31      ...       ...

LD       LD                               1:227964  ...       ...       1:31      ...       ...

TA       TA                               1:227964  ...       ...       1:31      ...       ...

Q        Q                                1:227964  ...       ...       1:31      ...       ...

W        W                                1:227964  ...       ...       1:31      ...       ...

P        P                                1:227964  ...       ...       1:31      ...       ...

yes? SET MEMORY/SIZE=1000

Cached data cleared from memory

yes? list sd[i=1:100,l=1]

             VARIABLE : SD

             FILENAME : GB_1km_197101.bin

             SUBSET   : 100 points (X)

             T        : NaN

At line 72 of file var_data_fmt.F

Fortran runtime error: End of record

 

 

Jörg Kaduk

Senior lecturer

http://www.geog.le.ac.uk/staff/jk61/

t: +44 (0)116 252 3848

Department of Geography, University of Leicester, University Road, Leicester LE1 7RH, UK

 

Elite Without Being Elitist

 

Times Higher Awards Winner 2007, 2008, 2009, 2010, 2011, 2012, 2013

 



 

--
Karl M. Smith, Ph.D.
JISAO Univ. Wash. and TMAP/PMEL NOAA
"The contents of this message are mine personally and do
not necessarily reflect any position of the Government
or the National Oceanic and Atmospheric Administration."


[Thread Prev][Thread Next][Index]
Contact Us
Dept of Commerce / NOAA / OAR / PMEL / Ferret

Privacy Policy | Disclaimer | Accessibility Statement