[Thread Prev][Thread Next][Index]
Re: Concatenating netcdf files along time dim
A Demo of the Multi-netCDF alternative to combining files
---------------------------------------------------------
Hi again Kek,
There has been a flurry of activity in response to your query
as I bet it is a situation that many ferreters encounter.
My previous response (and other respondees) addressed the
situation where you actually desire the combined datafile that
includes the data from all the individual timestep files.
If you don't NEED the combined file but just WANT ACCESS to
the data stored in the individual files that is a situation where
the multi-netcdf descriptor file that Ansley refers to is useful. A
small ASCII descriptor file can give access to many huge individual
netcdf files.
Below is a demonstration descriptor file that gives access
to the data in 9 individual netcdf files that share the same x,y,z
axes but each represent a different time
test001.cdf : 1-jan-2003
test002.cdf : 2-jan-2003
...
test009.cdf : 9-jan-2003
Here is a script that could generate those individual netcdf files
def axis/x=1:4:1 xax ; def axis/y=1:3:1 yax ; def axis/z=1:2:1 zax
def axis/t=1-jan-2003:1-jan-2003:1/t0=1-jan-2003/units=days tax
def grid/x=xax/y=yax/z=zax/t=tax grd
let/title="TestVar" v=x[g=grd]+10*y[g=grd]+100*z[g=grd]+0*t[g=grd]+1000
save/file=test001.cdf v
def axis/t=2-jan-2003:2-jan-2003:1/t0=2-jan-2003/units=days tax
let/title="TestVar" v=x[g=grd]+10*y[g=grd]+100*z[g=grd]+0*t[g=grd]+2000
save/file=test002.cdf v
def axis/t=3-jan-2003:3-jan-2003:1/t0=3-jan-2003/units=days tax
let/title="TestVar" v=x[g=grd]+10*y[g=grd]+100*z[g=grd]+0*t[g=grd]+3000
save/file=test003.cdf v
def axis/t=4-jan-2003:4-jan-2003:1/t0=4-jan-2003/units=days tax
let/title="TestVar" v=x[g=grd]+10*y[g=grd]+100*z[g=grd]+0*t[g=grd]+4000
save/file=test004.cdf v
def axis/t=5-jan-2003:5-jan-2003:1/t0=5-jan-2003/units=days tax
let/title="TestVar" v=x[g=grd]+10*y[g=grd]+100*z[g=grd]+0*t[g=grd]+5000
save/file=test005.cdf v
def axis/t=6-jan-2003:6-jan-2003:1/t0=6-jan-2003/units=days tax
let/title="TestVar" v=x[g=grd]+10*y[g=grd]+100*z[g=grd]+0*t[g=grd]+6000
save/file=test006.cdf v
def axis/t=7-jan-2003:7-jan-2003:1/t0=7-jan-2003/units=days tax
let/title="TestVar" v=x[g=grd]+10*y[g=grd]+100*z[g=grd]+0*t[g=grd]+7000
save/file=test007.cdf v
def axis/t=8-jan-2003:8-jan-2003:1/t0=8-jan-2003/units=days tax
let/title="TestVar" v=x[g=grd]+10*y[g=grd]+100*z[g=grd]+0*t[g=grd]+8000
save/file=test008.cdf v
def axis/t=9-jan-2003:9-jan-2003:1/t0=9-jan-2003/units=days tax
let/title="TestVar" v=x[g=grd]+10*y[g=grd]+100*z[g=grd]+0*t[g=grd]+9000
save/file=test009.cdf v
The descriptor file is called "multi.des" and to access it from ferret
> ferret
yes? use multi ! accesses data from all 9 netcdf files
yes? show data
currently SET data sets:
1> ./multi.des (default)
name title I J K L
V TestVar 1:4 1:3 1:2 1:9
yes? plot/i=1/j=1/k=1 v
yes? use combo.cdf ! access the same data from the combined
! file of the earlier demo
yes? plot/i=1/j=1/k=1/o vall ! confirms that both approaches
! are equivalent
----------------------------------
The demo descriptor file multi.des :
Note that each record begins with a blank character. Older versions
of ferret were a bit picky about that. The time unit is 86400 since
these are daily data and the time values that follow are given as
0,1,2,... (i.e. times in days from the time origin.) Some of the records
in multi.des may not be necessary - due to the earlier pickiness I have a
.des file that works and just edit it for new situations.
Good luck,
Mick
*****************************************************************************
* Science Applications Support at NOAA/PMEL, Seattle, WA. *
* adapted from a TMAP descriptor file *
*****************************************************************************
$FORMAT_RECORD
D_TYPE = ' MC',
D_FORMAT = ' 1A',
D_SOURCE_CLASS = 'GRIDDED DATA',
D_SOURCE = 'Demonstration files',
$END
$BACKGROUND_RECORD
D_EXPNUM = ' ',
D_MODNUM = ' ',
D_TITLE = 'Multi-netCDF demo',
D_FORCING_SET = ' ',
D_T0TIME = '1-JAN-2003:00:00',
D_TIME_UNIT = 86400.0,
D_TIME_MODULO = .false.,
D_NTEGRATE_TSTEP = 1.0,
D_TIME_RUN = ' ',
D_WHO_RANIT = 'spillane@pmel.noaa.gov',
D_TIME_MADE = '9-sep-2003',
D_WHO_MADEIT = 'Mick Spillane, SCIAPP',
D_AUX_MODNUM = 6*' ',
D_ADD_PARM = 15*' ',
D_GRID_FILENAME = 6*' ',
$END
$MESSAGE_RECORD
D_MESSAGE = ' ',
D_ALERT_ON_OPEN = F,
D_ALERT_ON_OUTPUT = F,
$END
**************************************************
$EXTRA_RECORD
$END
$STEPFILE_RECORD
S_FILENAME = 'test001.cdf',
S_AUX_SET_NUM = 0,
S_START = 0,
S_END = 0,
S_DELTA = 1.0,
S_NUM_OF_FILES = 1,
S_REGVARFLAG = ' ',
$END
**************************************************
$EXTRA_RECORD
$END
$STEPFILE_RECORD
S_FILENAME = 'test002.cdf',
S_AUX_SET_NUM = 0,
S_START = 1,
S_END = 1,
S_DELTA = 1.0,
S_NUM_OF_FILES = 1,
S_REGVARFLAG = ' ',
$END
**************************************************
$EXTRA_RECORD
$END
$STEPFILE_RECORD
S_FILENAME = 'test003.cdf',
S_AUX_SET_NUM = 0,
S_START = 2,
S_END = 2,
S_DELTA = 1.0,
S_NUM_OF_FILES = 1,
S_REGVARFLAG = ' ',
$END
**************************************************
$EXTRA_RECORD
$END
$STEPFILE_RECORD
S_FILENAME = 'test004.cdf',
S_AUX_SET_NUM = 0,
S_START = 3,
S_END = 3,
S_DELTA = 1.0,
S_NUM_OF_FILES = 1,
S_REGVARFLAG = ' ',
$END
**************************************************
$EXTRA_RECORD
$END
$STEPFILE_RECORD
S_FILENAME = 'test005.cdf',
S_AUX_SET_NUM = 0,
S_START = 4,
S_END = 4,
S_DELTA = 1.0,
S_NUM_OF_FILES = 1,
S_REGVARFLAG = ' ',
$END
**************************************************
$EXTRA_RECORD
$END
$STEPFILE_RECORD
S_FILENAME = 'test006.cdf',
S_AUX_SET_NUM = 0,
S_START = 5,
S_END = 5,
S_DELTA = 1.0,
S_NUM_OF_FILES = 1,
S_REGVARFLAG = ' ',
$END
**************************************************
$EXTRA_RECORD
$END
$STEPFILE_RECORD
S_FILENAME = 'test007.cdf',
S_AUX_SET_NUM = 0,
S_START = 6,
S_END = 6,
S_DELTA = 1.0,
S_NUM_OF_FILES = 1,
S_REGVARFLAG = ' ',
$END
**************************************************
$EXTRA_RECORD
$END
$STEPFILE_RECORD
S_FILENAME = 'test008.cdf',
S_AUX_SET_NUM = 0,
S_START = 7,
S_END = 7,
S_DELTA = 1.0,
S_NUM_OF_FILES = 1,
S_REGVARFLAG = ' ',
$END
**************************************************
$EXTRA_RECORD
$END
$STEPFILE_RECORD
S_FILENAME = 'test009.cdf',
S_AUX_SET_NUM = 0,
S_START = 8,
S_END = 8,
S_DELTA = 1.0,
S_NUM_OF_FILES = 1,
S_REGVARFLAG = ' ',
$END
**************************************************
$STEPFILE_RECORD
S_FILENAME = '**END OF STEPFILES**'
$END
**************************************************
[Thread Prev][Thread Next][Index]
Dept of Commerce /
NOAA /
OAR /
PMEL /
TMAP
Contact Us | Privacy Policy | Disclaimer | Accessibility Statement