Hi Saurabh,
If you really need to save it then you you can use multiple loops
! assume L is unlimited so we do not need to specify LLIMITS save by level to reduce memory.
save/i=1/j=1/k=1/l=1/m=1/ilimits=1:1:360/jlimits=1:126/ klimits=1:36/mlimits=1:11/ file=myfile.nc/clob myvar
repeat/l=1:312 (repeat/range=1:11/name=mm (repeat/range=1:36/name=kk save/app/file=myfile.nc/m=`mm`/k=`kk` myvar))
To reduce file size you can set the variable to a 4 byte real rather than the default double precision.
set var/outtype=outtype myvar
Also I'd strongly recommend using deflation and chunking for lossless compression
SET LIST/XCHUNK=360/YCHUNK=126/ZCHUNK=1/TCHUNK=1/ECHUNK=1/ DEFLATE=1/SHUFFLE/NCFORMAT=4
If you want to change the chunking for the X,Y,Z and E dimensions then you should change the looping above to match
e.g. if ZCHUNK=9
! Save blocks of 9 levels
repeat/l=1:312 (repeat/range=1:11/name=mm (repeat/range=1:36:9/name=kk save/app/file=myfile.nc/m=`mm`/k=`kk`:`kk+8` myvar))
Cheers,
Russ
On 23/07/18 13:15, saurabh rathore wrote:
Dear Satyesh,
The way that you mentioned is more like
repeat/l=1:312:1 (save/append/file=ensemble_mean.nc var;set mem/size=xxxx)
but I am having 11 models so this size of saving data is not possible as in my final variable I am having 5 dimensions i.e. i,k,l,k,m where i=1:360, j=1:126, k=1:36, m=1:11. here m=1:11 are the 11 models
sh gr histzd
GRID (G013) Ensemble Aggregetion Grid
nameaxis # pts start end subset
LON LONGITUDE 360mr 0E 1W full
LAT28_153 LATITUDE 126 r 62.5S 62.5N full
LEV1_36 DEPTH (m) 36 i- 5 1984.9 full
HISTM TIME 312 r 30-JAN-1980 00:00 30-DEC-2005 00:00 full
ENSEMBLE E (realization) 11 r 1 11 fullnormal F
now how to save this massive amount of data ?
regards, saurabh
On Thu, Jul 19, 2018 at 10:29 AM, saurabh rathore <rohitsrb2020@xxxxxxxxx> wrote:
Dear Satyesh,
thanks for your email. I will get back to you after trying this method.
regards, saurabh
On Tue, Jul 17, 2018 at 9:31 PM, Satyesh Ghetiya <satyeshghetiya@xxxxxxxxx> wrote:
Hi Saurabh,
You can use 'save' command with 'repeat' loop, every time listing just one subset of the full ensemble variable. For example if your variable is 'var', find the 'I' index range (by 'sh data') and use the following:Suppose 'I' index ranges from 1 to 144, then your syntax will become as follows (you can further look what 'ilimits' and 'klimits' etc. does) :
save/file=Yourfile.nc/ilimits=1:144/i=1 var[d=1]; repeat/range=2:144/name=aa (save/file=Yourfile.nc/append var[d=1,i=`aa`;])
I hope this is relevant and helps you.
--
On Tue, Jul 17, 2018 at 1:38 PM, saurabh rathore <rohitsrb2020@xxxxxxxxx> wrote:
Dear Ferreters,
I am having 11 CMIP models data in lat,lon,time,depth. I made an ensemble variable of dimension lat,lon,depth,time,ensemble_member i.e. 360,180,36,312,11. which is 360 longitudes point, 180 latitudes points, 36 depth levels, 312 time steps, 11 no. of models.
i want to save this 5-dimension ensemble variable into netcdf file. Is there any efficient way to save as my HPC is limited by memory.
cheers, saurabh
--
REGARDS
Saurabh RathoreResearch Scholar (PhD.)Centre For Oceans, Rivers, Atmosphere & Land Science TechnologyIndian Institute Of Technology, Kharagpurcontact :- 91- 8345984434
Hyderabad.NRSC,OIBMD, OSG, ECSA,Ghetiya Satyeshkumar Girdharbhai,Scientist/Engineer 'SC',
--
REGARDS
Saurabh RathoreResearch Scholar (PhD.)Centre For Oceans, Rivers, Atmosphere & Land Science TechnologyIndian Institute Of Technology, Kharagpurcontact :- 91- 8345984434
--
REGARDS
Saurabh RathoreResearch Scholar (PhD.)Centre For Oceans, Rivers, Atmosphere & Land Science TechnologyIndian Institute Of Technology, Kharagpurcontact :- 91- 8345984434