Hello Russ,This is single model ncdump -h.On Mon, Jul 23, 2018 at 3:40 PM, Russ Fiedler <russell.fiedler@xxxxxxxx> wrote:Hi,
Ok, this looks like the time dimension is fixed rather than unlimited therefore you needed to set LLIMITS. This could be a good or bad thing...
If your original data has the time axis as a fixed dimension then this is working as expected. If it is an unlimited or record dimension I would say that this behaviour is incorrect. Ansley, could you confirm that?
Could you confirm from an "ncdump -h single_member.nc" that the ensemble is on a fixed time axis?
Cheers,
Russ
On 23/07/18 14:45, saurabh rathore wrote:
Dear Russ
Following steps working correctly but just for L=1 when it goes to L=2 it is showing following error.
!-> REPEAT: L=2
!-> REPEAT: MM:1
!-> REPEAT: KK:1
!-> LIST/FORMAT=CDF/append/file=hist_russ.nc/m=1/k=1:4 histzd
LISTing to file hist_russ.nc
**TMAP ERR: Requested data range is outside of data set limits
Expected: HISTZD[L= 2 : 2 ] Found: HISTZD[L= 1 : 1 ]
LIST/FORMAT=CDF/append/file=hist_russ.nc/m=1/k=1:4 histzd
Command file, command group, or REPEAT execution aborted
But if I put llimits also in this command
save/i=1/j=1/k=1/l=1/m=1/ilimits=1:1:360/jlimits=1:126/klimi ts=1:36/mlimits=1:11/llimits=1 :312/file=myfile.nc/clob myvar
then my code is running without any error. So is it saving the correct data ? for all the models and for all time steps ?
regards, saurabh
On Mon, Jul 23, 2018 at 1:59 PM, Russ Fiedler <russell.fiedler@xxxxxxxx> wrote:
Hi Saurabh,
If you really need to save it then you you can use multiple loops
! assume L is unlimited so we do not need to specify LLIMITS save by level to reduce memory.
save/i=1/j=1/k=1/l=1/m=1/ilimits=1:1:360/jlimits=1:126/klimi ts=1:36/mlimits=1:11/file=myfi le.nc/clob myvar
repeat/l=1:312 (repeat/range=1:11/name=mm (repeat/range=1:36/name=kk save/app/file=myfile.nc/m=`mm`/k=`kk` myvar))
To reduce file size you can set the variable to a 4 byte real rather than the default double precision.
set var/outtype=outtype myvar
Also I'd strongly recommend using deflation and chunking for lossless compression
SET LIST/XCHUNK=360/YCHUNK=126/ZCHUNK=1/TCHUNK=1/ECHUNK=1/DEFLAT E=1/SHUFFLE/NCFORMAT=4
If you want to change the chunking for the X,Y,Z and E dimensions then you should change the looping above to match
e.g. if ZCHUNK=9
! Save blocks of 9 levels
repeat/l=1:312 (repeat/range=1:11/name=mm (repeat/range=1:36:9/name=kk save/app/file=myfile.nc/m=`mm`/k=`kk`:`kk+8` myvar))
Cheers,
Russ
On 23/07/18 13:15, saurabh rathore wrote:
Dear Satyesh,
The way that you mentioned is more like
repeat/l=1:312:1 (save/append/file=ensemble_mean.nc var;set mem/size=xxxx)
but I am having 11 models so this size of saving data is not possible as in my final variable I am having 5 dimensions i.e. i,k,l,k,m where i=1:360, j=1:126, k=1:36, m=1:11. here m=1:11 are the 11 models
sh gr histzd
GRID (G013) Ensemble Aggregetion Grid
nameaxis # pts start end subset
LON LONGITUDE 360mr 0E 1W full
LAT28_153 LATITUDE 126 r 62.5S 62.5N full
LEV1_36 DEPTH (m) 36 i- 5 1984.9 full
HISTM TIME 312 r 30-JAN-1980 00:00 30-DEC-2005 00:00 full
ENSEMBLE E (realization) 11 r 1 11 fullnormal F
now how to save this massive amount of data ?
regards, saurabh
On Thu, Jul 19, 2018 at 10:29 AM, saurabh rathore <rohitsrb2020@xxxxxxxxx> wrote:
Dear Satyesh,
thanks for your email. I will get back to you after trying this method.
regards, saurabh
On Tue, Jul 17, 2018 at 9:31 PM, Satyesh Ghetiya <satyeshghetiya@xxxxxxxxx> wrote:
Hi Saurabh,
You can use 'save' command with 'repeat' loop, every time listing just one subset of the full ensemble variable. For example if your variable is 'var', find the 'I' index range (by 'sh data') and use the following:Suppose 'I' index ranges from 1 to 144, then your syntax will become as follows (you can further look what 'ilimits' and 'klimits' etc. does) :
save/file=Yourfile.nc/ilimits=1:144/i=1 var[d=1]; repeat/range=2:144/name=aa (save/file=Yourfile.nc/append var[d=1,i=`aa`;])
I hope this is relevant and helps you.
--
On Tue, Jul 17, 2018 at 1:38 PM, saurabh rathore <rohitsrb2020@xxxxxxxxx> wrote:
Dear Ferreters,
I am having 11 CMIP models data in lat,lon,time,depth. I made an ensemble variable of dimension lat,lon,depth,time,ensemble_member i.e. 360,180,36,312,11. which is 360 longitudes point, 180 latitudes points, 36 depth levels, 312 time steps, 11 no. of models.
i want to save this 5-dimension ensemble variable into netcdf file. Is there any efficient way to save as my HPC is limited by memory.
cheers, saurabh
--
REGARDS
Saurabh RathoreResearch Scholar (PhD.)Centre For Oceans, Rivers, Atmosphere & Land Science TechnologyIndian Institute Of Technology, Kharagpurcontact :- 91- 8345984434
Hyderabad.NRSC,OIBMD, OSG, ECSA,Ghetiya Satyeshkumar Girdharbhai,Scientist/Engineer 'SC',
--
REGARDS
Saurabh RathoreResearch Scholar (PhD.)Centre For Oceans, Rivers, Atmosphere & Land Science TechnologyIndian Institute Of Technology, Kharagpurcontact :- 91- 8345984434
--
REGARDS
Saurabh RathoreResearch Scholar (PhD.)Centre For Oceans, Rivers, Atmosphere & Land Science TechnologyIndian Institute Of Technology, Kharagpurcontact :- 91- 8345984434
--
REGARDS
Saurabh RathoreResearch Scholar (PhD.)Centre For Oceans, Rivers, Atmosphere & Land Science TechnologyIndian Institute Of Technology, Kharagpurcontact :- 91- 8345984434
--REGARDSSaurabh RathoreResearch Scholar (PhD.)Centre For Oceans, Rivers, Atmosphere & Land Science TechnologyIndian Institute Of Technology, Kharagpurcontact :- 91- 8345984434