[Thread Prev][Thread Next][Index]

Re: [ferret_users] saving the file of ensemble members



Dear Russ

Following steps working correctly but just for L=1 when it goes to L=2 it is showing following error.

!-> REPEAT: L=2
!-> REPEAT: MM:1
!-> REPEAT: KK:1
 !-> LIST/FORMAT=CDF/append/file=hist_russ.nc/m=1/k=1:4 histzd
 LISTing to file hist_russ.nc
 **TMAP ERR: Requested data range is outside of data set limits
             Expected: HISTZD[L=           2 :           2 ] Found: HISTZD[L=           1 :           1 ]
LIST/FORMAT=CDF/append/file=hist_russ.nc/m=1/k=1:4 histzd
Command file, command group, or REPEAT execution aborted

But if I put llimits also in this command

save/i=1/j=1/k=1/l=1/m=1/ilimits=1:1:360/jlimits=1:126/klimits=1:36/mlimits=1:11/llimits=1:312/file=myfile.nc/clob myvar

then my code is running without any error. So is it saving the correct data ? for all the models and for all time steps ?

regards, saurabh

On Mon, Jul 23, 2018 at 1:59 PM, Russ Fiedler <russell.fiedler@xxxxxxxx> wrote:

Hi Saurabh,

If you really need to save it then you you can use multiple loops

! assume L is unlimited so we do not need to specify LLIMITS save by level to reduce memory.

save/i=1/j=1/k=1/l=1/m=1/ilimits=1:1:360/jlimits=1:126/klimits=1:36/mlimits=1:11/file=myfile.nc/clob myvar
repeat/l=1:312 (repeat/range=1:11/name=mm (repeat/range=1:36/name=kk save/app/file=myfile.nc/m=`mm`/k=`kk` myvar))


To reduce file size you can set the variable to a 4 byte real rather than the default double precision.

set var/outtype=outtype myvar

Also I'd strongly recommend using deflation and chunking for lossless compression

SET LIST/XCHUNK=360/YCHUNK=126/ZCHUNK=1/TCHUNK=1/ECHUNK=1/DEFLATE=1/SHUFFLE/NCFORMAT=4

If you want to change the chunking for the X,Y,Z and E dimensions  then you should change the looping above to match

e.g. if ZCHUNK=9

! Save blocks of 9 levels
repeat/l=1:312 (repeat/range=1:11/name=mm (repeat/range=1:36:9/name=kk save/app/file=myfile.nc/m=`mm`/k=`kk`:`kk+8` myvar))


Cheers,
Russ


On 23/07/18 13:15, saurabh rathore wrote:
Dear Satyesh,

The way that you mentioned is more like

repeat/l=1:312:1 (save/append/file=ensemble_mean.nc var;set mem/size=xxxx)

but I am having 11 models so this size of saving data is not possible as in my final variable I am having 5 dimensions i.e. i,k,l,k,m where i=1:360, j=1:126, k=1:36, m=1:11. here m=1:11 are the 11 models

sh gr histzd
    GRID (G013)  Ensemble Aggregetion Grid
 name                           axis              # pts                      start                      end                           subset
 LON                      LONGITUDE          360mr                  0E                           1W                            full
 LAT28_153             LATITUDE           126 r                    62.5S                       62.5N                        full
 LEV1_36                  DEPTH (m)           36 i-                     5                           1984.9                      full
 HISTM                        TIME               312 r               30-JAN-1980 00:00     30-DEC-2005 00:00       full
 ENSEMBLE           E (realization)        11 r                             1                             11                      full
 normal    F

now how to save this massive amount of data ?

regards, saurabh


On Thu, Jul 19, 2018 at 10:29 AM, saurabh rathore <rohitsrb2020@xxxxxxxxx> wrote:
Dear Satyesh,

thanks for your email. I will get back to you after trying this method. 

regards, saurabh

On Tue, Jul 17, 2018 at 9:31 PM, Satyesh Ghetiya <satyeshghetiya@xxxxxxxxx> wrote:
Hi Saurabh,

  You can use 'save' command with 'repeat' loop, every time listing just one subset of the full ensemble variable. For example if your variable is 'var', find the 'I' index range (by 'sh data') and use the following: 
 Suppose 'I' index ranges from 1 to 144, then your syntax will become as follows (you can further look what 'ilimits' and 'klimits' etc. does)  :

save/file=Yourfile.nc/ilimits=1:144/i=1 var[d=1];
repeat/range=2:144/name=aa (save/file=Yourfile.nc/append var[d=1,i=`aa`;])

I hope this is relevant and helps you.


On Tue, Jul 17, 2018 at 1:38 PM, saurabh rathore <rohitsrb2020@xxxxxxxxx> wrote:
Dear Ferreters,

I am having 11 CMIP models data in lat,lon,time,depth. I made an ensemble variable of dimension lat,lon,depth,time,ensemble_member i.e. 360,180,36,312,11. which is 360 longitudes point, 180 latitudes points, 36 depth levels, 312 time steps, 11 no. of models.

i want to save this 5-dimension ensemble variable into netcdf file. Is there any efficient way to save as my HPC is limited by memory.

cheers, saurabh

--


REGARDS

Saurabh Rathore
Research Scholar (PhD.)
Centre For Oceans, Rivers, Atmosphere & Land Science Technology
Indian Institute Of Technology, Kharagpur
contact :- 91- 8345984434



--
Ghetiya Satyeshkumar Girdharbhai,
Scientist/Engineer 'SC',
OIBMD, OSG, ECSA,
NRSC,
Hyderabad.



--


REGARDS

Saurabh Rathore
Research Scholar (PhD.)
Centre For Oceans, Rivers, Atmosphere & Land Science Technology
Indian Institute Of Technology, Kharagpur
contact :- 91- 8345984434



--


REGARDS

Saurabh Rathore
Research Scholar (PhD.)
Centre For Oceans, Rivers, Atmosphere & Land Science Technology
Indian Institute Of Technology, Kharagpur
contact :- 91- 8345984434





--


REGARDS

Saurabh Rathore
Research Scholar (PhD.)
Centre For Oceans, Rivers, Atmosphere & Land Science Technology
Indian Institute Of Technology, Kharagpur
contact :- 91- 8345984434

[Thread Prev][Thread Next][Index]
Contact Us
Dept of Commerce / NOAA / OAR / PMEL / Ferret

Privacy Policy | Disclaimer | Accessibility Statement