[Thread Prev][Thread Next][Index]

Re: [ferret_users] saving the file of ensemble members



Hi,

Thank you for writing back. 

For all of us in general, I'd like to suggest that unless a person needs a single file, perhaps for use with other software, the best practice for working with them in Ferret is to leave the ensemble members as separate files and use the on-the-fly aggregation. As you found, the file gets very big, and you lose flexibility, if for instance you find that there is a new version of one of the member  model runs and it should be replaced.

There is a newer syntax, available in Ferret v7.4 and enhanced in the upcoming release Ferret v7.42, which lets you put all the commands needed to define any aggregation (including Ensembles, Timeseries, FMRC's and Unions) into a script, and then give the "USE " command with the script name, as if the script is a dataset name.  So make a script  - this is a shortened version of the script below, and name the file histens.agg:


!! ------------
!!
histens.agg script defining the ensemble of models


use model1.nc
use model2.nc
use model3.nc

! change the variable name for ensembling them

set var/name=histbcc HISTZD[d=2]
set var/name=histccs HISTZD[d=3]

! regrid them for the same lat, lon, depth and time
! in my case only depth was different so i regrid it to depth only
! I regrid with respect to my first data set you can choose your own data set for regredding

let/d=2 histzd=histbcc[gz=HISTZD[d=1]@nrst]
let/d=3 histzd=histccs[gz=HISTZD[d=1]@nrst]

! make them into ensemble

ensemble histens=1,2,3

!! end of histens.agg script defining the ensemble
!! ------------

Then your Ferret session would be:

    yes? use histens.agg
    yes? show data
...




On 7/23/2018 11:29 PM, saurabh rathore wrote:
Dear All,

First of all thank you Russ, Satyesh and Ansley

I am sharing what I did to save all the models into one file as suggested by Russ, Satyesh and Ansley and its worked fine for me. Hope this will help you as well.


!!!!!!!!!!!!!! suppose you have 11 models !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!! all the models are having same variable !!!!!!!!!!!!!!!!!


use model1.nc
use model2.nc
use "
use "
use "
use "
use "
use "
use "
use "
use model11



!!!!!!!!!!!!!!!! change the variable name for ensembling them !!!!!!!!!!!!!!

set var/name=histbcc HISTZD[d=2]
set var/name=histccs HISTZD[d=3]
set var/name=histcmc HISTZD[d=4]
set var/name=histcnr HISTZD[d=5]
set var/name=histcsi HISTZD[d=6]
set var/name=histece HISTZD[d=7]
set var/name=histgfd HISTZD[d=8]
set var/name=histmpi HISTZD[d=9]
set var/name=histmri HISTZD[d=10]
set var/name=histnor HISTZD[d=11]



!!!!!!!!!!!!!!!!!!! regrid them for the same lat, lon, depth and time !!!!!!!!!!!!!

!!!!!!! in my case only depth was different so i regrid it to depth only !!!!!!!!!!

!!!!!!!!!!! I regrid with respect to my first data set you can choose your own data set for regredding !!!!!!


let/d=2 histzd=histbcc[gz=HISTZD[d=1]@nrst]
let/d=3 histzd=histccs[gz=HISTZD[d=1]@nrst]
let/d=4 histzd=histcmc[gz=HISTZD[d=1]@nrst]
let/d=5 histzd=histcnr[gz=HISTZD[d=1]@nrst]
let/d=6 histzd=histcsi[gz=HISTZD[d=1]@nrst]
let/d=7 histzd=histece[gz=HISTZD[d=1]@nrst]
let/d=8 histzd=histgfd[gz=HISTZD[d=1]@nrst]
let/d=9 histzd=histmpi[gz=HISTZD[d=1]@nrst]
let/d=10 histzd=histmri[gz=HISTZD[d=1]@nrst]
let/d=11 histzd=histnor[gz=HISTZD[d=1]@nrst]



!!!!!!!!!!!!!!! make them into ensemble !!!!!!!!!!!!!!!!

!!!!!!!!!! numbers represents number of ensemble members i.e. 11 in my case !!!!!!!!!!!!!!

!!!!!!!!!! but the name of ensemble variable will we same as original variable to save into netcdf file !!!!!!!!!!



ensemble histens=1,2,3,4,5,6,7,8,9,10,11


!!!!!!!!!!!!!!! it will show you ensemble variable attribute !!!!!!!!!!!!!!!!!!

sh da

sh gr histzd


!!!!!!!!!!!!!!!!! chunck them to improve efficieny as discussed by russ in email !!!!!!!!!!!!!!!

!!!!!!!!!! I chuncked them into 4 depth levels and 1 year !!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!! 1 year because I was haivng monthly time series so I chuncked into 12 months !!!!!!!!!

!!!!!!!!!! I was having 312 monthly time steps (26 years) and 36 vertical depth levels !!!!!!!!!!!!!!!!!!



set list/xchunk=360/ychunk=126/zchunk=9/tchunk=12/echunk=1/deflate=1/shuffle/ncformat=4



!!!!!!!!!!!!!!!!!!!!!! define limits of each axes !!!!!!!!!!!!!!!!!!!!!!!!!!



save/i=1/j=1/k=1/l=1/m=1/ilimits=1:360/jlimits=1:126/klimits=1:36/llimits=1:312/mlimits=1:11/file=hist_russ.nc/clob histzd



!!!!!!!!!!!!!!!!!!!!!!!!! save the ensemble data by a nested loop !!!!!!!!!!!!!!!!!!!!!!!!!!



repeat/range=1:312:12/name=ll (repeat/range=1:11/name=mm (repeat/range=1:36:9/name=kk save/append/file=hist_russ.nc/l=`ll`:`ll+11`/m=`mm`/k=`kk`:`kk+8` histzd;set mem/size=5000);set mem/size=9000)



!!!!!!!!!!!!! that is done !!!!!!!!!!!!!!!!!!!!!!!

exit


Hope this will help you.

Cheers, Saurabh

On Tue, Jul 24, 2018 at 3:41 AM, Ansley C. Manke <ansley.b.manke@xxxxxxxx> wrote:

Hi,

When you do a SAVE command on an ensemble dataset, Ferret issues a NOTE :

yes? ensemble fourfiles =  ens1, ens3, ens2, ens4
yes? show data
     currently SET data sets:
    5> fourfiles  (default)  Ensemble aggregation
 name     title          I         J         K         L         M         N
 SST      SST           1:10      1:9       ...       1:12      1:4       ...
 
yes? save/elimits=1:4/L=1:4/file=test.nc sst
 LISTing to file test.nc
           *** NOTE: Saving E or F axis with time axis uses /RIGID in time

This is telling us that the saving data on a grid which has both a time axis and E or F axes saves the time axis as a fixed-length axis. So yes, you will need to use /LLIMITS along with /MLIMITS when first writing to the file. We have not implemented the full NetCDF-4 standard which allows for all axes to be unlimited, and in this situation have in fact lost that capability with time axes as well.

I have added to the Ferret documentation to make this more clear.

Ansley


On 7/22/2018 11:36 PM, saurabh rathore wrote:
Thank you Russ and Satyesh,

I was misinterpreting the chunking. I will give one more try tomorrow and if all works fine I will provide the script with proper description for users.

Cheers, Saurabh

On Mon, Jul 23, 2018 at 4:30 PM, Satyesh Ghetiya <satyeshghetiya@xxxxxxxxx> wrote:
Hi all,
  Actually I also encountered the same error for another 4 dimensional data from NCEP,
TMAP ERR: Requested data range is outside of data set limits
             Expected: HISTZD[L=           2 :           2 ] Found: HISTZD[L=           1 :           1 ]
LIST/FORMAT=CDF/append/file=hist_russ.nc/m=1/k=1:4 histzd)     )   while working with listing data. That time I was using nested double loop. Then I did try and error to finally know that listing was working fine (without the Expected : [l=2:2] and Found: .... error)  with single loop. I could not identify the solution for now. I think double loop is making problem as the single loop is working fine. 

On Mon, Jul 23, 2018 at 11:40 AM, saurabh rathore <rohitsrb2020@xxxxxxxxx> wrote:
Yes, Its is saving each time step in a repeat loop. like l=1, l=2,.....so on. I tried using chunking in time steps but the result was strange and odd. I used these sets of instructions.

set list/xchunk=360/ychunk=126/zchunk=9/tchunk=6/echunk=1/deflate=1/shuffle/ncformat=4

(save/i=1/j=1/k=1/l=1/m=1/ilimits=1:360/jlimits=1:126/klimits=1:36/mlimits=1:11/llimits=1:312/file=hist_russ2.nc/clob histzd;set mem/size=9000)

repeat/range=1:312:6/name=ll (repeat/range=1:11/name=mm (repeat/range=1:36:9/name=kk save/append/file=hist_russ2.nc/l=`ll`:`ll+51`/m=`mm`/k=`kk`:`kk+3` histzd;set mem/size=9000);set mem/size=9000)

I dont know where is the mistake in time chuncking ?

Saurabh

On Mon, Jul 23, 2018 at 4:05 PM, Russ Fiedler <russell.fiedler@xxxxxxxx> wrote:


Hmm, that's odd. It's definitely an unlimited time dimension. I'm surprised that you need to specify LLIMITS. Maybe it's some strange interaction with ensembles.
If the saved file has an unlimited time dimension then all should be fine. It's a bit of a pain if you need to extend the time axis at any stage.

Russ


On 23/07/18 15:49, saurabh rathore wrote:
Here time axis is HISTM as I regridded it onto 360_days calender [gt=histm@asn]

On Mon, Jul 23, 2018 at 3:47 PM, saurabh rathore <rohitsrb2020@xxxxxxxxx> wrote:
​Hello Russ,

This is single model ncdump -h.​

On Mon, Jul 23, 2018 at 3:40 PM, Russ Fiedler <russell.fiedler@xxxxxxxx> wrote:
Hi,

Ok, this looks like the time dimension is fixed rather than unlimited therefore you needed to set LLIMITS. This could be a good or bad thing...
If your original data has the time axis as a fixed dimension then this is working as expected. If it is an unlimited or record dimension I would say that this behaviour is incorrect. Ansley, could you confirm that?

Could you confirm from an "ncdump -h single_member.nc" that the ensemble is on a fixed time axis?

Cheers,
Russ




On 23/07/18 14:45, saurabh rathore wrote:
Dear Russ

Following steps working correctly but just for L=1 when it goes to L=2 it is showing following error.

!-> REPEAT: L=2
!-> REPEAT: MM:1
!-> REPEAT: KK:1
 !-> LIST/FORMAT=CDF/append/file=hist_russ.nc/m=1/k=1:4 histzd
 LISTing to file hist_russ.nc
 **TMAP ERR: Requested data range is outside of data set limits
             Expected: HISTZD[L=           2 :           2 ] Found: HISTZD[L=           1 :           1 ]
LIST/FORMAT=CDF/append/file=hist_russ.nc/m=1/k=1:4 histzd
Command file, command group, or REPEAT execution aborted

But if I put llimits also in this command

save/i=1/j=1/k=1/l=1/m=1/ilimits=1:1:360/jlimits=1:126/klimits=1:36/mlimits=1:11/llimits=1:312/file=myfile.nc/clob myvar

then my code is running without any error. So is it saving the correct data ? for all the models and for all time steps ?

regards, saurabh

On Mon, Jul 23, 2018 at 1:59 PM, Russ Fiedler <russell.fiedler@xxxxxxxx> wrote:

Hi Saurabh,

If you really need to save it then you you can use multiple loops

! assume L is unlimited so we do not need to specify LLIMITS save by level to reduce memory.

save/i=1/j=1/k=1/l=1/m=1/ilimits=1:1:360/jlimits=1:126/klimits=1:36/mlimits=1:11/file=myfile.nc/clob myvar
repeat/l=1:312 (repeat/range=1:11/name=mm (repeat/range=1:36/name=kk save/app/file=myfile.nc/m=`mm`/k=`kk` myvar))


To reduce file size you can set the variable to a 4 byte real rather than the default double precision.

set var/outtype=outtype myvar

Also I'd strongly recommend using deflation and chunking for lossless compression

SET LIST/XCHUNK=360/YCHUNK=126/ZCHUNK=1/TCHUNK=1/ECHUNK=1/DEFLATE=1/SHUFFLE/NCFORMAT=4

If you want to change the chunking for the X,Y,Z and E dimensions  then you should change the looping above to match

e.g. if ZCHUNK=9

! Save blocks of 9 levels
repeat/l=1:312 (repeat/range=1:11/name=mm (repeat/range=1:36:9/name=kk save/app/file=myfile.nc/m=`mm`/k=`kk`:`kk+8` myvar))


Cheers,
Russ


On 23/07/18 13:15, saurabh rathore wrote:
Dear Satyesh,

The way that you mentioned is more like

repeat/l=1:312:1 (save/append/file=ensemble_mean.nc var;set mem/size=xxxx)

but I am having 11 models so this size of saving data is not possible as in my final variable I am having 5 dimensions i.e. i,k,l,k,m where i=1:360, j=1:126, k=1:36, m=1:11. here m=1:11 are the 11 models

sh gr histzd
    GRID (G013)  Ensemble Aggregetion Grid
 name                           axis              # pts                      start                      end                           subset
 LON                      LONGITUDE          360mr                  0E                           1W                            full
 LAT28_153             LATITUDE           126 r                    62.5S                       62.5N                        full
 LEV1_36                  DEPTH (m)           36 i-                     5                           1984.9                      full
 HISTM                        TIME               312 r               30-JAN-1980 00:00     30-DEC-2005 00:00       full
 ENSEMBLE           E (realization)        11 r                             1                             11                      full
 normal    F

now how to save this massive amount of data ?

regards, saurabh


On Thu, Jul 19, 2018 at 10:29 AM, saurabh rathore <rohitsrb2020@xxxxxxxxx> wrote:
Dear Satyesh,

thanks for your email. I will get back to you after trying this method. 

regards, saurabh

On Tue, Jul 17, 2018 at 9:31 PM, Satyesh Ghetiya <satyeshghetiya@xxxxxxxxx> wrote:
Hi Saurabh,

  You can use 'save' command with 'repeat' loop, every time listing just one subset of the full ensemble variable. For example if your variable is 'var', find the 'I' index range (by 'sh data') and use the following: 
 Suppose 'I' index ranges from 1 to 144, then your syntax will become as follows (you can further look what 'ilimits' and 'klimits' etc. does)  :

save/file=Yourfile.nc/ilimits=1:144/i=1 var[d=1];
repeat/range=2:144/name=aa (save/file=Yourfile.nc/append var[d=1,i=`aa`;])

I hope this is relevant and helps you.


On Tue, Jul 17, 2018 at 1:38 PM, saurabh rathore <rohitsrb2020@xxxxxxxxx> wrote:
Dear Ferreters,

I am having 11 CMIP models data in lat,lon,time,depth. I made an ensemble variable of dimension lat,lon,depth,time,ensemble_member i.e. 360,180,36,312,11. which is 360 longitudes point, 180 latitudes points, 36 depth levels, 312 time steps, 11 no. of models.

i want to save this 5-dimension ensemble variable into netcdf file. Is there any efficient way to save as my HPC is limited by memory.

cheers, saurabh

--


REGARDS

Saurabh Rathore
Research Scholar (PhD.)
Centre For Oceans, Rivers, Atmosphere & Land Science Technology
Indian Institute Of Technology, Kharagpur
contact :- 91- 8345984434



--
Ghetiya Satyeshkumar Girdharbhai,
Scientist/Engineer 'SC',
OIBMD, OSG, ECSA,
NRSC,
Hyderabad.



--


REGARDS

Saurabh Rathore
Research Scholar (PhD.)
Centre For Oceans, Rivers, Atmosphere & Land Science Technology
Indian Institute Of Technology, Kharagpur
contact :- 91- 8345984434



--


REGARDS

Saurabh Rathore
Research Scholar (PhD.)
Centre For Oceans, Rivers, Atmosphere & Land Science Technology
Indian Institute Of Technology, Kharagpur
contact :- 91- 8345984434





--


REGARDS

Saurabh Rathore
Research Scholar (PhD.)
Centre For Oceans, Rivers, Atmosphere & Land Science Technology
Indian Institute Of Technology, Kharagpur
contact :- 91- 8345984434





--


REGARDS

Saurabh Rathore
Research Scholar (PhD.)
Centre For Oceans, Rivers, Atmosphere & Land Science Technology
Indian Institute Of Technology, Kharagpur
contact :- 91- 8345984434



--


REGARDS

Saurabh Rathore
Research Scholar (PhD.)
Centre For Oceans, Rivers, Atmosphere & Land Science Technology
Indian Institute Of Technology, Kharagpur
contact :- 91- 8345984434





--


REGARDS

Saurabh Rathore
Research Scholar (PhD.)
Centre For Oceans, Rivers, Atmosphere & Land Science Technology
Indian Institute Of Technology, Kharagpur
contact :- 91- 8345984434



--
G Satyesh




--


REGARDS

Saurabh Rathore
Research Scholar (PhD.)
Centre For Oceans, Rivers, Atmosphere & Land Science Technology
Indian Institute Of Technology, Kharagpur
contact :- 91- 8345984434




--


REGARDS

Saurabh Rathore
Research Scholar (PhD.)
Centre For Oceans, Rivers, Atmosphere & Land Science Technology
Indian Institute Of Technology, Kharagpur
contact :- 91- 8345984434


[Thread Prev][Thread Next][Index]
Contact Us
Dept of Commerce / NOAA / OAR / PMEL / Ferret

Privacy Policy | Disclaimer | Accessibility Statement