|
Hi Upendra, all, It's a good idea to be a bit careful about this, because it depends on the data you're serving. Turning off subsampling could be made to be an option that the end user could set, but let's start with the installer making this choice. Here's a way for the installer to skip the subsampling step for an individual dataset or variable. What we can do is to add a Ferret property, say subsample_t. Then that could be set to false when you don't want to decimate the data in time. We could do the same thing for subsampling in general of course, but let's just start with this. So the dataset configuration would start with a Ferret property, either for the dataset or for individual variables - probably for the entire dataset. For instance, <datasets> <id-83553fbd37 name="NAVO NCOM Relocatable Model: Fukushima Regional Forecast" url="" class="moz-txt-link-rfc2396E" href="http://edac-dap3.northerngulfinstitute.org/thredds/dodsC/ncom_fukushima_agg/Fukushima_best.ncd">"http://edac-dap3.northerngulfinstitute.org/thredds/dodsC/ncom_fukushima_agg/Fukushima_best.ncd"> <properties> <ferret> <subsample_t>false</subsample_t> </ferret> </properties> ... Then we can add a line to the script that does the subsampling. At the start of the script LAS_set_strides.jnl, there are a number of commands testing for cases where we will not do subsampling, or striding. We can add another, ! Ferret property subsample_t can be set to zero or false and prevent subsampling in time.I have put this in before any checking for memory, so that if the script ran out of Ferret memory it would return an error message rather than do subsampling. I've attached the script with this change. We'll think about whether to add this to the script for the next LAS release. Ansley On 6/9/2011 1:31 PM, Upendra Dadi wrote: Hi, |
! LAS_set_strides.jnl
! ACM
! From the region and the grid we can define striding. Compute strides
! based first on Ferret memory and then if it is a graphics product, on
! pixel size.
! ACM 8/312/2007 Do not issue the SET AXIS/STRIDE= command if the stride is 1
! This partially works around TRAC #224.
! ACM 3/2008 If the transform-to-1-season per year has been set then do not
! stride in t
! ACM 4/2008 For descriptor data or for data whose axes are replaced cant
! do native strides. In these cases set up index strides with
! a call to LAS_set_strides_nonative
! 10-Jul-2008 ACM Fixes for Google Earth placemark strides
! TODO: When there's going to be a transformation or something, then these
! calculations may not be sufficient. Might be better done by the transform script.
! Ferret property subsample_t can be set to zero or false and
! prevent subsampling in time.
IF `($ferret_subsample_t"1|0|1|true>1|false>0|*>1") EQ 0 AND ($ferret_view"|t>1|T>1|*>0")` THEN EXIT/SCRIPT
! If the operation is writing a netCDF file, that is done in chunks. Do not use strides
IF `(STRINDEX( "($operation_ID)", "Data_Extract" ) NE 0) AND \
(STRINDEX( "($result_ferret_listing_type)", "netcdf" ) NE 0)` THEN EXIT/SCRIPT
! See LAS_set_strides_curvilinear for first cut on dealing with strides and curvi data.
! If it is a native curvilinear plot then LAS_set_strides_curvilinear has been called.
IF ($ferret_curvi_coord_lat"0|0|*>1") THEN EXIT/SCRIPT
! Hybrid Z not dealt with here...
IF ($do_hybrid_z"0|0|1|*>1") THEN EXIT/SCRIPT
! Property-property plots have their own version of striding
IF ($its_prop_prop_plot"0|0|1|*>1") THEN EXIT/SCRIPT
! ---------------------------------------------------------------------
! Determine striding based on the memory allocation
DEFINE SYMBOL memsize = ($FERRET_MEMORY"6.4")* 1.e6
! Cannot do native striding if this is a descriptor dataset, or if it is
! from a curvilienar dataset but working in the rectilinear portion. In
! this case set up index strides.
DEFINE SYMBOL no_native = 0
IF ($data_var_only"0|*>1") THEN
DEFINE SYMBOL check_for_des = `($data_var_only)[d=($data_num)],RETURN=dsetpath`
ELSE
DEFINE SYMBOL check_for_des = `($data_var)[d=($data_num)],RETURN=dsetpath`
ENDIF
!IF `($analysis_def"0|*>1") EQ 0` THEN
IF `STRINDEX("($check_for_des)", ".des") GT 0 OR ($no_native_strides_xy"0|*>1")` THEN
DEFINE SYMBOL no_native = 1
LET pvar = ($data_var)[d=($data_num),($region_xyzt)]
GO LAS_set_strides_nonative.jnl
IF ($set_ndx_strides"0|*>1") THEN
DEFINE SYMBOL ferret_plot_var = ($expr1)($data_var)[d=($data_num),($set_ndx_strides)]($expr2)
ENDIF
IF ($analysis_def"0|*>1") THEN
LET analysis_plot_var = ($data_var_only)[d=($data_num),($set_ndx_strides)]
DEFINE SYMBOL ferret_plot_var = ($expr1)analysis_plot_var($analysis_def)($expr2)
ENDIF
EXIT/SCRIPT
ENDIF
!ENDIF
DEFINE SYMBOL nvars = ($data_count)
! If ($data_count) larger than 1 add the size of the variables.
LET pvar = ($ferret_plot_var)
DEFINE SYMBOL var_size = ($var_size"0") + `pvar,RETURN=size`
! The exception to the above is side-by-side plots; each var loaded separately
IF `STRINDEX( "($operation_ID)", "Side" ) NE 0` THEN
DEFINE SYMBOL var_size = `pvar,RETURN=size`
LET nvars = 1
ENDIF
IF `($num)+1 EQ ($nvars)` THEN
LET ndims = 0
LET xsiz = `pvar,RETURN=isize`
IF `xsiz GT 1` THEN LET ndims = `ndims + 1`
LET ysiz = `pvar,RETURN=jsize`
IF `ysiz GT 1` THEN LET ndims = `ndims + 1`
LET zsiz = `pvar,RETURN=ksize`
IF `zsiz GT 1` THEN LET ndims = `ndims + 1`
LET tsiz = `pvar,RETURN=lsize`
IF `tsiz GT 1` THEN LET ndims = `ndims + 1`
LET varsiz = xsiz* ysiz* zsiz* tsiz
DEFINE SYMBOL stride_for_mem = `INT( 1+ varsiz/(.5*($memsize)) )`
IF `ndims EQ 2` THEN DEFINE SYMBOL stride_for_mem = `INT(($stride_for_mem)^0.5)`
IF `ndims EQ 3` THEN DEFINE SYMBOL stride_for_mem = `INT(($stride_for_mem)^0.33)`
IF `ndims EQ 4` THEN DEFINE SYMBOL stride_for_mem = `INT(($stride_for_mem)^0.25)`
ENDIF
! If not a graphics product, apply the strides
IF `STRINDEX( "($operation_ID)", "Grid_for_GE" ) NE 0` THEN EXIT/SCRIPT
IF `STRINDEX( "($operation_ID)", "Plot" ) EQ 0 AND \
STRINDEX( "($operation_ID)", "zoom" ) EQ 0 AND \
STRINDEX( "($operation_ID)", "Anim" ) EQ 0 AND \
STRINDEX( "($operation_ID)", "Grid_for_GE" ) EQ 0` THEN
GO LAS_set_strides_for_mem
EXIT/SCRIPT
ENDIF
! If its a vector plot, let Ferret or a user setting of xskip,yskip decimate the vectors
IF `STRINDEX( "($operation_ID)", "vector" ) NE 0` THEN EXIT/SCRIPT
! ---------------------------------------------------------------------
! Now for graphics or GE products, do striding by pixel size, often larger stride than
! needed to fit in Ferret memory
LET maxpix = 600 ! default
IF ($ferret_size"0|0.06667>1|*>0") THEN LET maxpix = 300 ! small
IF ($ferret_size"0|0.25>1|*>0") THEN LET maxpix = 500 ! medium
IF ($ferret_size"0|0.5>1|*>0") THEN LET maxpix = 600 ! default
IF ($ferret_size"0|0.8333>1|*>0") THEN LET maxpix = 700 ! large
IF `STRLEN("($ferret_view)") EQ 1` THEN
IF ($ferret_decimate%0|none>1%) THEN EXIT/SCRIPT
IF ($ferret_decimate%0|default>1%) THEN LET maxpix = `maxpix*4`
IF ($ferret_decimate%0|light>1%) THEN LET maxpix = `maxpix*10`
IF ($ferret_decimate%0|extra>1%) THEN LET maxpix = `maxpix`
ENDIF
!IF `STRINDEX("($operation_ID)", "Anim" ) GT 0` THEN LET maxpix = 50
IF `($region_x_range"0|*>1") EQ 0` THEN
DEFINE SYMBOL no_region_x_range = 1
IF ($region_($num)_x_range"0|*>1") THEN\
DEFINE SYMBOL region_x_range = ($region_($num)_x_range)
ENDIF
IF `($region_y_range"0|*>1") EQ 0` THEN
DEFINE SYMBOL no_region_y_range = 1
IF ($region_($num)_y_range"0|*>1") THEN\
DEFINE SYMBOL region_y_range = ($region_($num)_y_range)
ENDIF
CANCEL SYMBOL no*axis
DEFINE SYMBOL xaxname = `($data_var),RETURN=xaxis`
IF `STRCMP("($xaxname)", "NORMAL") EQ 0` THEN DEFINE SYMBOL noxaxis = 1
IF `(STRINDEX("($ferret_view)","x") NE 0) AND \
($region_x_range"0|0|*>1") AND \
($no_native_strides_xy"0|*>1") NE 1 AND \
($region_x"0|0|*>1") AND \
(($noxaxis"0|1|*>1") NE 1)` THEN
LET xx = x[gx=($data_var)[d=($data_num)]]
LET pixsize = `xx[($region_x)@ngd]`
IF ($ferret_google_plot"0|*>1") THEN
IF `STRINDEX( "($operation_ID)", "Grid" ) NE 0` THEN
LET nx = `xx[($region_x)@ngd]` !total no. of grid points in x axis, will be used by the Google Earth placemarks
ENDIF
ENDIF
IF `pixsize GT maxpix` THEN DEFINE SYMBOL xstride = `INT((pixsize/maxpix)+1)`
DEFINE SYMBOL xstride = `MAX(($xstride"1"), ($stride_for_mem"1"))`
! Check whether X axis is modulo. check_modulo_factor.jnl checks whether the
! axis is modulo and if so, whether the stride is a factor of the lentgh of
! the axis, resetting it if possible to retain the modulo character of the axis.
GO check_modulo_factor x
DEFINE SYMBOL axname = `($data_var)[d=($data_num)],RETURN=xaxis`
IF `STRINDEX("($axname)", "(") EQ 0` THEN
IF `($xstride) GT 1 AND ($axis_stride_($axname)"0") EQ 0` THEN
SET AXIS/STRIDE=($xstride) ($axname)
LET x_strided_var = ($data_var)[d=($data_num)]
DEFINE SYMBOL axis_stride_($axname) = ($xstride)
DEFINE SYMBOL axis_stride_x = ($xstride)
ENDIF
ENDIF
IF ($SYMBOL no_region_x_range"0|*>1") THEN CANCEL SYMBOL region_x_range
ENDIF
DEFINE SYMBOL xaxname = `($data_var),RETURN=yaxis`
IF `STRCMP("($xaxname)", "NORMAL") EQ 0` THEN DEFINE SYMBOL noyaxis = 1
IF `(STRINDEX("($ferret_view)","y") NE 0) AND \
($region_y_range"0|0|*>1") AND \
($no_native_strides_xy"0|*>1") NE 1 AND \
($region_y"0|0|*>1") AND \
(($noyaxis"0|1|*>1") NE 1)` THEN
LET yy = y[gy=($data_var)[d=($data_num)]]
LET pixsize = `yy[($region_y)@ngd]`
IF ($ferret_google_plot"0|*>1") THEN
IF `STRINDEX( "($operation_ID)", "Grid" ) NE 0` THEN
LET ny = `yy[($region_y)@ngd]` !total no. of grid points in y axis, will be used by the Google Earth placemarks
ENDIF
ENDIF
IF `pixsize GT maxpix` THEN DEFINE SYMBOL ystride = `2*INT((pixsize/maxpix)+1)`
DEFINE SYMBOL ystride = `MAX(($ystride"1"), ($stride_for_mem"1"))`
DEFINE SYMBOL axname = `($data_var)[d=($data_num)],RETURN=yaxis`
IF `STRINDEX("($axname)", "(") EQ 0` THEN
IF `($ystride) GT 1 AND ($axis_stride_($axname)"0") EQ 0` THEN
SET AXIS/STRIDE=($ystride) ($axname)
LET y_strided_var = ($data_var)[d=($data_num)]
DEFINE SYMBOL axis_stride_($axname) = ($ystride)
DEFINE SYMBOL axis_stride_y = ($ystride)
ENDIF
ENDIF
IF ($SYMBOL no_region_y_range"0|*>1") THEN CANCEL SYMBOL region_y_range
ENDIF
DEFINE SYMBOL xaxname = `($data_var),RETURN=zaxis`
IF `STRCMP("($xaxname)", "NORMAL") EQ 0` THEN DEFINE SYMBOL nozaxis = 1
IF `(STRINDEX("($ferret_view)","z") NE 0) AND \
($region_z_range"0|0|*>1") AND \
($region_z"0|0|*>1") AND \
(($nozaxis"0|1|*>1") NE 1)` THEN
LET zz = z[gz=($data_var)[d=($data_num)]]
LET pixsize = `zz[($region_z)@ngd]`
IF `pixsize GT maxpix` THEN DEFINE SYMBOL zstride = `INT((pixsize/maxpix)+1)`
DEFINE SYMBOL zstride = `MAX(($zstride"1"), ($stride_for_mem"1"))`
DEFINE SYMBOL axname = `($data_var)[d=($data_num)],RETURN=zaxis`
IF `STRINDEX("($axname)", "(") EQ 0` THEN
IF `($zstride) GT 1 AND ($axis_stride_($axname)"0") EQ 0` THEN
SET AXIS/STRIDE=($zstride) ($axname)
LET z_strided_var = ($data_var)[d=($data_num)]
DEFINE SYMBOL axis_stride_($axname) = ($zstride)
DEFINE SYMBOL axis_stride_z = ($zstride)
ENDIF
ENDIF
ENDIF
! If this is an animation then the time step is controlled by that. Do not do
! striding in time.
DEFINE SYMBOL xaxname = `($data_var),RETURN=taxis`
IF `STRCMP("($xaxname)", "NORMAL") EQ 0` THEN DEFINE SYMBOL notaxis = 1
LET trange = STRCMP("($region_t_lo%0%)", "($region_t_hi%1%)")
IF `(STRINDEX("($ferret_view)","t") NE 0) AND \
($region_t"0|0|*>1") AND ( ($DO_TRANS_1_SEAS_PER_YEAR"0") EQ 0) AND \
(trange NE 0 AND STRINDEX("($operation_ID)", "Animation_Frames") EQ 0) AND \
(($notaxis"0|1|*>1") NE 1)` THEN
LET tt = t[gt=($data_var)[d=($data_num)]]
LET pixsize = `tt[($region_t)@ngd]`
DEFINE SYMBOL tstride = `INT((pixsize/maxpix)+1)`
DEFINE SYMBOL tstride = `MAX(($tstride"1"), ($stride_for_mem"1"))`
! Check whether T axis is modulo. check_modulo_factor.jnl checks whether the
! axis is modulo and if so, whether the stride is a factor of the lentgh of
! the axis, resetting it if possible to retain the modulo character of the axis.
GO check_modulo_factor t
DEFINE SYMBOL axname = `($data_var)[d=($data_num)],RETURN=taxis`
IF `STRINDEX("($axname)", "(") EQ 0` THEN
IF `($tstride) GT 1 AND ($axis_stride_($axname)"0") EQ 0` THEN
SET AXIS/STRIDE=($tstride) ($axname)
LET t_strided_var = ($data_var)[d=($data_num)]
DEFINE SYMBOL axis_stride_($axname) = ($tstride)
DEFINE SYMBOL axis_stride_t = ($tstride)
ENDIF
ENDIF
ENDIF
! Add a label about the striding at the upper left.
LET need_comma = 0
IF `($xstride"0|1>0|*>1") + ($ystride"0|1>0|*>1") + ($zstride"0|1>0|*>1") + ($tstride"0|1>0|*>1") GT 0` THEN
DEFINE SYMBOL n_left_labels = `($n_left_labels"0")+1`
DEFINE SYMBOL upper_left_($n_left_labels) = Subsampled
CANCEL SYMBOL need_comma
IF `($xstride"0|1>0|*>1")` THEN
DEFINE SYMBOL upper_left_($n_left_labels) =\
($upper_left_($n_left_labels)) ($xstride) in X
LET need_comma = 1
ENDIF
IF `($ystride"0|1>0|*>1")` THEN
IF `need_comma GT 0` THEN DEFINE SYMBOL upper_left_($n_left_labels) =\
($upper_left_($n_left_labels)),
DEFINE SYMBOL upper_left_($n_left_labels) =\
($upper_left_($n_left_labels)) ($ystride) in Y
LET need_comma = 1
ENDIF
IF `($zstride"0|1>0|*>1")` THEN
IF `need_comma GT 0` THEN DEFINE SYMBOL upper_left_($n_left_labels) =\
($upper_left_($n_left_labels)),
DEFINE SYMBOL upper_left_($n_left_labels) =\
($upper_left_($n_left_labels)) ($zstride) in Z
LET need_comma = 1
ENDIF
IF `($tstride"0|1>0|*>1")` THEN
IF `need_comma GT 0` THEN DEFINE SYMBOL upper_left_($n_left_labels) =\
($upper_left_($n_left_labels)),
DEFINE SYMBOL upper_left_($n_left_labels) =\
($upper_left_($n_left_labels)) ($tstride) in T
LET need_comma = 1
ENDIF
DEFINE SYMBOL note_num = `($note_num"0") + 1`
DEFINE SYMBOL note_($note_num)_lab = ($upper_left_($n_left_labels))
ENDIF
! End of $RCSfile ------------LAS_set_strides.jnl--------------------------