Re: Real time data in LAS

[Dave Brown <dbrown@xxxxxxxxxxxx>]

Hi Jon,

I have a related issue that I would like to bring up. The NCAR/UCAR 
Community Data Portal (http:/dataportal.ucar.edu) needs to link
from various places to particular datasets served by a single LAS.
For example, on the CDP home page, under the "Live Access Server"
heading, clicking on the first entry "CDAS" leads to the following
URL:

http://dataportal.ucar.edu:9030/las6/servlets/dataset?catitem=217

The last part of the URL, the catitem number, is generated automatically
when genLAS.pl is run to configure the database. When changes are made
to the catalog and genLAS.pl is run to reconfigure the server, there is
no assurance that the catitem number will remain the same. This means
that whenever the LAS is reconfigured, all links to particular datasets
served by the LAS must be checked and possibly updated. Because we
want to have links from various pages, not all of which can be anticipated
at this point, it seems that this requirement will become very cumbersome
as the CDP grows. I'm wondering if it would be possible to implement
a means for the LAS site administrator to specify a name to be used as
the catitem identifier instead of an automatically generated number.
 -dave     



On Tue, 23 Sep 2003, Jonathan Callahan wrote:

> All,
> 
> Support for real time updates is high on our agenda for future 
> enhancements.  The only site I'm aware of that has real time updates is 
> the NAVO site using LAS5.  Their data looks like it's updated twice 
> daily and I assume they regenerate the entire interface when they get 
> new data.
> 
> http://pdas.navo.navy.mil/las/main.pl?
> 
> We have added a couple of enhancements to LAS v6.2 (due out next week!) 
> that are in preparation for better support of real time updates.  First 
> is a new property associated with the product server that can disable 
> the cache on a per dataset/variable basis.  It will look like this in 
> the dataset XML:
> 
> <properties>
>   <product_server>
>     <use_cache>false</use_cache>
>   </product_server>
> </properties>
> 
> Setting "use_cache" to "false" will prevent LAS requests from retrieving 
> previously generated results from cache.  This is particularly useful 
> for model forecast results whose times are often forecast hour:  1-60.  
> Each day the underlying data will change but an LAS request for the 24 
> hour forecast will be exactly the same.  Disabling caching for these 
> variables guarantees that a new image is generated.
> 
> The next item that is not done but that should be straightforward is a 
> modification to las/xml/perl/genLas.pl.  Whenever you run 'make' in the 
> las/server/ directory, you are actually running the genLas.pl file which 
> reads the configuration XML files and then repopulates the MySQL 
> database with new information.  All of the selection widgets in the LAS 
> interface are retrieved on-the-fly from information stored in the MySQL 
> database.  If you have a lot of datasets updating at different intervals 
> you might not want to run genLas.pl on the whole set every time there 
> was an update.  However, by altering genLas.pl to accept a dataset name 
> as an argument, the MySQL database could be updated on a dataset by 
> dataset basis.  Whatever cron job you use to update your data files 
> would have the job, after the update, of running genLas.pl to update the 
> LAS interface for that dataset.  There would be no need to restart 
> Tomcat as the LAS templates are not changing, only the information in MySQL.
> 
> We have a code freeze on right now in order to do pre-release testing of 
> LAS, but I encourage anyone to have a look at modifying genLas.pl in 
> this manner.  This mod would be useful even without the <use_cache> 
> property.
> 
> As far as OPeNDAP is concerned, there are a few issues here.  First you 
> want to be careful that you are not using OPeNDAP caching as you might 
> get old results even though LAS is generating a new image.   Inside the 
> .dodsrc file -- if you have one -- you will see:
> 
> # DODS client configuration file. See the DODS
> # users guide for information.
> USE_CACHE=0
> MAX_CACHE_SIZE=20
> MAX_CACHED_OBJ=5
> IGNORE_EXPIRES=1
> CACHE_ROOT=/home/ja8/mclean/.dods_cache/
> DEFAULT_EXPIRES=86400
> ALWAYS_VALIDATE=1
> 
> Notice that you can turn off OPeNDAP caching with USE_CACHE=0.  
> Alternatively, you can set ALWAYS_VALIDATE=1 which should guarantee that 
> you always access updated files.  The caveat is that you should test 
> this functionality to see if it really works.  Older versions of Ferret 
> have older versions of the DODS libraries that may not property support 
> this functionality.  The standard OPeNDAP NetCDF server is actually a 
> CGI process and doesn't use Tomcat so no Tomcat restart should be necessary.
> 
> NOTE__  The OPeNDAP aggregation server does use Tomcat and may require a 
> restart whenever data is updated.  Joe Mclean in our group is working 
> with the aggregation server developer to debug this and other 
> aggregation server issues. (Not may others are using the aggregation 
> server, we hear, so this probably doesn't apply to most of you yet.)
> 
> The bottom line is that we intend for LAS to be a good solution for data 
> with real time updates.  We feel that we are very close and would like 
> to work more closely with someone with real time needs to work out the 
> last few wrinkles.  In your case, Tony, that'll happen very soon.
> 
> 
> -- Jon
> 
> Tony Jolibois wrote:
> 
> >         Dear LAS users,
> >
> > After solar data, I would like to know if there's a LAS which serve 
> > real time data : how can we put some data every 10 minutes, if we have 
> > to regenerate the LAS interface ? And if the data are stored in an 
> > OPeNDAP, we also have to restart the Tomcat server every 10 min. Is 
> > the LAS a good technical solution for real time data ?
> >
> > Any information would be appreciate.
> >
> > Tony
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > Tony Jolibois
> > CLS, Direction Océanographie Spatiale
> > Département Produits et Services
> > 8-10 rue Hermes,
> > 31526 Ramonville-St-Agne, Cedex, France
> > Email: tjolibois@cls.fr
> > Tel: (+33) (0) 561 393 797
> > Fax: (+33) (0) 561 393 782
> > Internet: http://www.cls.fr <http://www.cls.fr/>
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
>