[Thread Prev][Thread Next][Index]

Re: Real time data in LAS


Support for real time updates is high on our agenda for future enhancements.  The only site I'm aware of that has real time updates is the NAVO site using LAS5.  Their data looks like it's updated twice daily and I assume they regenerate the entire interface when they get new data.


We have added a couple of enhancements to LAS v6.2 (due out next week!) that are in preparation for better support of real time updates.  First is a new property associated with the product server that can disable the cache on a per dataset/variable basis.  It will look like this in the dataset XML:


Setting "use_cache" to "false" will prevent LAS requests from retrieving previously generated results from cache.  This is particularly useful for model forecast results whose times are often forecast hour:  1-60.  Each day the underlying data will change but an LAS request for the 24 hour forecast will be exactly the same.  Disabling caching for these variables guarantees that a new image is generated.

The next item that is not done but that should be straightforward is a modification to las/xml/perl/genLas.pl.  Whenever you run 'make' in the las/server/ directory, you are actually running the genLas.pl file which reads the configuration XML files and then repopulates the MySQL database with new information.  All of the selection widgets in the LAS interface are retrieved on-the-fly from information stored in the MySQL database.  If you have a lot of datasets updating at different intervals you might not want to run genLas.pl on the whole set every time there was an update.  However, by altering genLas.pl to accept a dataset name as an argument, the MySQL database could be updated on a dataset by dataset basis.  Whatever cron job you use to update your data files would have the job, after the update, of running genLas.pl to update the LAS interface for that dataset.  There would be no need to restart Tomcat as the LAS templates are not changing, only the information in MySQL.

We have a code freeze on right now in order to do pre-release testing of LAS, but I encourage anyone to have a look at modifying genLas.pl in this manner.  This mod would be useful even without the <use_cache> property.

As far as OPeNDAP is concerned, there are a few issues here.  First you want to be careful that you are not using OPeNDAP caching as you might get old results even though LAS is generating a new image.   Inside the .dodsrc file -- if you have one -- you will see:

# DODS client configuration file. See the DODS
# users guide for information.

Notice that you can turn off OPeNDAP caching with USE_CACHE=0.  Alternatively, you can set ALWAYS_VALIDATE=1 which should guarantee that you always access updated files.  The caveat is that you should test this functionality to see if it really works.  Older versions of Ferret have older versions of the DODS libraries that may not property support this functionality.  The standard OPeNDAP NetCDF server is actually a CGI process and doesn't use Tomcat so no Tomcat restart should be necessary.

NOTE__  The OPeNDAP aggregation server does use Tomcat and may require a restart whenever data is updated.  Joe Mclean in our group is working with the aggregation server developer to debug this and other aggregation server issues. (Not may others are using the aggregation server, we hear, so this probably doesn't apply to most of you yet.)

The bottom line is that we intend for LAS to be a good solution for data with real time updates.  We feel that we are very close and would like to work more closely with someone with real time needs to work out the last few wrinkles.  In your case, Tony, that'll happen very soon.

-- Jon

Tony Jolibois wrote:
        Dear LAS users,

After solar data, I would like to know if there's a LAS which serve real time data : how can we put some data every 10 minutes, if we have to regenerate the LAS interface ? And if the data are stored in an OPeNDAP, we also have to restart the Tomcat server every 10 min. Is the LAS a good technical solution for real time data ?

Any information would be appreciate.

Tony Jolibois
CLS, Direction Océanographie Spatiale
Département Produits et Services
8-10 rue Hermes,
31526 Ramonville-St-Agne, Cedex, France
Email: tjolibois@cls.fr
Tel: (+33) (0) 561 393 797
Fax: (+33) (0) 561 393 782
Internet: http://www.cls.fr

[Thread Prev][Thread Next][Index]

Dept of Commerce / NOAA / OAR / PMEL / TMAP
Contact Us | Privacy Policy | Disclaimer | Accessibility Statement