[Thread Prev][Thread Next][Index]

Re: Real time data in LAS



Dave,

This has been bugging us enough that LAS v6.2 will have a fix.  You will be able to direct users to the LAS constraints page by creating a URL using dataset and variable titles (with escaped spaces) like this:

http://foo.bar.com/las/servlets/dataset?dset=Levitus%20Climatology/SALINITY

If you are using categories, this functionality comes with a caveat.  You will need to know the path defined by your categories and use that as your reference.  For our NVODS server this might look like:

http://foo.bar.com/las/servlets/dataset?dset=by%20Dataset/Levitus%20Climatology/SALINITY

The problem that is not solved in LAS v6.2 involves having a single, unique dataset referenced in multiple hierarchies.  This is also seen when you use the search field on our NVODS server where a single dataset will appear as a result multiple times, once for each hierarchy in which it is found.

The solution is to have both hierarchical categories for user interface presentation and a hidden, flat representation for non-UI access.  We just haven't had the time to code it yet.  It's high on our agenda as an item for LAS 6.3 which hopefully won't take 9 months to get out. ;-)


-- Jon


Dave Brown wrote:
Hi Jon,

I have a related issue that I would like to bring up. The NCAR/UCAR 
Community Data Portal (http:/dataportal.ucar.edu) needs to link
from various places to particular datasets served by a single LAS.
For example, on the CDP home page, under the "Live Access Server"
heading, clicking on the first entry "CDAS" leads to the following
URL:

http://dataportal.ucar.edu:9030/las6/servlets/dataset?catitem=217

The last part of the URL, the catitem number, is generated automatically
when genLAS.pl is run to configure the database. When changes are made
to the catalog and genLAS.pl is run to reconfigure the server, there is
no assurance that the catitem number will remain the same. This means
that whenever the LAS is reconfigured, all links to particular datasets
served by the LAS must be checked and possibly updated. Because we
want to have links from various pages, not all of which can be anticipated
at this point, it seems that this requirement will become very cumbersome
as the CDP grows. I'm wondering if it would be possible to implement
a means for the LAS site administrator to specify a name to be used as
the catitem identifier instead of an automatically generated number.
 -dave     



On Tue, 23 Sep 2003, Jonathan Callahan wrote:

  
All,

Support for real time updates is high on our agenda for future 
enhancements.  The only site I'm aware of that has real time updates is 
the NAVO site using LAS5.  Their data looks like it's updated twice 
daily and I assume they regenerate the entire interface when they get 
new data.

http://pdas.navo.navy.mil/las/main.pl?

We have added a couple of enhancements to LAS v6.2 (due out next week!) 
that are in preparation for better support of real time updates.  First 
is a new property associated with the product server that can disable 
the cache on a per dataset/variable basis.  It will look like this in 
the dataset XML:

<properties>
  <product_server>
    <use_cache>false</use_cache>
  </product_server>
</properties>

Setting "use_cache" to "false" will prevent LAS requests from retrieving 
previously generated results from cache.  This is particularly useful 
for model forecast results whose times are often forecast hour:  1-60.  
Each day the underlying data will change but an LAS request for the 24 
hour forecast will be exactly the same.  Disabling caching for these 
variables guarantees that a new image is generated.

The next item that is not done but that should be straightforward is a 
modification to las/xml/perl/genLas.pl.  Whenever you run 'make' in the 
las/server/ directory, you are actually running the genLas.pl file which 
reads the configuration XML files and then repopulates the MySQL 
database with new information.  All of the selection widgets in the LAS 
interface are retrieved on-the-fly from information stored in the MySQL 
database.  If you have a lot of datasets updating at different intervals 
you might not want to run genLas.pl on the whole set every time there 
was an update.  However, by altering genLas.pl to accept a dataset name 
as an argument, the MySQL database could be updated on a dataset by 
dataset basis.  Whatever cron job you use to update your data files 
would have the job, after the update, of running genLas.pl to update the 
LAS interface for that dataset.  There would be no need to restart 
Tomcat as the LAS templates are not changing, only the information in MySQL.

We have a code freeze on right now in order to do pre-release testing of 
LAS, but I encourage anyone to have a look at modifying genLas.pl in 
this manner.  This mod would be useful even without the <use_cache> 
property.

As far as OPeNDAP is concerned, there are a few issues here.  First you 
want to be careful that you are not using OPeNDAP caching as you might 
get old results even though LAS is generating a new image.   Inside the 
.dodsrc file -- if you have one -- you will see:

# DODS client configuration file. See the DODS
# users guide for information.
USE_CACHE=0
MAX_CACHE_SIZE=20
MAX_CACHED_OBJ=5
IGNORE_EXPIRES=1
CACHE_ROOT=/home/ja8/mclean/.dods_cache/
DEFAULT_EXPIRES=86400
ALWAYS_VALIDATE=1

Notice that you can turn off OPeNDAP caching with USE_CACHE=0.  
Alternatively, you can set ALWAYS_VALIDATE=1 which should guarantee that 
you always access updated files.  The caveat is that you should test 
this functionality to see if it really works.  Older versions of Ferret 
have older versions of the DODS libraries that may not property support 
this functionality.  The standard OPeNDAP NetCDF server is actually a 
CGI process and doesn't use Tomcat so no Tomcat restart should be necessary.

NOTE__  The OPeNDAP aggregation server does use Tomcat and may require a 
restart whenever data is updated.  Joe Mclean in our group is working 
with the aggregation server developer to debug this and other 
aggregation server issues. (Not may others are using the aggregation 
server, we hear, so this probably doesn't apply to most of you yet.)

The bottom line is that we intend for LAS to be a good solution for data 
with real time updates.  We feel that we are very close and would like 
to work more closely with someone with real time needs to work out the 
last few wrinkles.  In your case, Tony, that'll happen very soon.


-- Jon

Tony Jolibois wrote:

    
        Dear LAS users,

After solar data, I would like to know if there's a LAS which serve 
real time data : how can we put some data every 10 minutes, if we have 
to regenerate the LAS interface ? And if the data are stored in an 
OPeNDAP, we also have to restart the Tomcat server every 10 min. Is 
the LAS a good technical solution for real time data ?

Any information would be appreciate.

Tony
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tony Jolibois
CLS, Direction Oc?anographie Spatiale
D?partement Produits et Services
8-10 rue Hermes,
31526 Ramonville-St-Agne, Cedex, France
Email: tjolibois@cls.fr
Tel: (+33) (0) 561 393 797
Fax: (+33) (0) 561 393 782
Internet: http://www.cls.fr <http://www.cls.fr/>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      
    

  

[Thread Prev][Thread Next][Index]

Dept of Commerce / NOAA / OAR / PMEL / TMAP
Contact Us | Privacy Policy | Disclaimer | Accessibility Statement