National Oceanic and Atmospheric Administration
NOAA DChart/Dapper Administration
 

Loading data

The Dapper package includes a database loader (dapperload.sh) that updates the server with metadata from netCDF or Opendap profile files. Dapper currently supports the following profile conventions:

The loader is only used to aggregate in-situ data. Individual netCDF files of any convention can be served through by using the Dapper netCDF service.

Dapper assumes that each station in the database is a profile -- that is, each station is at a fixed (x,y,t) and measurements at the stations consist of measurements at multiple depth or heights. If each station is a time series (fixed (x,y,z) and varying t) then the dataset name must either contain the string "_time_" or end with the string "_time". For example, a dataset named "foo" will be considered to be a profile datasets, but a dataset named "foo_time_series" will be a time series.

There are three level of hierarchy for aggregated data. A dataset contains an aggregated group of profile files. A database (the top level) contains a group of datasets.

The database loader allows you to create or remove a database and dataset, load or unload a set of profile files from a dataset, and list the contents of a database or dataset.

Note: the database schema for the releases before 0.9.0 is not compatible with later releases. You need to drop the database from MySQL and recreate and reload the database if you loaded data with version 0.5.x.

You can get information on command options to the loader by executing dapperload.sh without any arguments:

    dapperload.sh
              

Dataset creation/removal

The create command creates a new database or adds a dataset to an existing database. For example:

    dapperload.sh -u <user> -p <pass> create newdatabase newdataset

creates the new dataset newdataset in the database newdatabase. If newdatabase doesn't exist it will be created.


The command:

    dapperload.sh -u <user> -p <pass> remove newdatabase newdataset

removes newdataset from the database newdatabase.

Adding profiles to a dataset

The load command adds files to a dataset. The files can be in a JAR or ZIP archive. For example:

    dapperload.sh -u <user> -p <pass> \
           load newdatabase newdataset /data/file1.nc /data/file2.nc

adds the files /data/file1.nc and /data/file2.nc to the dataset.

    dapperload.sh -u <user> -p <pass> \
           load newdatabase newdataset /data/archive.zip

adds all of the files in the archive /data/archive.zip to Dapper.

Although Dapper can read data from compressed zip or jar archives, the server will be much slower if compression is used.

If the files are already in the dataset, metadata from the file will be updated only if the file has been modified since the last time the files were loaded (unless the -f argument is specified).

You can specify a file containing the files to add by with the -l argument :

    dapperload.sh -u <user> -p <pass> \
           -l files load newdatabase newdataset

All files in a dataset should follow the same netCDF conventions. Results are unpredictable if the profile files follow different conventions.

Unloading profiles from a dataset

The unload command removes files from a dataset. For example:

    dapperload.sh -u <user> -p <pass> \
           remove newdatabase newdataset /data/file1.nc /data/file2.nc

removes the files /data/file1.nc and /data/file2.nc from the dataset.

Listing the contents of a database or dataset

The ls command list the files in a dataset or the datasets in a database. The command:

    dapperload.sh -u <user> -p <pass> ls newdatabase
              

lists all of the datasets in the database newdatabase. The command:

    dapperload.sh -u <user> -p <pass> ls newdatabase newdataset
                  

lists all of the files in the dataset newdataset.

Command reference

Usage: dapperload [OPTIONS] create database dataset_name
dapperload [OPTIONS] remove database dataset_name
dapperload [OPTIONS] load database dataset_name [file [file...]]
dapperload [OPTIONS] unload database dataset_name [file [file...]]
dapperload [OPTIONS] ls database [dataset_name]
Commands:
create create a new database and dataset
remove remove a dataset from a database
load load a set of files into a dapper
database and dataset
unload unload a set of files from a dapper
database and dataset
ls list datasets from a dapper
database or files in a dataset
Options for all commands:
-u [username] database username
-p [password] database password
-h [host] hostname of database server
-P [port] port number of database server
-v print incomprehensible debug information
to log file
Options for load command:
-f force updates of all files
Without this flag the database is only updated with files
that have been modified since the last database update
-l file containing list of files to add
Options for unload command:
-l file containing list of files to unload
DChart Home | Dapper Home | Documentation & Support Home
DOC
| NOAA | OAR | PMEL | EPIC
Privacy Policy | Disclaimer | Webmaster