yes, I understand. I should also mention this tool:
http://coastwatch.pfeg.noaa.gov/erddap/index.html
Erddap is a means of collecting together data from any of a variety of data formats and methods of access, viewing them and getting subsets. It's not difficult to install and can be set up either for your own use on a local machine or as a way to make data available to others.
On the page I pointed to there is a range of introductory material, including a video intro. It might be worth a look.
Ansley
http://coastwatch.pfeg.noaa.gov/erddap/index.html
On 4/14/2014 12:18 PM, Ge Peng - NOAA Affiliate wrote:
Hi Ansley,
I was hoping that I do not have to deal with each individual file.
Thanks for your detailed information and tips - they are very helpful.
--- Peng
On Mon, Apr 14, 2014 at 2:00 PM, Ansley Manke <ansley.b.manke@xxxxxxxx> wrote:
Hi Peng,
Descriptor files won't work for this kind of data. Descriptor files are for the case of a set of gridded datasets which have the same variables on the same grid in the non-Time directions, and time step information consistent with different timestep(s) on a time axis in each file. In effect you need to have a gridded dataset that has its timesteps in separate files.
There are dataset types in CF which describe non-gridded data: collections of profiles, trajectories, and so forth. Your data is really of this type. The CF document is here, http://cf-convention.github.io/1.6.html, and you'd be talking about "Discrete Geometries" datasets; Chapter 9 and Appendix H.
There are some facilities for aggregating datasets as NCML files, if you have the data in a thredds server. I do not know the details about aggregating collections of swaths, but the information about that would be at Unidata (who support NetCDF and thredds). https://www.unidata.ucar.edu/software/
All of this is to say that there are some tools coming along for this kind of data, both in the CF/NetCDF community and for Ferret and LAS. We have been developing scripts and functions for Ferret to work with such collections, and our initial use of these - for collections of trajectories - will be described in an upcoming FAQ.
Right now, of course things depend on what you plan to do with the data.
To make subsets of the files, I'd suggest a Ferret script or a Unix script to call a Ferret subsetting script, having Ferret make a subset of each dataset in turn. Here's a sketch of what this might look like:
!
! subset.jnl Script for subsetting one swath file
! Example:
! go subset T2m.N14.Y99.D365
!
! The subset-region information could also be passed in as arguments.
define symbol infile= $1
define symbol outfile = ($infile)_out.nc
use "($infile).nc"
let lonmask = if lon gt 140 and lon lt 170 then 1
let latmask = if lat gt 65 then 1
let timemask = if time gt 3600 and time lg 4800 then 1
! This mask is 1 where all the XYT conditions are satisfied
let xytmask = lonmask* latmask* timemask
let mtime = xytmask* time
let mlon = xytmask* lon
...
! Now, these variables represent the original data, masked so at index values
! I corresponding to the region, there is data, otherwise values are missing.
! You could call COMPRESSL to keep only the good data; assign the variables
! the correct units and titles, and finally, write a new file with the subset.
let out_time = COMPRESSL(mtime)
set att/like=time out_time
...
let npts = `out_time[L=@ngd]`
save/clobber/file="($outfile)" out_time[L=1:`npts`]
...
save/append/file="($outfile)" out_t2m[L=1:`npts`]
cancel data/all
Ansley
On 4/11/2014 12:18 PM, Ge Peng - NOAA Affiliate wrote:
Hi everyone,
--I have a 33-yr dataset of daily swath files in one dimension (time, l=1:NL).NL varies with each daily file. Time, lat and lon are not on a fixed and evenly spaced grid.
The following is from ncdump -h of one of such files:
netcdf T2m.N14.Y99.D365 {dimensions:time = UNLIMITED ; // (137263 currently)variables:float t2m(time) ;t2m:long_name = "temperature at 2 m" ;t2m:units = "C" ;t2m:coordinates = "lon lat" ;float lat(time) ;lat:units = "degrees_north" ;lat:valid_range = -90., 90. ;float lon(time) ;lon:units = "degrees_east" ;lon:valid_range = 0., 360. ;float time(time) ;time:units = "seconds" ;time:long_name = "seconds from 0 UTC" ;}
The following is the output of show data in ferret:yes? show datacurrently SET data sets:1> ./T2m.N14.Y99.D365.nc (default)name title I J K LT2M temperature at 2 m ... ... ... 1:137263LAT ... ... ... 1:137263LON ... ... ... 1:137263
I would like to use a .desr file to load all nc files or some nc files for a given time period and create a mask to isolate T2M using a range of lon, lat, and time. for example, 140 < lon < 170; 65 < lat < 90; 3600 < time < 4800.
Is there a way to obtain the dimension of the records (time dimension in this case) to create a .desr file without having to open all nc files first?
I am not used to working with data this kind. Any suggestion/tips on ways to load and work with them are appreciated. It is preferred to do so without re-gridding unless I have no other way.
Thanks,
--- Peng
Ge Peng, Ph.D
Research Scholar
Cooperative Institute for Climate and Satellites NC
North Carolina State University
NOAA's National Climatic Data Center
151 Patton Ave, Asheville, NC 28801
ge.peng@xxxxxxxx
o: +1 828 257 3009
f: +1 828 257 3002
Following CICS-NC on Facebook
--
Ge Peng, Ph.D
Research Scholar
Cooperative Institute for Climate and Satellites NC
North Carolina State University
NOAA's National Climatic Data Center
151 Patton Ave, Asheville, NC 28801
ge.peng@xxxxxxxx
o: +1 828 257 3009
f: +1 828 257 3002
Following CICS-NC on Facebook
Following CICS-NC on Facebook
|
|