[Thread Prev][Thread Next][Index]

Re: [las_users] LAS V7 : Unable to make intermediate netCDF file.



Hi,

Thanks.  This is indeed the problem.  It a bit of time figure out a solution, but I'll send a patch as soon as I have one.

Roland

Serra, Mr. Efren, Contractor, Code 7542 wrote:

Roland,

 

The actual data type from the MySQL describe command is:

 

+----------------------+------------------+------+-----+---------+-------+

| Field                | Type             | Null | Key | Default | Extra |

+----------------------+------------------+------+-----+---------+-------+

| t                    | int(10) unsigned |      | MUL | 0       |       |

+----------------------+------------------+------+-----+---------+-------+

 

I thought about this after sending you the last email.

Efren A. Serra (Contractor)
DeVine Consulting, Inc.
Naval Research Laboratory
Marine Meteorology Division
7 Grace Hopper Ave., STOP 2
Monterey, CA 93943
Code 7542
Office: 831-656-4650


From: Roland Schweitzer [mailto:Roland.Schweitzer@xxxxxxxx]
Sent: Monday, November 02, 2009 9:26 AM
To: Serra, Mr. Efren, Contractor, Code 7542
Cc: oar.pmel.las_users@xxxxxxxx
Subject: Re: [las_users] LAS V7 : Unable to make intermediate netCDF file.

 

Efren,

What is the actual mySQL data type used for the column that stores time in the database?

Turns out that's what I'm checking to see if I should convert the time values.  I may have to shore up the logic that deals with deciding whether or not to convert the time values based on your particular database schema.

Roland

Serra, Mr. Efren, Contractor, Code 7542 wrote:

Roland,

 

I think you’re absolutely correct.  We’re storing Argo profile/salinity data in a mysql RDBMS.  The logic that handles the profile data is actually storing hours since the UNIX epoch.

 

Here’s a typical SELECT statement:

 

SELECT pres,y,x,t,temp FROM profiles WHERE (temp<99999.0 OR temp>99999.0) AND ((x>=-180.0) AND (x<=179.0)) AND ((y>=-90) AND (y<=89)) AND ((pres>=0) AND (pres<=5000)) AND ((t>=349069.0) AND (t<=349069.0))

 

Multiplying 349069 by 3600 and using date +”%Y%m%d” -d “UTC 1970-01-01 1256648400 secs ” gives Tue Oct 27 13:00:00 UTC 2009 which is what I’m selecting on the UI.  The database_access element is configured as:

 

    <time>t</time>

    <time_type>double</time_type>

    <time_sample>346536</time_sample>

    <time_format></time_format>

    <time_units>hours since 1970-01-01 00:00:00</time_units>

    <hours_1970></hours_1970>

 

I thought this what I was supposed to do from the documentation at http://ferret.pmel.noaa.gov/Ferret/LAS/home/documentation/installer-documentation/adding-scattered-data/using-a-database/creating-the-dataset-configuration-file/.  What time_units should I have used?  Thank you very much for the reply.

Efren A. Serra (Contractor)
DeVine Consulting, Inc.
Naval Research Laboratory
Marine Meteorology Division
7 Grace Hopper Ave., STOP 2
Monterey, CA 93943
Code 7542
Office: 831-656-4650


From: Roland Schweitzer [mailto:Roland.Schweitzer@xxxxxxxx]
Sent: Monday, November 02, 2009 6:37 AM
To: Serra, Mr. Efren, Contractor, Code 7542
Cc: oar.pmel.las_users@xxxxxxxx
Subject: Re: [las_users] LAS V7 : Unable to make intermediate netCDF file.

 

Efren,

Thanks for sending along the debug output.  That's certainly a big help, but unfortunately it's not enough to say definitively why it's failing.  The trickiest part of the translation from the database result set to the intermediate netCDF file is the conversion of the time from how it's stored in the database to how it's stored in the netCDF file.  My guess is that is where the process is failing.

If time is stored in the database as a float or a double representing "unit of time" since "date" you should be able to specify the time_units in the database configuration and set the time_type to double and the conversion process will simply take the date values and store them in the netCDF time variable with the appropriate units.

If time is stored as a string that must be interpreted as a calendar data then you'll need to specify the format of the string using pattern derived from these symbols so that the code can convert the calendar date into a double value so store in the netCDF file.

Check out you're storing time and how you've configured it to be handled in the configuration process (according to these hints).  If you can't sort it out send the details of how time is stored in your database to the list and we'll figure it out together.

Roland

Serra, Mr. Efren, Contractor, Code 7542 wrote:

Folks,

 

I'm having the following issue with mysql profile data:

 

2009-10-29 21:16:57,192 DEBUG service.database.DatabaseTool - Created sqlFile

2009-10-29 21:16:57,192 DEBUG service.database.DatabaseTool - preparing to read sql file

2009-10-29 21:16:57,192 DEBUG service.database.DatabaseTool - Statement ready:  SELECT pres,y,x,t,temp FROM profiles WHERE (temp<99999.0 OR temp>99999.0) AND ((x>=-180.0) AND (x<=179.0)) AND ((y>=-90) AND (y<=89)) AND ((pres>=0) AND (pres<=5000)) AND ((t>=349045.0) AND (t<=349069.0))

2009-10-29 21:16:57,192 DEBUG service.database.DatabaseTool - Connecting to datbase: jdbc:mysql://localhost:3306/argo2?autoReconnect=true

2009-10-29 21:16:57,204 DEBUG service.database.DatabaseTool - executeQuery:  SELECT pres,y,x,t,temp FROM profiles WHERE (temp<99999.0 OR temp>99999.0) AND ((x>=-180.0) AND (x<=179.0)) AND ((y>=-90) AND (y<=89)) AND ((pres>=0) AND (pres<=5000)) AND ((t>=349045.0) AND (t<=349069.0))

2009-10-29 21:16:57,210 DEBUG service.database.DatabaseTool - Done with query.

2009-10-29 21:16:57,211 DEBUG service.database.DatabaseTool - Got netcdf filename: /godae/apache-tomcat-6.0.18/webapps/las/output/C83F6589068DB3AD77FEFD2C83BC1B9A_netcdf.nc

2009-10-29 21:16:57,211 DEBUG service.database.IntermediateNetcdfFile - Create new empty netCDF file.

2009-10-29 21:16:57,211 INFO  service.database.DatabaseBackendService - Database backend request failed: Unable to make intermediate netCDF file.

 

The database query returns 245 records.  Any help would be greatly appreciated.  Thank you.

 

Efren A. Serra (Contractor)

DeVine Consulting, Inc.

Naval Research Laboratory

Marine Meteorology Division

7 Grace Hopper Ave., STOP 2

Monterey, CA 93943

Code 7542

Office: 831-656-4650

 

 

 



[Thread Prev][Thread Next][Index]


Contact Us
Dept of Commerce / NOAA / OAR / PMEL / TMAP

Privacy Policy | Disclaimer | Accessibility Statement