[Thread Prev][Thread Next][Index]

Re: [ExternalEmail] Re: [ferret_users] saving very large .nc file with repeat




Oops a couple of errors there

set var/ouytype=real ssh_detrended

and I missed a slash in the following.

def sym savecom "save/($chunking)/NCFORMAT=4/deflate=1/shuffle/file=ssh_detrend.nc"


Russ

On 21/04/17 13:38, Russ Fiedler wrote:

Hi Paul,

Rather than doing this over all space and time it will be easier to to this one latitude at a time. The way you are doing it requires reading in the complete data set multiple times. If you SET MODE DIAGNOTIC you'll see what I mean. I think you are probably flushing the values of slope and intercept and have to recompute them.

Also, why save a complete copy of the detrended data? It's huge! It's over half a TB as you are storing in double precision. Lucky it's only SSH and not a 3D variable! You only need to save the slope and intercepts.

What I do is save the i=1,j=1 value with ILIMITS and JLIMITS specified. This will set up the file

let P  = T[GT=SSH,l=1:7305]
let Q = SSH[l=1:7305]
SET GRID Q
GO regresst

let SSH_Detrended = SSH - qhat

! Save all values

save/i=1/j=1/l=1/clob/ilimits=1:3600/jlimits=1:2700/file=ssh_detrend.nc ssh_detrended
repeat/j=1:2700 save/app/file=ssh_detrend.nc ssh_detrended                                  ! See note below on chunking        

! Save just the slope and intercept

save/i=1/j=1/clob/file=slope_int.nc/ilimits=1:3600/jlimits=1:2700 slope,intercept
repeat/j=1:2700 save/app/file=slope_int.nc slope,intercept

If the original data is is only 4 byte real then SET VAR/OUTTYPE=FLOAT is probably useful.

You may also want to chunk/compress the data in time and space depending on how you intend to access it.

Something like

set var/outputtype=real ssh_detrended

def sym XC 40
def sym YC 30
def sym TC 100

def sym chunking "XCHUNK=($XC)/YCHUNK=($YC)/TCHUNK=($TC)"
def sym savecom "save/($chunking)/NCFORMAT=4/deflate=1/shuffle file=ssh_detrend.nc"
 
($savecome)/i=1/j=1/l=1/clob/ilimits=1:3600/jlimits=1:2700 ssh_detrended

repeat/range=1:2700:($YC)/name=jb ( repeat/range=1:3600:($XC)/name=ib ($savecome)/app/i=`ib+($XC)-1`/j=`jb+($YC)-1` ssh_detrended)

Play with XC,YC and TC as you wish.



Russ

On 21/04/17 11:57, Paul Goddard wrote:
Hello,

I am attempting to detrend SSH data over time at each ocean grid cell. The problem is that the data are very large, the grid is i=1:3600 , j=1:2700, and l=1:7305.

In the past, I was able to save large data by repeating (looping) over the time dimension. However, since a detrending calculation must happen as well; it is taking too long to even save the first year (going on 2 hours... and may crash before it saves l=1..)

Any ideas on the best way to save such large data? Given that this is the resolution of many of the ocean models for CMIP6, I think I better learn a good way to complete these tasks.

Tthank you in advance, Paul

Here is my script:

can data/all
can var/all
can win/all

use "./CM2.6_Control_SSH_01810101-02001231.nc"

set memory/size=99999

let P  = T[GT=SSH,l=1:7305]
let Q = SSH[i=1:3600,j=1:2700,l=1:7305]
SET GRID Q
GO regresst

let SSH_Detrended = SSH[i=1:3600,j=1:2700,l=1:7305] - qhat[i=1:3600,j=1:2700,l=1:7305]

!Control save
save/clobber/file = "/archive/Paul.Goddard/CM2.6/Storm_Surge_Project/CM2.6_Control_SSH_DETRENDED_01810101-02001231.nc" SSH_DETRENDED[l=1]

repeat/l=2:7305 (\
save/append/file = "/archive/Paul.Goddard/CM2.6/Storm_Surge_Project/CM2.6_Control_SSH_DETRENDED_01810101-02001231.nc" SSH_DETRENDED[l=`l`])




[Thread Prev][Thread Next][Index]
Contact Us
Dept of Commerce / NOAA / OAR / PMEL / Ferret

Privacy Policy | Disclaimer | Accessibility Statement