Oops a couple of errors there
set var/ouytype=real ssh_detrended
and I missed a slash in the following.
def sym savecom "save/($chunking)/NCFORMAT=4/deflate=1/shuffle/file=ssh_ detrend.nc "
Russ
On 21/04/17 13:38, Russ Fiedler wrote:
Hi Paul,
Rather than doing this over all space and time it will be easier to to this one latitude at a time. The way you are doing it requires reading in the complete data set multiple times. If you SET MODE DIAGNOTIC you'll see what I mean. I think you are probably flushing the values of slope and intercept and have to recompute them.
Also, why save a complete copy of the detrended data? It's huge! It's over half a TB as you are storing in double precision. Lucky it's only SSH and not a 3D variable! You only need to save the slope and intercepts.
What I do is save the i=1,j=1 value with ILIMITS and JLIMITS specified. This will set up the file
let P = T[GT=SSH,l=1:7305]
let Q = SSH[l=1:7305]
SET GRID Q
GO regresst
let SSH_Detrended = SSH - qhat
! Save all values
save/i=1/j=1/l=1/clob/ilimits=1:3600/jlimits=1:2700/file=ssh _detrend.nc ssh_detrended
repeat/j=1:2700 save/app/file=ssh_detrend.nc ssh_detrended! See note below on chunking
! Save just the slope and intercept
save/i=1/j=1/clob/file=slope_int.nc/ilimits=1:3600/jlimits= slope,intercept1:2700
repeat/j=1:2700 save/app/file=slope_int.nc slope,intercept
If the original data is is only 4 byte real then SET VAR/OUTTYPE=FLOAT is probably useful.
You may also want to chunk/compress the data in time and space depending on how you intend to access it.
Something like
set var/outputtype=real ssh_detrended
def sym XC 40
def sym YC 30
def sym TC 100
def sym chunking "XCHUNK=($XC)/YCHUNK=($YC)/TCHUNK=($TC)"
def sym savecom "save/($chunking)/NCFORMAT=4/deflate=1/shuffle file=ssh_detrend.nc"
($savecome)/i=1/j=1/l=1/clob/ilimits=1:3600/jlimits=1:2700 ssh_detrended
repeat/range=1:2700:($YC)/name=jb ( repeat/range=1:3600:($XC)/ name=ib ($savecome)/app/i=`ib+($XC)-1` /j=`jb+($YC)-1` ssh_detrended)
Play with XC,YC and TC as you wish.
Russ
On 21/04/17 11:57, Paul Goddard wrote:
Any ideas on the best way to save such large data? Given that this is the resolution of many of the ocean models for CMIP6, I think I better learn a good way to complete these tasks.In the past, I was able to save large data by repeating (looping) over the time dimension. However, since a detrending calculation must happen as well; it is taking too long to even save the first year (going on 2 hours... and may crash before it saves l=1..)Hello,I am attempting to detrend SSH data over time at each ocean grid cell. The problem is that the data are very large, the grid is i=1:3600 , j=1:2700, and l=1:7305.
Tthank you in advance, Paul
Here is my script:
can data/all
can var/all
can win/all
use "./CM2.6_Control_SSH_01810101-02001231.nc "
set memory/size=99999
let P = T[GT=SSH,l=1:7305]
let Q = SSH[i=1:3600,j=1:2700,l=1:7305]
SET GRID Q
GO regresst
let SSH_Detrended = SSH[i=1:3600,j=1:2700,l=1:7305] - qhat[i=1:3600,j=1:2700,l=1: 7305]
!Control save
save/clobber/file = "/archive/Paul.Goddard/CM2.6/Storm_Surge_Project/CM2.6_ Control_SSH_DETRENDED_ " SSH_DETRENDED[l=1]01810101-02001231.nc
repeat/l=2:7305 (\
save/append/file = "/archive/Paul.Goddard/CM2.6/Storm_Surge_Project/CM2.6_ Control_SSH_DETRENDED_ " SSH_DETRENDED[l=`l`])01810101-02001231.nc