Oops a couple of errors there
set var/ouytype=real ssh_detrended
and I missed a slash in the following.
def sym savecom
"save/($chunking)/NCFORMAT=4/deflate=1/shuffle/file=ssh_detrend.nc"
Russ
On 21/04/17 13:38, Russ Fiedler wrote:
Hi Paul,
Rather than doing this over all space and time it will be easier
to to this one latitude at a time. The way you are doing it
requires reading in the complete data set multiple times. If you
SET MODE DIAGNOTIC you'll see what I mean. I think you are
probably flushing the values of slope and intercept and have to
recompute them.
Also, why save a complete copy of the detrended data? It's huge!
It's over half a TB as you are storing in double precision.
Lucky it's only SSH and not a 3D variable! You only need to save
the slope and intercepts.
What I do is save the i=1,j=1 value with ILIMITS and JLIMITS
specified. This will set up the file
let P = T[GT=SSH,l=1:7305]
let Q = SSH[l=1:7305]
SET GRID Q
GO regresst
let SSH_Detrended = SSH - qhat
! Save all values
save/i=1/j=1/l=1/clob/ilimits=1:3600/jlimits=1:2700/file=ssh_detrend.nc
ssh_detrended
repeat/j=1:2700 save/app/file=ssh_detrend.nc
ssh_detrended ! See note below
on chunking
! Save just the slope and intercept
save/i=1/j=1/clob/file=slope_int.nc/ilimits=1:3600/jlimits=1:2700
slope,intercept
repeat/j=1:2700 save/app/file=slope_int.nc slope,intercept
If the original data is is only 4 byte real then SET
VAR/OUTTYPE=FLOAT is probably useful.
You may also want to chunk/compress the data in time and space
depending on how you intend to access it.
Something like
set var/outputtype=real ssh_detrended
def sym XC 40
def sym YC 30
def sym TC 100
def sym chunking "XCHUNK=($XC)/YCHUNK=($YC)/TCHUNK=($TC)"
def sym savecom "save/($chunking)/NCFORMAT=4/deflate=1/shuffle
file=ssh_detrend.nc"
($savecome)/i=1/j=1/l=1/clob/ilimits=1:3600/jlimits=1:2700
ssh_detrended
repeat/range=1:2700:($YC)/name=jb (
repeat/range=1:3600:($XC)/name=ib
($savecome)/app/i=`ib+($XC)-1`/j=`jb+($YC)-1` ssh_detrended)
Play with XC,YC and TC as you wish.
Russ
On 21/04/17 11:57, Paul Goddard wrote:
Hello,
I am attempting to detrend SSH data over time at each
ocean grid cell. The problem is that the data are very
large, the grid is i=1:3600 , j=1:2700, and l=1:7305.
In the past, I was able to save large data by repeating
(looping) over the time dimension. However, since a
detrending calculation must happen as well; it is taking
too long to even save the first year (going on 2 hours...
and may crash before it saves l=1..)
Any ideas on the best way to save such large data? Given
that this is the resolution of many of the ocean models for
CMIP6, I think I better learn a good way to complete these
tasks.
Tthank you in advance, Paul
Here is my script:
can data/all
can var/all
can win/all
use "./ CM2.6_Control_SSH_01810101-02001231.nc"
set memory/size=99999
let P = T[GT=SSH,l=1:7305]
let Q = SSH[i=1:3600,j=1:2700,l=1:7305]
SET GRID Q
GO regresst
let SSH_Detrended = SSH[i=1:3600,j=1:2700,l=1:7305] -
qhat[i=1:3600,j=1:2700,l=1:7305]
!Control save
save/clobber/file =
"/archive/Paul.Goddard/CM2.6/Storm_Surge_Project/ CM2.6_Control_SSH_DETRENDED_01810101-02001231.nc"
SSH_DETRENDED[l=1]
repeat/l=2:7305 (\
save/append/file =
"/archive/Paul.Goddard/CM2.6/Storm_Surge_Project/ CM2.6_Control_SSH_DETRENDED_01810101-02001231.nc"
SSH_DETRENDED[l=`l`])
|