As to the file size, you might use SAVE/OUTTYPE=float By default Ferret saves data in double precision and that's more than needed for lots of sorts of data.-AnsleyOn Sat, Apr 22, 2017 at 1:31 PM, Paul Goddard <pgoddard@xxxxxxxxxxxxxxxxx> wrote:I also ran the chunking method. This created 3.4 GB file in about 4 hours. When loaded and shade in ferret/pyferret, the data are missing values for the complete file. Also, when I ncdump -h, I receive an error NetCDF: Unknown file format. I am a novice when it comes to understanding how the computer is compressing the file via chunk, though I am trying to understand through the ferret doc and its reference to unidata and its documention on netcdf4 file compression.I was able to produce my desired detrended dataset via repeating through one latitude at a time rather than through time and space as I previous attempted. This took about 24 hours; and the resulting .nc is the same size as the original data 265 GB. So, not the quickest, and probably not the way I should go in the future if a quicker/more efficient method exists.Thanks Russ,I tried several of your methods and slight variations of, trying to understand how they work and what size the finished .nc file would be.
An update on working with my large dataset:
I include my modified script below, thanks for the help, Paul
use "./CM2.6_Control_SSH_01810101-02001231.nc "
set memory/size=99999
let P = T[GT=SSH,l=1:7305]
let Q = SSH[l=1:7305]
SET GRID Q
GO regresst
let SSH_Detrended = SSH - qhat
def sym XC 40
def sym YC 30
def sym TC 100
def sym chunking "XCHUNK=($XC)/YCHUNK=($YC)/TCHUNK=($TC)"
def sym filepath "./CM2.6_Control_SSH_DETRENDED_01810101-02001231_Chunks.nc "
def sym savecom save/($chunking)/NCFORMAT=4/deflate=1/shuffle/file="($filepa th)" On Thu, Apr 20, 2017 at 9:05 PM, Russ Fiedler <russell.fiedler@xxxxxxxx> wrote:
Oops a couple of errors there
set var/ouytype=real ssh_detrended
and I missed a slash in the following.
def sym savecom "save/($chunking)/NCFORMAT=4/deflate=1/shuffle/file=ssh_detr end.nc "
Russ
On 21/04/17 13:38, Russ Fiedler wrote:
Hi Paul,
Rather than doing this over all space and time it will be easier to to this one latitude at a time. The way you are doing it requires reading in the complete data set multiple times. If you SET MODE DIAGNOTIC you'll see what I mean. I think you are probably flushing the values of slope and intercept and have to recompute them.
Also, why save a complete copy of the detrended data? It's huge! It's over half a TB as you are storing in double precision. Lucky it's only SSH and not a 3D variable! You only need to save the slope and intercepts.
What I do is save the i=1,j=1 value with ILIMITS and JLIMITS specified. This will set up the file
let P = T[GT=SSH,l=1:7305]
let Q = SSH[l=1:7305]
SET GRID Q
GO regresst
let SSH_Detrended = SSH - qhat
! Save all values
save/i=1/j=1/l=1/clob/ilimits=1:3600/jlimits=1:2700/file=ssh _detrend.nc ssh_detrended
repeat/j=1:2700 save/app/file=ssh_detrend.nc ssh_detrended! See note below on chunking
! Save just the slope and intercept
save/i=1/j=1/clob/file=slope_int.nc/ilimits=1:3600/jlimits=1 slope,intercept:2700
repeat/j=1:2700 save/app/file=slope_int.nc slope,intercept
If the original data is is only 4 byte real then SET VAR/OUTTYPE=FLOAT is probably useful.
You may also want to chunk/compress the data in time and space depending on how you intend to access it.
Something like
set var/outputtype=real ssh_detrended
def sym XC 40
def sym YC 30
def sym TC 100
def sym chunking "XCHUNK=($XC)/YCHUNK=($YC)/TCHUNK=($TC)"
def sym savecom "save/($chunking)/NCFORMAT=4/deflate=1/shuffle file=ssh_detrend.nc"
($savecome)/i=1/j=1/l=1/clob/ilimits=1:3600/jlimits=1:2700 ssh_detrended
repeat/range=1:2700:($YC)/name=jb ( repeat/range=1:3600:($XC)/name =ib ($savecome)/app/i=`ib+($XC)-1` /j=`jb+($YC)-1` ssh_detrended)
Play with XC,YC and TC as you wish.
Russ
On 21/04/17 11:57, Paul Goddard wrote:
Any ideas on the best way to save such large data? Given that this is the resolution of many of the ocean models for CMIP6, I think I better learn a good way to complete these tasks.In the past, I was able to save large data by repeating (looping) over the time dimension. However, since a detrending calculation must happen as well; it is taking too long to even save the first year (going on 2 hours... and may crash before it saves l=1..)Hello,I am attempting to detrend SSH data over time at each ocean grid cell. The problem is that the data are very large, the grid is i=1:3600 , j=1:2700, and l=1:7305.
Tthank you in advance, Paul
Here is my script:
can data/all
can var/all
can win/all
use "./CM2.6_Control_SSH_01810101-02001231.nc "
set memory/size=99999
let P = T[GT=SSH,l=1:7305]
let Q = SSH[i=1:3600,j=1:2700,l=1:7305]
SET GRID Q
GO regresst
let SSH_Detrended = SSH[i=1:3600,j=1:2700,l=1:7305] - qhat[i=1:3600,j=1:2700,l=1:730 5]
!Control save
save/clobber/file = "/archive/Paul.Goddard/CM2.6/Storm_Surge_Project/CM2.6_Contr ol_SSH_DETRENDED_01810101-0200 " SSH_DETRENDED[l=1]1231.nc
repeat/l=2:7305 (\
save/append/file = "/archive/Paul.Goddard/CM2.6/Storm_Surge_Project/CM2.6_Contr ol_SSH_DETRENDED_01810101-0200 " SSH_DETRENDED[l=`l`])1231.nc