Re: [ferret_users] saving very large .nc file with repeat

[Paul Goddard <pgoddard@xxxxxxxxxxxxxxxxx>]

Thanks Saurabh,

That is a mistake, I do try to enclose the set/memory in the loop. 

Unfortunately, my script crashes before l=1 is saved; so I need another method in addition to this advice

Thanks,

Paul

On Thu, Apr 20, 2017 at 7:32 PM, saurabh rathore <rohitsrb2020@xxxxxxxxx> wrote:

HI Paul,

you can use repeat command with the set memory command so it will clear the ram at every time step.

suppose i am having 4000 time steps of variable test_var and i want to save them. I will use this repeat command like this

repeat/l=1:4000:1 (save/append/file=test.nc test_var; set mem/size=2000)

for your problem you can use it.

repeat/l=2:7305 (\
save/append/file = "/archive/Paul.Goddard/CM2.6/Storm_Surge_Project/CM2.6_Control_SSH_DETRENDED_01810101-02001231.nc" SSH_DETRENDED[l=`l`]; set mem/size=2000)

regards, saurabh

On Fri, Apr 21, 2017 at 11:57 AM, Paul Goddard <pgoddard@xxxxxxxxxxxxxxxxx> wrote:

Hello,

I am attempting to detrend SSH data over time at each ocean grid cell. The problem is that the data are very large, the grid is i=1:3600 , j=1:2700, and l=1:7305.

In the past, I was able to save large data by repeating (looping) over the time dimension. However, since a detrending calculation must happen as well; it is taking too long to even save the first year (going on 2 hours... and may crash before it saves l=1..)

Any ideas on the best way to save such large data? Given that this is the resolution of many of the ocean models for CMIP6, I think I better learn a good way to complete these tasks.

Tthank you in advance, Paul

Here is my script:

can data/all
can var/all
can win/all

use "./CM2.6_Control_SSH_01810101-02001231.nc"

set memory/size=99999

let P  = T[GT=SSH,l=1:7305]
let Q = SSH[i=1:3600,j=1:2700,l=1:7305]
SET GRID Q
GO regresst

let SSH_Detrended = SSH[i=1:3600,j=1:2700,l=1:7305] - qhat[i=1:3600,j=1:2700,l=1:7305]

!Control save
save/clobber/file = "/archive/Paul.Goddard/CM2.6/Storm_Surge_Project/CM2.6_Control_SSH_DETRENDED_01810101-02001231.nc" SSH_DETRENDED[l=1]

repeat/l=2:7305 (\
save/append/file = "/archive/Paul.Goddard/CM2.6/Storm_Surge_Project/CM2.6_Control_SSH_DETRENDED_01810101-02001231.nc" SSH_DETRENDED[l=`l`])

--

REGARDS

Saurabh Rathore

Research Scholar (PhD.)

Centre For Oceans, Rivers, Atmosphere & Land Science Technology

Indian Institute Of Technology, Kharagpur

contact :- 91- 8345984434